Description

shipper is a tool that tries to reduce the friction cost of shipping software releases to as near zero as possible.

Its job is to make it possible for you to run the command shipper version=NNN | sh -e -x in the top-level directory of a project and have a release be properly exported to all the places that you normally deliver it — your personal website, Linux source code archive sites, forge sites, and distribution submission queues; also, to send appropriate notifications to freshcode.club, mailing lists, project IRC channels, and Twitter.

Another goal is to arrange your shipping process in such a way that any one piece of metadata (like your project name, version, or synoptic description) only has to be kept in one place and modified once per release. Metadata is collected from several locations that naturally own pieces of it.

In normal use, you only absolutely need to set only two variables: destinations which is the list of destinations to ship to, and version which is the release version of the software you are shipping. The default package name is the basename of the directory from which shipper is run.

When adding freshcode.club as a destination, you may also want to add “Project-Tags” headers to your project specfiles that will become Project tags.

Once you have shipper up and running, you can experiment with more advanced features such as: having the program generate project web pages for you using a simple templating system, and automatically tagging your shipped releases.

The output of shipper, run in your top-level project source directory, is a shellscript. To ship your project, feed the script to a shell (of course you can review it before you run it). The only shell features required are command execution and here-documents. No wildcard expansion, shell control syntax, or shell variable evaluation will be required. In practical use you will probably want to pass the shell the -e option (any command failure aborts the script, so notifications sent later don't go out if an early upload fails) and the -x option (to echo shipping commands as they are performed).

Theory of Operation

shipper generates commands to push deliverables out to destinations. Deliverables include: source tarballs, tarball checksums, zip archives, ChangeLog files, README files,and various other project metadata files. Destinations include both private destinations like websites, FTP archive sites and mailing lists, and public destinations like soureceforge.net, freshcode.com, savannah.nongnu.org and berlios.de.

shipper goes through the following steps each time it runs:

First, shipper performs pre-ship checks (this may be disabled by the -N or --no-check). If the current directory is a version-control repository (git, hg, and svn are supported) it complains and bails out if the repository has uncommitted local changes. Then, if the project metadata includes a validation check command, that is run and shipper bails out with a complaint if it returns a nonzero status.

Next, shipper reads the .shipper file in your home directory and the current directory (if you like XDG conventions, make a ~/.config/shipper instead). These can set various control variables and templates that aren't project specific. These are regular Python files; see the section called “Configuration in More Detail” for more detail.

If that step didn't find your identity (a nonempty value of whoami), the program's second step is to try to figure out what your name and email address are. It snoops the profiles associated with git, bzr, and hg looking for these. If it can't deduce this information, it will complain and exit.

Next, shipper reads per-project information from a project metadata file in the current directory, either a Debian control file or an RPM spec. This is where it will normally find the project name, a short text description, its declared homepage location, and other similar things.

The program then reads .shipper for project-specific configuration settings.

Next, if there is a .git/config file, it looks for project-specific settings in a [shipper] section. This is where you put settings that you don't want to export in the tarball, like hosting-site links.

Finally, any override settings specified on the shipper command line are applied. Normally version, the release version designation, will be set here.

The -d (dump) option causes shipper to dump all configuration values and exit immediately after collecting them in step 3.

The next phase is finding deliverables. These are either unversioned project metadata (like a README or NEWS file) or versioned product files such as source tarballs.

For some unversioned metadata - README, COPYING, AUTHORS, NEWS, BUGS, and TODO - the program looks for a corresponding asciidoc (.adoc) file and renders it to HTML, which will be used in generating the resource table that goes in the default templated web page.

The last step before generating upload commands may be to generate an HTML file for uploading, if you have specified the html_target variable. (Under some circumstances informational files for a remote Gitweb repository may be generated at this stage as well.)

The -g option tells shipper to use gitlab-igor to put tarballs and their checksums on the project's GitLab Releases page rather than on the normal resource page. For this to work, gitlab-igor.py must be in your $PATH and you must have a Personal Access Token declared in your ~/.netrc file.

The option -w (web-only) tells shipper to generate upload commands for web pages and updates for gitweb views only, rather than for shipping (and possibly tagging) a release.

The option -p (phase) tells shipper to commands for some phases only. The phases are:

The -w option is equivalent to "-p web". Any -p option overrides -w.

Otherwise, shipper then generates commands to upload deliverables. The -x may be used to suppress delivery to specified destinations that would normally be performed (e.g., due to a destination variable set in one of your .shipper files); it may be a comma-separated list.

After deliverables have shipped successfully shipper determines if the directory it is in is a working copy in a version-control system that supports release tagging. If so, and the tag_template variable is set to a nonempty value, it generates commands to tag the release just shipped. See the section called “After Uploads” for discussion of what version-control systems are supported and what actions will be taken.

Note that shipper makes one important assumption about the structure of your website(s). Beneath each directory in your destinations list, there will be one subdirectory for each project, with the directory leaf name being the same as the project. Thus, for example, if you have three projects named ruby, diamond and sapphire, and your personal site is at gemstones.net:/public/www/precious/, shipper will expect to be able to drop deliverables in three directories gemstones.net:/public/www/precious/ruby, gemstones.net:/public/www/precious/diamond/, and gemstones.net:/public/www/precious/sapphire/. Note that shipper will not create these project directories for you if they're missing; this is deliberate, so that uploads to sites that are not prepared for them will fail noisily.

After generating commands to do uploads, shipper may generate additional commands to perform project release notifications to freshcode.com, email lists, and IRC channels.

Configuration in More Detail

The behavior of shipper depends on a handful of internal variables. Some of these variables have defaults computed at startup time. All can be set or overridden in the per-user ~/.config/shipper and ~/.shipper files, and overridden again in any per-project control file, specfile, or .shipper file. All .shipper files are Python code and the syntax of variable settings is Python's; control and RPM files are parsed according to their native conventions.

If a variable is set in a config file, that value is locked in (except for the destinations variable which can be appended to from a specfile, see below) Variables that are not set in a config file may be set by the values of fields in your project specfile.

Any variable, locked or not, can be overidden from the command line by an argument of the form "name=value".

For basic use, it is only necessary to set one variable in your ~/.config/shipper or ~/.shipper file: destinations, the list of destinations to ship to. Normally you'll set this globally, pointing all your projects at your main distribution website and public drop sites. It is also possible to add destinations on a per-project basis by giving a comma-separated list in a #Destinations: comment in one of the specfile. Finally, you can set the variable in a per-project .shipper to ignore your global destination list.

Here are the configuration variables you will normally set in your ~/.config/shipper or ~/.shipper file:

Here are the variables you will normally set in a project control file or git config:

There is one variable you will normally set as a shipper command-line option:

Finally, these variables are computed and available for substitution:

All these variables are available for substitution at the time a web page or email announcement is generated. In general, any variable you set in your ~/.shipper file will be available at the time the web page or email announcement is generated. Use the Python "%(variable)s" syntax, not shell-substitution syntax.

Finding Local Deliverables

The following files are considered stock deliverables and may be shipped if they are present when shipper starts up:

Stock deliverables which are textfiles and contain the string "shipper: ignore this." will not be shipped by the generated script. This is especially useful for archival GNU-style ChangeLogs, which you may want to avoid shipping but which need to be present to pacify autotools.

Additionally, if the html_template and html_target variables are set, a templated HTML page named by it will be composed and shipped to websites by the generated script.

In the resource-table element of the generated web page, shipper generates explanatory text to go with each deliverable. For web pages, it tries to extract the contents of the title element. Anything else gets a canned description of its type.

Shipping to Destinations

In operation, shipper walks through a list of destinations, finding the required deliverables for each one and generating the required shipping actions to push them out to the destination. Here are the destination types shipper knows about:

informational files for a remote Gitweb repository may also be shipped at this stage. These are description and README.html, and are generated on the theory that a gitolite instance is likely to have a Gitweb view as well.

For this update to work, the repository must have the writer-is-owner option set. Additionally, in .gitolite.rc on the server, the readme must be uncommented.

After Uploads

After upload commands have been generated, shipper looks at where it is running. If it can recognize that the current directory is under version control, and the tag_template option has been enabled, it will generate additional commands to tag the just-shipped tree as an external release.

Currently, the version-control systems supported for tagging after shipping are Subversion, git, hg, and bzr. Due to variations in tagging behavior in these systems, this feature behaves slightly differently depending on which one is in use.

If the run is in the trunk of a Subversion repository (that is, the current directory is name trunk and has a subdirectory named .svn) and there is a tags peer directory, and the tags directory does not have a subdirectory named for the release version, then shipper generates commands to copy and tags the contents of trunk.

If it is in a git, hg, or bzr repository (that is, there is a subdirectory named .git, .bzr, or .hg) it will generare commands to tag the release. If there is a remote repo to be pushed to, local changes and tags wil be pushed. In all three of these cases the tag attempt will fail if the tag previously existed. See also the discussion of the tagtemplate variable.

After tagging, commands to ship email notifications are generated.

The last command generated, if applicable, will be the IRC and Twitter broadcasts.

Configuring Twitter broadcasts

To enable Twitter broadcasters, you first need to get developer access on Twitter, then register an app. It doesn't matter what you call the app; what you need are the four credentials that Twitter calls an API key, an API secret key, an access token, and an access token secret.

Once you have these, put them in your shipper config file by setting for variables like this (with the actual values filled in):

twitter_consumer_key="AAAAAAAAAAAAAAAAAAAAAAAAAAAAA" # (API key)
twitter_consumer_secret="BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"
twitter_access_token="CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC"
twitter_access_token_secret="DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD"

Then make sure the Twython library is installed on your system. The commnd "sudo pip3 install twython" should do this. Once this is set up, shipper will generate tweets with the same subsitutions it uses for IRC notifications: the relevant config variable is "Instant-Message".

Twitter broacasting is performed only when shipper is passed the -i option. This is not intended to be used manually; rater, it's there so that another instance can generate a "shipper -i" command as the last part of a series of shipping and notification actions.

Finally, install gitlab-igor if you intend to use the -g option to park your release deliverables on GitLab.

NEWS/HISTORY/ChangeLog format

In order to generate its various kinds of announcements, shipper needs to extract the version number and relase notes particular to the current release. In order to accomplish this, it tries to parse your NEWS, HISTORY, or ChangeLog files, in that order. It also recognizes these with the subset .adoc in case they are in asciidoc markup in conformance with GitHub and GitLab conventions

The file is analyzed as a list of stanzas - continuous stretches of nonblank lines separated by blank lines. If the first line of the first stanza begins with a digit, it is selected. Otherwise the first stanza is assumed to be a title header and skipped; then the second stanza is selected.

The first line of the selected stanza is examined. If it begins with a decimal digit, the leading part of the line consisting of decimal digits and periods is interpreted as the version number of the release. (This will be overridden if the invocation of shipper specifies a version.)

That line is then scanned for a date of the form yyyy-mm-dd. If no such date is found, or the date does not match today's date in local time, shipper aborts.

The first line of the selected stanza is discarded. All subsequent lines are treated as release notes.

These rules match conventions that are very widely used in NEWS files. If you normally use a different format, you can play nicely with both human viewers and the parser by adding a title line and following stanza with a first line something like “Recent Changes”.

Command-line Options

The -d option generates a configuration dump to standard output.

The -x option specifies a destination, or a comma-separated list of destinations, not to ship to. This overrides the setting of the destinations variable by shippper config files and control or spec files.

The -v option makes shipper chatty about what it's doing. The -h option prints a usage message and exits.

The -n option disables normal evaluation of the user's profile under $HOME in faver of a profile specified as the argument. It is intended to support regression testing of shipper and is not likely to be of general interest.

Arguments, if present, must be of the form name=value and can be used to override earlier settings from your profiles. The most frequently useful override will probably be "version=".

Hints and Tips

The following variable definition in your makefile will ensure that the makefile version is derived from (and thus always consistent with) the specfile version.

VERSION=$(shell sed <*.spec control -n -e '/Version: \(.*\)/s//\1/p')

A makefile production like the following will allow you to type make release and be sure that all the deliverables shipper knows about will be rebuilt before being shipped.

release: project-$(VERS).tar.gz project-$(VERSION).md5 index.html
	shipper version=$(VERS) | sh -e -x

The -e option is recommended to make all errors fatal. This way if an upload fails for some reason, the later notifications won't happen. The -x option makes the script echo its commands as they're performed.

You will want to change project to your project name. You will need to write a production that builds your tarball and checksum files.

It is also a good idea to have a similar production that just refreshes web pages and other public metadata without shipping tarballs or touching repositories.

refresh: index.html
	shipper -N -w version=$(VERS) | sh -e -x

In an SConstruct file, you may want something like this:

version = version

and something like

env.Alias("release", [tarball], 'shipper version=%s | sh -e -x' % version)

with 'tarball' being a production that makes a release tarball (and checksum file, if desired).

Requirements

This program requires the following helpers for shipping to specific destination types:

The first four must be on your executable $PATH at the time this program is invoked, or shipping to the relevant destination types will fail. If the Twython library is not importable, shipping announcements to Twitter will fail silently.

EXAMPLES

This is most of my .config/shipper file. I have omitted a bulky customized web page template.

# Set defaults for shipper(1)

# Identify myself
whoami = "Eric S. Raymond <esr@thyrsus.com>"

# My ID on patreon
patreon_id = "esr"

# By default, ship to my primary website
destinations = ["login.ibiblio.org:/public/html/catb/esr/"]

# Customize date format
date = time.strftime("%d %b %Y")

Here is the control file for the shipper project:

# This is not a real Debian control file, though the syntax is compatible.
# It's project metadata for the shipper tool

Package: shipper

Description: Automated shipping of open-source project releases.
 shipper is a power distribution tool for developers with multiple
 projects who do frequent releases.  It automates the tedious process
 of shipping a software release and (if desired) templating a project
 web page. It can deliver releases in correct form to SourceForge,
 and Savannah, and knows how to post a release announcement via
 freecode-submit.

Homepage: http://www.catb.org/~esr/shipper

XBS-IRC-Channel: irc://chat.freenode.net/shipper

XBS-Logo: shipper-logo.png

#XBS-Project-Tags: packaging, distribution

XBS-Validate: make check

And here is the shipper config section in its repository:

[shipper]
	destinations =  mailto:esr@thyrsus.com
	htmltarget = index.html
	repositoryurl = http://thyrsus.com/gitweb/?p=shipper.git
	developerclone = git@thyrsus.com:shipper
	anonymousclone = git://thyrsus.com/repositories/shipper.git
	tagtemplate = %(version)s

Note that by putting these entries in the git configuration rather than the control file you make it possible to (a) hide potentially senstive email addresses, and (b) rehost the project repository without having to ship a release to update the control-file contents. Instead, just update these entries and use shipper to regenerate the displayed URLs on the project web page.

This shipper project does not have a .shipper in its directory.

Bugs

The program cannot update Savannah webspace, which is only accessible through a CVS checkin procedure.

The program can put downloadables in place on SourceForge, but not set the "Preferred Download". Usually this will point to the most recent download subdirectory by default, which is the right thing.

The shipper framework is extensible and it is relatively easy to add new destination types and new deliverables; it is a bug that we do not yet support all major forge sites as destinations.

Files

$HOME/.shipper; $HOME/.config/shipper; $PWD/.shipper. $PWD/.git/config.

Author

Eric S. Raymond <esr@thyrsus.com>.

There is a project web page at http://www.catb.org/~esr/shipper/.

See Also

freecode-submit(1), lftp(1), scp(1), ssh(1).