De-Autoconfiscation HOWTO

This is step 2.

The next step will use the generated config.h and the Makefile.

Your de-obfuscated file will support out-of-tree builds, but your conversion build needs to be in-tree in order to deal with an autotools trick. If you look in your Makefiike you will notice that there are no dependencies for individual .o files.

This is because the gcc command lines generated by autotools to make .o files from .c files use magic options to analyze each C file’s dependencies and dump them into a file with the same basename but an extension of .Po (or sometimes .Plo), in a hidden directory named ".deps"; then it generates include directives to pull in those hidden files.

While this is clever because it avoids anyone having to maintain those dependencies by hand, it conflicts with having a single auditable point of truth for the entire build. The makemake tool evaluates those includes and inlines the generated dependencies.

In order to do that, it needs the .Po/.Plo files to be generated by the pre-conversion build. Autotools itself has a complicated way to reconcile this with out-of-tree nuilds that makemake can’t replicate.

That’s why your conversion build needs to be in-tree. Afterwards you won’t have this problem and can do out-of-tree builds.

This is steps 3, 4, and 5.

Run deconfig against your code directory and inspect the resulting patch (you will need unifdef installed for this to work). It looks at the symbols in your config.h or config.h.in and identifies which ones are conditionalizing code that should always work on a modern C11//POSIX/SUSv2 system. Then it generates a patch that removes the unneeded code. Any <config.h> inclusions can be removed because of this cleanup is also removed.

If you see an unifdef error message during this step, your code is engaging in preprocessor abuse which you should patch out for the duration of the conversion at least. Here’s an example of code that triggers this:

The problem is following a preprocessor with a comment that has a continuation. This is allowed by the C standard; an issue has been filed with the unifdef maintainer.

Ahead of the patch it may also generate a list of guard symbols that it can’t discard; VERSION is likely to be one of these.

The patch will also create a file named config.mk containing all the residual definitions it could extract. Your de-obfuscated Makefiles will include it to replace what config.h formerly did.

The patch will probably also generate a "tests.mk" file containing specifications for tests to be performed by autodafe configure; this will be appended to your Makefile in a later step.

Don’t add tests.mk or config.mk to your repository. The reasons not to do this will become clear as you read more steps.

To do its job, deconfig relies on a large table of features (include-file names, function names, and type names) known to be in C11 and POSIX/SuSv2. This table is not complete, and you may see it report guard symbols as portability problems that it should not. These instances should be reported to the maintainers as bugs.

This is steps 6 and 7.

Now you need to go through the list of residual symbols from the previous step.

VERSION, PACKAGE, PACKAGE_TARNAME, PACKAGE_URL, and PACKAGE_BUGREPORT are special. If these macros occur in your code at all, you will probably want to move where they are set from config.mk to your base Makefile. Remember that in your new build flow config.mk will be clobbered and rewritten every time you run configure.

To do this, append to CFLAGS a -D option for each symbol, set from the corresponding Makefile macro.

It’s good to avoid perturbing your Makefile every time you ship a release. Here is one example of a way to do this:

What this does is extract VERSION from the top stanza in the NEWS file, assuming it has stanza headers like the NEWS.adoc in autodafe. You can modify the sed expression to fit your project’s conventions.

This means the project Makefile can be stable across releases, with your release stamp changing only when you update your NEWS file. If you don’t do this, you should set VERSION to something nonempty that is obviously not a version number so it will be conspicuous if you make a tarball without an intentional version.

If you have any residuals other than these, you should read the configure man page to find out how to write configuration tests.

Some residual symbols may want to be set from configure command-line options. Use CONFIG_ENABLE for this. Other residual symbols should be turned into tests that configure can run.

If you can’t figure out how to write your tests using the primitives that configure provides, write an outboard Python script to do them and call it with CHECK_SCRIPT.

You’re finished with this step when there are no more residual symbols. You can use deconfig to check this - just run it again; when it outputs nothing but a patch band that removes config.h, you have no residuals. Apply that patch to be done.

This is step 8.

To de-obfuscate your Makefiles, run this command in your top-level directory:

This will walk through your code tree looking for Makefiles, moving them so they are named Makefile.bak, and then filtering them and sending the output to Makefile.

The de-obfuscation transform is not very complex. It consists of (1) deleting things (macro definitions, and productions to make autotools intermediate products) that are no longer useful, and (2) selectively expending Make macros in the same way make does.

This is step 9.

After the preceding step, you have in theory severed your dependency on the autotools machinery. To know whether you have done so in practice, make clean followed by make; watch for compilation errors and run your normal test suite.

If your build fails with a link error, your local libtool instance might be broken. Run "makelibtool" in your build directory to generate a fresh instance, then patch your makefile to call "./libtool" rather than "libtool". Try your make again.

Any other failure at this point probably means makemake has a bug and you should file an issue with the maintainers.

If you succeed, this would be a good time to check in your changes.

This is steps 10 to 14.

Your objective in this step is to satisfy yourself that configure works.

If step 3 generated a test.mk, join a copy to the end of your new Makefile. This puts the test directives where configure can see them.

Save a copy of the config.mk that deconfig generated. It contains all the settings needed to make your build run. You’ll be able to delete the saved copy when this step is done.

Run configure. This will produce a new config.mk. Test your build with it.

This is steps 15 and 16.

You can check your new Makefile into your repository now.

To finally jettison autotools, run this command in your top-level directory.

This will remove all autotools scripts and intermediate-product files except your Makefiles (which you modified in the previous step). It will remove Makefile.bak files. It doesn’t make any further changes to your Makefile.

Then you will probably want to check a copy of configure into your repository.

It’s a good idea to test your build again after this step. You’re watching for hidden dependencies on missing intermediate products.

This is optional step 17.

If the -D options are not overwhelming your build logs, you can remove the remaining <config.h> includes from your code and you’re done.

Otherwise, add CHECK_CONFIG(config.h) to your Makefile to tell configure to generate a config.h that the existing config.h includes can use. When you do this, configure will no longer generate -D options into config.mk.

This isn’t the default behavior because it has a subtle long-term cost. It mean there are now two communication paths between your build recipe and your compilations rather than one, making the resulting flow more difficult to debug.

This is optional step 18.

After the previous step, you have ensured that no relative of the xz crack is lurking in the undergrowth of your build recipe. It now consists of Makefiles that are relatively easy to inspect.

But "relatively" is not "absolutely". The productions that autotools generates for utility targets such as "clean" and "dist" can be pretty grotty; you may want to replace them with more customized and readable equivalents.

Autotools generates a large number of boilerplate utility productions for you; dist, clean, install, and uninstall are the most important of these.

While this is convenient, the generated code is bulky and ugly, making your de-obfuscated makefile more difficult to read. Moreover, it is exactly the kind of thicket in which a clever attacker might seek to hide exploit hooks.

Therefore, it is probably best if you follow up on your conversion by removing these productions and writing simpler custom ones. You’ll get some help from makemake. The -x and -e options are useful at this stage to simplify the Makefile.

You can use makemake -x as a filter to strip out targets with names that match a specified regular expression. The -x option takes a |-separated list of regular expressions. Huge amounts of ugly boilerplate disappear with, for example, "-x \'dist.|clean.|install.|uninstall.'".

Passing the empty string to makemake -x is special; it removes all autotools boilerplate (dist, clean, install, dicumentation builders, tags and cscope productions) except what is required to support the default "make all" build of the software.

You don’t have to use -x on your first conversion. De-obfuscation is idempotent - makemake will do nothing to a de-obfuscated file unless you specify an -x option. (You may also see changes if you run a later version of makemake than the one you did your conversion with.)

The advantage of makemake -x over simply removing them after de-obfuscation is that if any macros become unreferenced due to removal, those settings will be removed as well.

With the -e option, you can eliminate much of the clutter produced by the autotools templating code, which generates lots of trivial macro assignments and references

Here is a use of makemake -e that specializes a Makefile for Unix rather than leaving file extensions as macros to be templated in:

See also the FAQ on cross-platform portability at the end of this document.

This is step 19.

You no longer have autotools as a build dependency. Document that.

If you’re building shared libraries you have a build dependency on standalone libtool instead. Document that.

If your installation instructions say to run autogen.sh or any other pre-configure script mean to set up autotools, you should delete that from your build documentation.

Autotools was written in the shadow of the Unix wars of the 1980s, when even small projects might need to have dozens of conditional-code sections to deal with quirks in the C compilers and the divergent Unix APIs of the time. The main reason going back to a simpler build flow is viable is because standardization worked. Modern Unixes, even the minor ones, are highly compliant with C99 and POSIX/SUSv2, and those standards have pretty good coverage of what programmers usually want to do.

This means that many autotools-using projects are using it to hammer down problems that have mostly disappeared out from under them. The expected number of -D options you need to pass to your compiler has plummeted.

The best way to find out if this is true for your project is to try it. The autodafe tools are designed to make experimenting relatively painless and easy to revert. You might be surprised at how small your config.mk ends up being.

You’ll lose your build-time dependency on autotools.

If you are building shared libraries you will keep a build-time dependency on libtool.

Once you’ve done your conversion, you won’t need deconfig and makemake again.

If your code is pure POSIX/SUSv2 and you have no configuration options, that’s it. Otherwise you will have a build-time dependency on autodafe’s configure utility, and through that on Python 3.

Recommended practice is for you to check a copy of autodafe configure into your repository, replacing the autotools configure script you nay formerly have had there.

From the point of view of people building your software little will change unless you started with your build recipe with autogen.sh; that’s gone. The normal build sequence is stll "./configure; make".

The syntax for configure --enable options is slightly different. Otherwise the only change people building your software are likely to notice is that configure suddenly runs to completion much sooner and doesn’t spam them with progress messages.

I have been asked this question on X about cmake, mkmk, meson, scons, bazel, and doubtless others I have forgotten. The answers for all of them are the same.

Anyone who is moved to respond to the previous complaint with "Try my favorite build system you haven’t mentioned! It is really super-duper keen and I am sure you will love it and bow before its awesomeness!" is advised against saying this to me face-to-face when I am within reach of an object I could club you to death with.

In that case, deconfig and makemake won’t help you at all - but autodafe configure might.

The difference is that deconfig and makemake are very specific to the problem of disentanging a project from autotools. The configure tool isn’t; it might serve you well if you need a fast, simple configuration system.

Probably not. Most of the platform-dependency details in your build are hidden within your local compiler/linker tools. There is an exception near file extensions.

In the Makefiles generated by autotools, platform dependencies are all expressed as the values of make macros at the front of the file.

One of these these is OBJEXT, the file extension for object files - ".o" on Unix including OS/X, ".obj" under Windows). Another is EXEEXT, the file extension for executables - empty on Unix, ".exe" under Windows). A third is SOEXT, the file extension for shared libraries; ".so" on most Unixes, ".dynlib" on OS/X, ".dll" on Windows.

If you’re building binaries or shared libraries, all that should be strictly necessary is for you to set these macros to tell the generated compilation commands to do the right thing. Normally configure sets these for you.

Yes - makemake produces makefiles that should operate normally even if no deconfig patch has been applied.

Specifically: when you are running makemake against a directory, it will only strip out Makefile references to config.h and -DHAVE_CONFIG_H if it sees no config.h file in the directory top level.

One minor exception: if you are building shared libraries, be aware that makemake always changes your Makefile to look for libtool as an executable in your normal $PATH rather than in the top level of your build directory. You can easily patch your Makefile to revert this.

Note: If you have previously run deconfig and it reported no residuals, the patch it generated deleted config.h. This is how deconfig passes the informatoion that makefile referebces to config.h and -DHAVE_CONFIG_H can be removed.

Usually because it isn’t needed anymore. Autotools was designed in a more primitive time for a range of far more variegated and hostile Unix environments than exist today.

The main design objective of autodafe is to prevent supply-chain attacks that could be inserted in an autotools build, as happened with the xz Trojan. While this doesn’t constrain the once-and-done conversion tools deconfig and makemake, it creates a lot of pressure to keep configure small and simple and auditable.

If there is a ${TERRIFYING_AUTOTOOLS_FEATURE} that you need for your build, you have a couple of different options: