Hacker’s Guide to Autodafe

For details on how to apply these tools, see the De-Autoconfiscation HOWTO.

Prerequisites

To run this software you will need Python 3.

The deconfig tool needs to be able to call unifdef(1)

To format the documentation you will need asciidoctor.

To validate the code you will need a POSIX-compatible shell, make, pylint and shellcheck.

To test it, you will need to have autotools installed.

Goals and philosophy

The proximate goal of this project is to eliminate the piles of intermediate products and scripts that autotools from being a jungle in which exploits like the xz crack of 2024 can hide.

The wider goal is to abolish the festering complexity sink that autotools recipes have become.

The tools will transform your source tree and autotools build to a simpler one using bare makefiles. There are two goals for this tranformation: (1) it must be correctness-preserving, and (2) it must improve the readability of the product makefile. (1) is more important than (2).

Another goal is for the Makefile to be a single point of truth for the build recipe. Everything lives there, not just the make rules but the configuration checks as well. This design rule is direct from experiance with the xz crack; we want to make it as difficult as possible for supply-chain attackers to hide what they are doing.

This is why makemake undoes the autotools hack of generating .c.o dependency rules into .deps and including them dynamically. It’s cute, but it’s also spooky action at a distance ind violates SPOT.

There is a limited exception to the SPOT rule in that CHECK_SCRIPT can call an outboard Python script. This is unpleasant but necessary, because no fixed repertoire of test types could possibly capture te bizarre complexities of the gnarliest configure.ac files. We expect use of this feature to be rare across the general run of projects.

Patches to improve the readability of what makemake produces from its input are always welcome. But always bear in mind that potential users need to have absolute confidence that it won’t break their production build.

We emphasize readability because the intimidating complexity of auditing an autotools build is what provided cover for the xz hack. Transparency leads to quality and security.

Another property we want to preserve is idempotency. That is, if you run any one version of makemake on the output of makemake, the result is unchanged. (Of course later versions of makemake may perform better complexity reduction than earlier ones.) Idempotency reduces the friction cost of using the tool.

Problems

The Makefile outputs of autotools were not meant for human eyes. Reducing them to a more readable form is tricky. They contain lots of clutter of various types, including:

Internal autototools machinery meant to support templating by autotools so it can simply set make macros. All this stuff should be removed by default.
Bulky generic utility productions like "clean", "dist", "check" and "install". We can’t remove this stuff by default without breaking correctness, but we can give the user easy options to remove it so they can make simpler custom productions.

There is still work to be done in both areas. Patches are welcome.

Testing

If you decide to try to improve these tools, there are two scripts you should know about in the tests directory: test_deconfig and test_makemake. You can point these at a pristine source tree amd they will do all the autotools grinding required to fully configure a copy of the source tree it before running the tools on it.

The normal behavior of test_makemake is to run an autotools buld, then run deconfig and makemake to de-autoconfiscate the build and do it again, demonstating that the transformed build still works.

An important thing to know, though, is that you can stop their parts of the build sequence at any point by passing the right command-line option to test_deconfig or test_makemake. You can then examine the intermediate products (the patch and config.mk generated by deconig) and run the tools for the remaining stages yourself with debugging turned up.

Assumptions

This section describes assumptions that are not made explicit in the code. They are documented here to enlighted people interested in working on the code, and so they can be reviwed for correctness. corrections are welcome.

The makefiles generated by autotools consist of a list of configuration and user-declard macros followed by boilerplate make productions. The boilerplate begins with the production for "all".

All the platform dependencies are expressed in the macros, not the boilerplate.

If you have enough of a shell environment to run the generated utility predictions at all, you have a full set of POSIX-complient shell tools. Thus, the layer of indirection expressed by config macros like $(AR), $(STRIP) etcetera is now pointless clutter sand has been removed. If for some odd reason you need to redirect one of these commands, containers are your friend.

Configuration

The following macro definition may be generated, in the order listed, at the beginning of every autotools-generated Makefile; makemake reorders them, but this is a reference given in case you have to unnderstand the raw thing. Additional config macros may be written depending on the tests in your configure.ac file.

If the description begins with an asterisk, the macro occurs only in internal automake productions that makemake removes. If it begins with a plus-sign, it is a primary configuration macro that you might well need to tweak.

Most build recipes will set only a relatively small subset of these. De-obfuscation of your generated Makefile removes unused definitions.

pkgdatadir: Package data directory. The project name is generated into it.
pkglibdir: Package library directory. The project name is generated into it.
pkgincludedir: Package library directory (where its .h files should go). The project name is generated into it.
am__cd: *This is not a configuration macro and should not be overridden. A variant of the cd command that avoids writing to standard output when CDPATH is set in the environment. A suitable command is spliced in directly.
install_sh_DATA, install_sh_PROGRAM, install_sh_SCRIPT: *These are not a configuration macros and should not be overridden. Autotools ariants of the install program with options for installing data files (including documentation) programs, and scripts.
INSTALL_HEADER: *This is not a configuration macro and should not be overridden. It is a variant of the install program used to install .h files.
transform: Involved in some cryptic magic when installing/uninstalling prograns. See program_transform_name
NORMAL_INSTALL, PRE_INSTALL, POST_INSTALL, NORMAL_UNINSTALL, PRE_UNINSTALL, POST_UNINSTALL: These macros allow you to install script hooks at various places in generated installation productions. The default hooks do nothing.
build_triplet, host_triplet: *Strings describing the host enviroment (where the build toolchain lives) and where it is being built for. A representative value for both is "x86_64-unknown-linux-gnu" with the "unknown" part being an (unspecified) vendor ID. This is not used in autotools-generated code (presumably because the information it used is already baked into your compiler) but is available for the user’s custom scripts. It is flagged as a portability problem, and makemake composes it from the components autotools generates.
subdir: This is not a configuration macro and should not be overridden. In a build system with multiple Makefile.am files in a toplevel directory annd subdirectiries, this macro tells the generated Makefile which diretory it is in. Altering this would proably mess up various generated utility productions/
DIST_COMMON: This is not a configuration macro and should not be overridden. Names of common files whuch should be included in a distribution tarball if present: AUTHORS COPYING ChangeLog INSTALL NEWS TODO, according to GNU conventions. Before translation this included the names of several autotools intermediate products.
ACLOCAL_M4: *Location of an autotools script.
amaclocal_m4_deps, amconfigure_deps, am__CONFIG_DISTCLEAN_FILES: More autotools debris. These are discarded and wll not appear in your debfuscated Makefile.
mkinstalldirs: Expands to a script command for installing directories.
CONFIG_HEADER: Location of the autotools config headr, normally config.h.
CONFIG_CLEAN_FILES: More autotools debris. These are discarded and wll not appear in your debfuscated Makefile.
SOURCES: This is not a configuration macro and should not be overidden. Autitools generates it to agrgegate all the source-code files from the SOURCES macros in your MAkefile.am section
DIST_SOURCES: This is not a configuration macro and should not be overidden. It is like SOURCES but specifically used in utitlity productions for making tarballs.
RECURSIVE_TARGETS: This is not a configuration macro and should not be overidden. It aggregates all of the recursive utility targets generated in the autotools boilerplate related to building, installation, and unistallation.
RECURSIVE_CLEAN_TARGETS: This is not a configuration macro and should not be overridden. It aggregates all of the recursive utility targets generted in the autotools boilerplate related to cleaning.
ETAGS, CTAGS: The names of binaries used to produce elisp and C tag files Used in generated utility productions.
DIST_SUBDIRS: This is not a configuration macro and should not be overridden. Where to look for productions as part of dostribution making.
DISTFILES: This is not a configuration macro and should not be overidden. Aggregates all filenames that should be includes in a distribution tarball.
distdir: This is not a configuration macro and should not be overidden. It’s the name, composed from PACKAGE and VERSION, of the directory to buiuld a distribution in.
top_distdir: This is not a configuration macro and should not be overidden. Unlike distdir it is not relative to the directory the makefile the definition is in, but refers to the toplevel directory of the project tree.
am__remove_distdir: This is not a configuration macro and should not be overidden. It expands to a tricky bit of shell hackery used for deleting a distribution directory.
DIST_ARCHIVES: Names of compressed archives to build when making distribution tarballs.
GZIP_ENV: Quality option to pass gzip when making compressed archives.
distuninstallcheck_listfiles: This is not a configuration macro and should not be overidden. It expands to a shell cimmand used to list files in a source tree for purposes of performing uninstalls.
distcleancheck_listfiles: This is not a configuration macro and should not be overidden. It expands to a shell cimmand used to list files in a source tree for purposes of performing cleanung
ACLOCAL: *An internal autotools commabd that will not appear in your de-obfuscated Makefile.
AMTAR: *An internal autotools commabd that will not appear in your de-obfuscated Makefile.
AR: *Name of the archiver used for making static libraries. Won’t appear in your de-obfuscated Makefile as we assume a POSIX environment including ar(1).
AUTOCONF, AUTOHEADER, AUTOMAKE: *Configured-in locations of various autotools scripts.
AWK: *Name of the AWK to use in utility productions. We assume a POSIX-compatible awk. Note tgat historicall mawk was assumed here which is not portable.
CC: +Name of the C compiler
CCDEPMODE: Mode switch used internally in some proctions related to Cygwin. All references are commented out in Unix builds.
CFLAGS: +Flags to be passed to the C compiler
CPP: How to invoke the C preprocessor.
CPPFLAGS: Flags to pass the C preprocessor.
CXX: Name of the C+ compiler.
CXXCPP: How to invoke the C++ compier’s preprocessor/.
CXXDEPMODE: All references are commented out in Unix builds. Mode switch used internally in some proctions related to Cygwin.
CXXFLAGS: + Flags to pass the C++
CYGPATH_W: Cygwin version of echo. Typically is "echo". All references are commented out in Unix builds.
DEFS: *This is not a configuration macro and should not be overidden. It’s used to pass b-DHAVE_CONFIG_H to the C compiker. It will not appear in a de-obfuscated Makefile unless it has been set to a non-default value.
DEPDIR: The name of the directory used for .Po and related files in a build
ECHO: *The echo(1) program, which is in POSIX. Uses do not assume that -n works, it isn’t.
ECHO_C: *This is not a configuration macro and should not be overidden. It is an internal autotools hack, an equivalent of echo(1) that processes control characters (not a POSIX featuew). Lacking any alternative we map it to GNU coreutils printf(1) and cross our fingers. ECHO_T is similarly murky. We’ll flag the printf expansion as a portability issue.
ECHO_N: *This is not a configuration macro and should not be overidden. echo -n isn’t portable, but as with mkdir -p there has never been any other way to try to get this behavior from echo(1) so the indirection is pointless clutter. Best we can do is perform the substitution, then flag it as a portability problem if it ever actuaklly occurs.
ECHO_T: *This is not a configuration macro and should not be overidden. See ECHO_C
EGREP: *Name of a program that produces egrep(1) behavior. The -E option is supported by POSIX grep(1).
EXEEXT: The extension used for object file: .o or ,obj,
F77: The name of a FORTRAB 77 compiler.
FFLAGS: Flags for FORTAN 77 invocations.
GREP: *Name of a program that produces grep(1) behavior. But grep(1) is in POSIX.
INSTALL, INSTALL_DATA, INSTALL_PROGRAM, INSTALL_SCRIPT, INSTALL_STRIP_PROGRAM: *Various places autotools generated installation commands for templating into the generated Makefile.
LDFLAGS: Flags to pass the linker.
LIBOBJS: This is not a configuration macro and should not be overidden. It is a conventional make macro that aggregates .o files for a library.
LIBS: This is a conventional make macro that aggregates libraries files for building a binary.
LIBTOOL: This is not a configuration macro and should not be overidden. It’s the formula for running libtool to build shared libraries

LTLIBOBJS = This is not a configuration macro and should not be overidden. It aggergaes shateable object files (normally with the extension .lo) to be passed to libtool command line.

MAKEINFO: An absolute path to a mskinfo interpeter. This is flagged as a portability problem if it occurs.
MKDIR_P: *The -p option of mkdir(1) is in POSIX.
OBJEXT: The extension for object files; .o on Unix, .obj on Windows.
PACKAGE: The name of the package. You will inerit this because it was set in autotools.
PACKAGE_BUGREPORT: Email address of the package maintainer.
PACKAGE_NAME: *In autotools this was a redundant copy of PACKAGE. Your de-obfuscated Makefile will not cotain it.
PACKAGE_STRING: *In autotools this was configured as if it were $(PACKAGE) $(VERSION). Your de-obfuscated Makefile will not contain it.
PACKAGE_TARNAME: In autotools this was configured as though it were $(PACkAGE); makemake does the text substitution, though you can still set it to something else.
PACKAGE_VERSION: *In autotools this was a redundant copy of VERSION. Your de-obfuscated Makefile will not cotain it.
PATH_SEPARATOR: Separator used in PATH and CDPATH shell macros.
RANLIB: *The name of an old-school archive maker. Never had any value other than ranlib, so we use that.
SED: *sed(1) is in POSIX.
SET_MAKE: *This is not a configuration macro and should not be overidden. From the autoconf manual: "If the Make command, $MAKE if set or else 'make', predefines $(MAKE), define output macro SET_MAKE to be empty. Otherwise, define SET_MAKE to a macro definition that sets $(MAKE), such as "MAKE=make". Calls AC_SUBST for SET_MAKE". The will not
SHELL: The GNU Make manual advises that every Makefile should have a SHELL definision to /bin/sh in order to prevent unexpected behavior in older make implementations that import SHELL from the environment, which can be bad for the reproducibility of your builds. GNU make does not have this misfeature. We assume POSIX sh, not bash.
STRIP: *Name of he program used for stripping symbol tables and other data from binaries when installing them for production.
VERSION: +You must configure this.
XMKMF: *Name of the command used to render a makefile from an Imakefile.
X_CFLAGS, X_EXTRA_LIBS, X_LIBS, X_PRE_LIBS: Flags and libraries to include in build lines when code links to X support.
abs_builddir, abs_srcdir, abs_top_builddir, abs_top_srcdir: Absolute paths used for cross-compiling. These throw a warning if they are found in your translated makefile If that happens you will have to fix them up by hand.
ac_ct_CC, ac_ct_CXX, ac_ct_F77: *These are autconfonf-configured names for compilers.
am__include: *This is not a configuration macro and should not be overidden. It’s an automake macro that expands to the expected final segment for include directories.
amleading_dot, amquote: *These are not configuration macros and should not be overidden. They are random bits of automake debris that automake expands to their values.
amtar, amuntar: *Autotools generated these fragments of shell to make tar gommands.
bindir: Where the project should istall its binaries.
build: *Same as build_triplet, which see.
build_alias, build_cpu, build_os, build_vendor: + Build target identification strings. These are flagged as portability problems.
builddir: Where the building of objects and binaries should be done.
datadir: Where to insrall architecture-independent data used by your program. Usually the same as datarootdir, but could be set to some project-specific subdirectory.
datarootdir: The root directory for archirtecture-independent data shared among pprograms. Not specific to your project.
docdir: Where to install the project’s documentation.
dvidir: Where to put DVI files.
exec_prefix: Prefix to use to when composing directory names forexecutables to land in.
host: *Same as host_triplet, which see.
host_alias, host_cpu, host_os, host_vendor: + Host identification strings. These are flagged as portability problems.
htmldir: Where to install HTML files.
includedir: The include directory for .h files.
infodir: The installation directory for info files.
install_sh: *Location of an autotools script. We relace this with POSIX install(1).
libdir: Where the project should install libraries that can be linked to by other programs.
libexecdir: Directory forachitecture-independent executables meant to be called by programs rather than users.
localedir: Directory for locale files.
localstatedir: A directory where macro data files specific to a particular system or instance of a software package are stored. This directory typically contains data files that may change during the normal operation of the software, such as log files, temporary files, and run-time state information.
mandir: Installation directory for manual pages.
mkdir_p: *The -p option of mkdir(1) is in POSIX.
oldincludedir: A legacy location for installing .h include files. It would be very surprising id this were ever anyhhing but ${prefix}/include.
pdfdir: Where to install PDF files.
prefix: + Master prefix for constructing installation-directory names.
program_transform_name: Specify a sed(1) transformation that should be applied to the names of programs during the installation process. This transformation is typically used to modify the names of executable files to conform to a specific naming convention or to rename them for compatibility with the target platform’s naming conventions.
psdir: Where to put Postscript files.
sbindir: Where to install static binaries/
sharedstatedir: State directory shared across a networked file system common to multiple hosts.
srcdir: Where to find the sources for bulding the programs in the directory of the Makefile defining this macro.
sysconfdir: The system conficuration directory, typically /etc modified by the master installation prefix.
target_alias: Alias of the target system type for build purposes. If this value is present in your deobfuscated file it is flagged as a portability problem.
top_builddir: The top level of the build directory of this project. Won’t be the same as builddir if this makefile is in a subdirectory
top_srcdir: This is not a configuration macro and should not be overidden. The top level of the source directory of this project. Won’t be the same as srcdir if this makefile is in a subdirectory.

Envoi

The project name is officially pronounced "auto-da-fe". Because having an autotools build is very much like a state of sin. We will, however, look benignly on those who choose to pronounce it "autodaffy".

Yes, makemake is named after the minor planet Makemake, but it’s not pronounced that way. The project logo is the astrological symbol for Makemake.

References

Open Group Standards

GCC Common prefined macros