comparator(1) Manual Page

comparator works by first chopping the specified trees into overlapping shreds (by default 3 lines long) and computing a hash of each shred. The resulting list of shreds is examined and all unique hashes are thrown out. Then comparator generates a report listing all cliques of lines with duplicate hashes, with overlapping ranges merged. The output format can (not by coincidence) be stepped through with the next-error function of Emacs.

A consequence of the method is that comparator will find common code segments wherever they are. It is insensitive to rearrangements of the source trees. With appropriate options, code can be normalized at input to ignore differences in whitespace and (in C code) brace placement. An option to strip comments also exists.

The program is capable of generating a hash list for a source code tree which can be used in place of the tree itself. Thus, it is possible to do comparisons with source trees without having access to the actual source code, providing someone who has access is willing to ship you a hash list. These hash lists are called 'SCF files' and made with the extension .scf; the name stands for Source Comparison Format.

In its normal mode of operation, comparator expects two or more command-line arguments which are the root directories of the source-code trees to be compared, and generates a report to standard output. Progress messages are emitted to standard error. Some arguments may be the names of SCF files.

When given a single path argument which is a tree, the program generates an SCF hash list to standard output.

When the -c option is specified, the program generates a SCF file from each tree specified on the command line. The name of the resulting file is the argument name with .scf appended.

The -o option directs output to a specified file. This may be used with -c to override the normal .scf convention, or simply as an alternative to output redirection when generating a final report. One advantage of -o is that the program will not create the output file until it is ready to write; thus, unlike shell redirects, it will not leave an empty file lying around if it aborts early.

The -d option changes current directory to the specified tree before walking down each argument path to generate hashes. This will be useful if you want to generate a report for trees in a directory other than your current, but want the filenames in the report to be relative.

The -s option changes the shred size. Smaller shred sizes are more sensitive to small code duplications, but produce correspondingly noisier output. Larger ones will suppress both noise and small similarities.

The -m option sets the minimum-sized span of lines that will be output. By default this is zero; setting it to a value higher than the shred size will eliminate a lot of junk from the report. These are two separate parameters because shred size controls how likely common segments are to not get recognized because of unshared things near them, while minimum span size defines the smallest features we want to see in the output report.

Normally, comparator performs significance filtering before emitting a span into the common-segment report. The -n option suppresses this, emitting all common spans. See the discussion of significance filtering below.

The -v option enables progress and timing messages to standard error.

The -x option enables some debugging output. It is for developers; if you need to understand it, read the code.

comparator uses a heuristic for figuring out which files in a tree are eligible to be compared. Files with .c and .h extensions are always eligible. Files with .o or ~ extensions are always ignored. CVS, RCS, SCCS, and .svn directories are ignored. Other files are checked to see if the percentage of printable characters near the front of the file is above 90%.

The -N option passes in its argument as a normalization specification. A normalization specification consists of a comma-separated list of tokens. The first token names a normalizer, e.g., a set of rules for transforming code into a canonical form. Subsequent tokens (if present) are options to the normalizer.

Normally, comparator tries to throw out common segments that are just noise. A segment is considered noise if one of the following conditions holds:

Significance filtering can be disabled with the -n option.

filterator is a postprocessor for the SCF-B output of comparator that produces an actual code listing.

The -d option acts as it does for comparator.

The -m option sets the minimum size of overlap to be reported. Setting this to a slightly larger number than the comparator shred size is a useful way to filter out spurious matches; thus, while comparator defaults to a shred size of 3, filterator defaults to a minimum size of 5.

The -f option takes a Python regexp as argument and throws out all cliques for which no associated filename matches the regexp. This might be useful if you want to see a common-segment report only for files with a particular extension such as .h or .S.

The -F option takes an argument which is a file containing a list of filenames. Any clique that does not contain one of these filenames is discarded.

The -n option suppresses code listing, so that an SCF-B output is produced instead. You can use this to filter common-segment reports with -m, -f, and -F options of the tool and then hand-filter them for junk before generating a listing.

Each filterator line beginning with % is the filename from which the following common span of lines is displayed. filterator displays each span of lines, unnormalized, from the first file in the clique of matches that it can find and open.

Occasionally the filterator output looks as though the code has failed to properly merge two or more overlapping line ranges. When this happens, the ranges have different match sets; they may overlap in one tree, but not in others. A clue to this is the number of matches reported for each range.

comparator.py is a Python module that parses and interprets the format emitted by comparator. It can be used to write report generators. The filterator program is an example; it is a trivial wrapper around comparator.py.