A long time ago, I ran a comparison of various command-line compressors in Linux. Recently, intrigued by the rise of parallel computing and the emergence of multi-processor versions of old *nix favorites like gzip and bzip2, I thought I’d give the benchmark another go.

The Setup

My machine is an Intel Core 2 Quad Q6600 [2.4Ghz] on a Gigabyte GA-N680SLI-DQ6 (nVidia 680i), with 4GB of Corsair XMS2(PC2 6400). The tests were run on a vanilla installation of Ubuntu 10.10 x64, fully updated. The compressors themselves were compiled by me from source code, so I would have the latest and greatest at the time of testing.

I did not use a RAM disk, since I wanted all of my available RAM available for compression (some of the tests used aggressive settings). I did use a freshly-formatted SATA disk for the purpose, however. For further information, see the methodology and results.

Timing was done with the GNU time command.

The Compressors

  • gzip. Perhaps the most well-known compressor on Linux, gzip is a DEFLATE-based compressor which offers moderate compression at a very fast speed. Note: I made the mistake of retrieving the sources from www.gzip.org, which has links to v1.2.4, from 1999. The most recent stable version is 1.4, from GNU’s servers. The version reflected in the benchmark, therefore, is out of date.
  • bzip2. An old favorite, bzip2 is a basic block-level compressor which is more aggressive than its cousin, gzip, at the cost of speed.
  • xz. A relatively new compressor, xz arose from the ashes of LZMAUtils, an implementation of the LZMA compression spec from Igor Pavlov’s 7-zip project. xz is is the command-line equivalent, and similar in usage to gzip and bzip2; more to the point, it’s been accepted by the GNU toolchain and now has its own filter in tar (-J).
  • rzip. Originally written by Andrew Tridgell (of Samba fame, and who indirectly caused git, when you think about it) as a doctoral project. It functions similarly to gzip and bzip2, according to its author, but attempts to take advantage of “long-distance redundancies” in larger files; that is, where bzip2 may only process 900K chunks at a time, rzip will seek to look beyond that in order to eek out additional compression.
  • lzop. One of the Lempel-Ziv-Oberhumer (LZO) compressors, it focuses on decompression speed rather than pure compression.
  • compress. Though not really a contender in any real zip, I included the old compress program for comparative purposes. Compress is a fast LZW compressor which creates .Z files, and can be triggered as a filter in tar with -Z. The implementation used is ncompress.
  • zip. An old standby, zip creates, well, .zip files, using the DEFLATE algorithm. One difference between zip and gzip is that zip creates a structured archive (no tar necessary).
  • lzip. Another LZMA-based compressor, lzip is heavily-asynchronous, emphasizing small decompression times at the expensive of larger initial compression time.
  • p7zip. The *nix version of Igor Pavlov’s 7-Zip for Windows, p7zip has a slew of options (including different compressors, such as ppmd), but we’ll be focusing on lzma2.
  • pigz. A parallel implementation of gzip. Like most parallel implementations, pigz sacrifices a small amount of compression efficiency for large gains in speed.
  • lbzip2. A parallel implementation of bzip2.
  • pbzip2. Another parallel implementation of bzip2.
  • quicklz (qpress). A very fast implementation of LZO, the QuickLZ library was at version 1.5.0 at the time of the benchmark, but its companion compressor program, QPress, was linked against 1.4.1.
  • lxz. A parallel implementation of the xz compressor. By the same creator as lbzip2. It supports compression only, so the benchmark will not contain decompression numbers for this tool. The author considers it a stopgap until xz gets multithreading support.
  • lrzip. Long ago, a fork of rzip, Con Kolivas’ lrzip has become a full-fledged compressor with multiple possible algorithms to choose from; the only one tested was LZMA. Note that the version tested here (0.571) was replaced shortly after I ran my benchmarks by v0.600, which was a large rewrite ostensibly improving decompression speed.
  • plzip. A straightfoward parallelization of the lzip compressor.
  • A note on options.

    Most of the compressors tested follow the same basic pattern for specifying compression strength, modeled after the options of gzip and bzip2. -# specifies compression strength, where # is a number. On some compressors, 0 is the lowest; on others, it’s 1. My approach was to test the lowest strength, the highest strength, and the default strength (either 5 or 6, depending).

    On some multithreaded compressors, it was necessary to specify the number of threads or processors to use. Where this was required, I use a value of 4, the number of cores in my processor.

    The Methodology

    Corpus. I downloaded the 11/3/2009 pages-articles.xml.bz2 from Wikipedia’s mirror an uncompressed it. 23+ GB. Uh oh. Realizing that I didn’t have all the time in the world to spend compressing things, I truncated the file to its first 1’073’741’824 bytes (1GB) using the truncate command.

    For each compressor/setting, the compression was run three times; the compressed size was recorded; then the decompression was run three times. Each run’s time was recorded, and the average and standard deviation was calculated for both compression and decompression.

    Note that in some cases, operation speed was limited by the speed of the hard drive (I’ll cover this in detail in the next section); there isn’t much to be done about this, other than the note it. Multiple runs should tease out inconsistencies and show where there is a real discrepancy as opposed to a consistent bottleneck.

    The Results

    The data table itself is far too large to fit in this template, but you can see it as Google Spreadsheet here.

    There’s no way to crown a clear winner, since all compressors mean tradeoffs. If I were as smart and capable as Werner Bergman, I could have a formula worked up for compressor efficiency, but I’m more curious to see the differences between single-threaded compressors and their multithreaded variants.

    To get it out of the way: xz at maximum compression had the best compression ratio of all tested configurations. At 243’350’864 bytes, it shrank the test corpus to 22.7% of its original size. This came at a heavy cost, though, as compression at this level took an average of 27 minutes, give or take 200 seconds.

    The fastest compression, by far, came from QuickLZ, whose lowest setting managed to process the entire 1GB file in an average of 6.3 seconds. Its maximum setting doubled the time to 12 seconds and shaved off an additional 75MB, but the price of such fantastic speed is much less compression: the maximum setting reduced our test corpus to only 44.4% of its original size.

    Remember that there is no best compressor; there is only a compressor that is best for your needs. Compression is all about diminishing returns: it takes a mere 12 seconds to get from 100% to 44% with QuickLZ. It takes an additional 27 minutes to get from 44% to 22% with xz.

    When looking at the decompression times, one might notice that the scores bottom out at between 12-15 seconds. My guess is this does not reflect the decompression speed at much as the write speed of the hard drive. In theory, one might see even lower numbers with a RAMdisk, but in practice, decompression speeds which outpace storage write speeds are unlikely to be taken advantage of outside a benchmark scenario.

    The numbers for bzip2 vary wildly, for reasons I haven’t figured out. I ran a second iteration of three tests for each setting and got similar numbers, so I stuck with my initial measurements. Other compressors don’t have such high standard deviations relative to their runtimes, so I don’t think it was a problem with my setup.

    Generally speaking, the parallel implementations of single-threaded compressors fare extremely well. Let’s look at the average times for some of the standard ones. Note that compression ratios are given as the percentage of the original filesize, not the percent reduction, so lower is better.

    Compressor Compression Time (s) Compression Ratio Decompression Time (s)
    bzip2 (min) 186.509 31.59% 63.342
    lbzip2 (min) 38.852 31.63% 18.520
    pbzip2 (min) 38.072 31.61% 13.533
    bzip2 (default) 169.474 28.14% 64.432
    lbzip2 (default) 42.085 28.15% 19.140
    pbzip2 (default) 40.698 28.33% 14.630
    bzip2 (max) 206.307 27.18% 66.460
    lbzip2 (max) 52.091 27.19% 23.092
    pbzip2 (max) 52.438 27.19% 20.617
    Compressor Compression Time (s) Compression Ratio Decompression Time (s)
    gzip (min) 36.146 40.09% 19.195
    pigz (min) 8.892 40.06% 12.199
    gzip (default) 82.452 34.65% 16.912
    pigz (default) 16.673 34.69% 12.852
    gzip (max) 134.889 34.24% 16.513
    pigz (max) 25.051 34.28% 12.129
    Compressor Compression Time (s) Compression Ratio Decompression Time (s)
    xz (min) 138.272 33.82% 50.916
    lxz (min) 29.890 33.80%
    xz (default) 1164.772 24.45% 34.114
    lxz (default) 39.997 24.96%
    xz (max) 1643.367 22.66% 31.968
    lxz (max) 699.268 23.07%

    As you can see, most parallel implementations decrease compression and decompression times by a factor of between 3 and 5, while taking, at most, a few tenths of a percent increase in final file size.

§7042 · May 3, 2011 · Tags: , , , ·

2 Comments to “Linux Command-Line Compressors Compared”

  1. Velumani Jaganathan says:

    It was a wonderful article on compression.
    I do have a question.
    What is the max file size supported by these utilities in Linux?
    Thanks in advance.


  2. Ben says:

    As far as I know, max file size is really determined by the filesystem, not the compressors themselves.

Leave a Reply