A Modest Construct » benchmarks

Linux Command-Line Compressors Compared

Ben — Tue, 03 May 2011 13:22:31 +0000

A long time ago, I ran a comparison of various command-line compressors in Linux. Recently, intrigued by the rise of parallel computing and the emergence of multi-processor versions of old *nix favorites like gzip and bzip2, I thought I’d give the benchmark another go.

The Setup

My machine is an Intel Core 2 Quad Q6600 [2.4Ghz] on a Gigabyte GA-N680SLI-DQ6 (nVidia 680i), with 4GB of Corsair XMS2(PC2 6400). The tests were run on a vanilla installation of Ubuntu 10.10 x64, fully updated. The compressors themselves were compiled by me from source code, so I would have the latest and greatest at the time of testing.

I did not use a RAM disk, since I wanted all of my available RAM available for compression (some of the tests used aggressive settings). I did use a freshly-formatted SATA disk for the purpose, however. For further information, see the methodology and results.

Timing was done with the GNU time command.

The Compressors

gzip. Perhaps the most well-known compressor on Linux, gzip is a DEFLATE-based compressor which offers moderate compression at a very fast speed. Note: I made the mistake of retrieving the sources from www.gzip.org, which has links to v1.2.4, from 1999. The most recent stable version is 1.4, from GNU’s servers. The version reflected in the benchmark, therefore, is out of date.
bzip2. An old favorite, bzip2 is a basic block-level compressor which is more aggressive than its cousin, gzip, at the cost of speed.
xz. A relatively new compressor, xz arose from the ashes of LZMAUtils, an implementation of the LZMA compression spec from Igor Pavlov’s 7-zip project. xz is is the command-line equivalent, and similar in usage to gzip and bzip2; more to the point, it’s been accepted by the GNU toolchain and now has its own filter in tar (-J).
rzip. Originally written by Andrew Tridgell (of Samba fame, and who indirectly caused git, when you think about it) as a doctoral project. It functions similarly to gzip and bzip2, according to its author, but attempts to take advantage of “long-distance redundancies” in larger files; that is, where bzip2 may only process 900K chunks at a time, rzip will seek to look beyond that in order to eek out additional compression.
lzop. One of the Lempel-Ziv-Oberhumer (LZO) compressors, it focuses on decompression speed rather than pure compression.
compress. Though not really a contender in any real zip, I included the old compress program for comparative purposes. Compress is a fast LZW compressor which creates .Z files, and can be triggered as a filter in tar with -Z. The implementation used is ncompress.
zip. An old standby, zip creates, well, .zip files, using the DEFLATE algorithm. One difference between zip and gzip is that zip creates a structured archive (no tar necessary).
lzip. Another LZMA-based compressor, lzip is heavily-asynchronous, emphasizing small decompression times at the expensive of larger initial compression time.
p7zip. The *nix version of Igor Pavlov’s 7-Zip for Windows, p7zip has a slew of options (including different compressors, such as ppmd), but we’ll be focusing on lzma2.
pigz. A parallel implementation of gzip. Like most parallel implementations, pigz sacrifices a small amount of compression efficiency for large gains in speed.
lbzip2. A parallel implementation of bzip2.
pbzip2. Another parallel implementation of bzip2.
quicklz (qpress). A very fast implementation of LZO, the QuickLZ library was at version 1.5.0 at the time of the benchmark, but its companion compressor program, QPress, was linked against 1.4.1.
lxz. A parallel implementation of the xz compressor. By the same creator as lbzip2. It supports compression only, so the benchmark will not contain decompression numbers for this tool. The author considers it a stopgap until xz gets multithreading support.
lrzip. Long ago, a fork of rzip, Con Kolivas’ lrzip has become a full-fledged compressor with multiple possible algorithms to choose from; the only one tested was LZMA. Note that the version tested here (0.571) was replaced shortly after I ran my benchmarks by v0.600, which was a large rewrite ostensibly improving decompression speed.
plzip. A straightfoward parallelization of the lzip compressor.

A note on options.

Most of the compressors tested follow the same basic pattern for specifying compression strength, modeled after the options of gzip and bzip2. -# specifies compression strength, where # is a number. On some compressors, 0 is the lowest; on others, it’s 1. My approach was to test the lowest strength, the highest strength, and the default strength (either 5 or 6, depending).

On some multithreaded compressors, it was necessary to specify the number of threads or processors to use. Where this was required, I use a value of 4, the number of cores in my processor.

The Methodology

Corpus. I downloaded the 11/3/2009 pages-articles.xml.bz2 from Wikipedia’s mirror an uncompressed it. 23+ GB. Uh oh. Realizing that I didn’t have all the time in the world to spend compressing things, I truncated the file to its first 1’073’741’824 bytes (1GB) using the truncate command.

For each compressor/setting, the compression was run three times; the compressed size was recorded; then the decompression was run three times. Each run’s time was recorded, and the average and standard deviation was calculated for both compression and decompression.

Note that in some cases, operation speed was limited by the speed of the hard drive (I’ll cover this in detail in the next section); there isn’t much to be done about this, other than the note it. Multiple runs should tease out inconsistencies and show where there is a real discrepancy as opposed to a consistent bottleneck.

The Results

The data table itself is far too large to fit in this template, but you can see it as Google Spreadsheet here.

There’s no way to crown a clear winner, since all compressors mean tradeoffs. If I were as smart and capable as Werner Bergman, I could have a formula worked up for compressor efficiency, but I’m more curious to see the differences between single-threaded compressors and their multithreaded variants.

To get it out of the way: xz at maximum compression had the best compression ratio of all tested configurations. At 243’350’864 bytes, it shrank the test corpus to 22.7% of its original size. This came at a heavy cost, though, as compression at this level took an average of 27 minutes, give or take 200 seconds.

The fastest compression, by far, came from QuickLZ, whose lowest setting managed to process the entire 1GB file in an average of 6.3 seconds. Its maximum setting doubled the time to 12 seconds and shaved off an additional 75MB, but the price of such fantastic speed is much less compression: the maximum setting reduced our test corpus to only 44.4% of its original size.

Remember that there is no best compressor; there is only a compressor that is best for your needs. Compression is all about diminishing returns: it takes a mere 12 seconds to get from 100% to 44% with QuickLZ. It takes an additional 27 minutes to get from 44% to 22% with xz.

When looking at the decompression times, one might notice that the scores bottom out at between 12-15 seconds. My guess is this does not reflect the decompression speed at much as the write speed of the hard drive. In theory, one might see even lower numbers with a RAMdisk, but in practice, decompression speeds which outpace storage write speeds are unlikely to be taken advantage of outside a benchmark scenario.

The numbers for bzip2 vary wildly, for reasons I haven’t figured out. I ran a second iteration of three tests for each setting and got similar numbers, so I stuck with my initial measurements. Other compressors don’t have such high standard deviations relative to their runtimes, so I don’t think it was a problem with my setup.

Generally speaking, the parallel implementations of single-threaded compressors fare extremely well. Let’s look at the average times for some of the standard ones. Note that compression ratios are given as the percentage of the original filesize, not the percent reduction, so lower is better.

bzip2
Compressor	Compression Time (s)	Compression Ratio	Decompression Time (s)
bzip2 (min)	186.509	31.59%	63.342
lbzip2 (min)	38.852	31.63%	18.520
pbzip2 (min)	38.072	31.61%	13.533
bzip2 (default)	169.474	28.14%	64.432
lbzip2 (default)	42.085	28.15%	19.140
pbzip2 (default)	40.698	28.33%	14.630
bzip2 (max)	206.307	27.18%	66.460
lbzip2 (max)	52.091	27.19%	23.092
pbzip2 (max)	52.438	27.19%	20.617

gzip
Compressor	Compression Time (s)	Compression Ratio	Decompression Time (s)
gzip (min)	36.146	40.09%	19.195
pigz (min)	8.892	40.06%	12.199
gzip (default)	82.452	34.65%	16.912
pigz (default)	16.673	34.69%	12.852
gzip (max)	134.889	34.24%	16.513
pigz (max)	25.051	34.28%	12.129

xz
Compressor	Compression Time (s)	Compression Ratio	Decompression Time (s)
xz (min)	138.272	33.82%	50.916
lxz (min)	29.890	33.80%
xz (default)	1164.772	24.45%	34.114
lxz (default)	39.997	24.96%
xz (max)	1643.367	22.66%	31.968
lxz (max)	699.268	23.07%

As you can see, most parallel implementations decrease compression and decompression times by a factor of between 3 and 5, while taking, at most, a few tenths of a percent increase in final file size.

File Compressors in 64-bit

Ben — Sun, 07 Mar 2010 04:22:20 +0000

Though I’m not the sort of person who believes that native 64-bit compilations of programs will automagically make them perform faster or better, I do like to keep an eye on the state of the art, since I was an early adopter of native 64-bit OSes (I’ve been using 64-bit Linux since about Fedora Core 2 or 3, and beta versions of Windows XP x64) when AMD launched their K8 platform.

Previously, I’ve casually benchmarked the Javascript speeds of 64-bit browsers v. their 32-bit counterparts (here); more recently, I benchmarked a 64-bit compile of FLAC against several other 32-bit compiles of the same version (here).

This time, I decided to test various and sundry file compression utilities—more specifically, those which offer both 32- and 64-bit versions of themselves. This benchmark did not exhaustively test all potential combinations of compression options (if you’re interested in that, see Werner Bergman’s excellent Maximum Compression and Matt Mahoney’s Data Compression Programs), nor will it compare various compressors to each other; neither will it even list how well the programs actually compressed, since that’s not really a consideration here. The sole purpose of the benchmark was to compare the execution time of a 32-bit program with its 64-bit version.

The corpus in this case was enwiki9, compressed to a RAM disk to minimize the potential effects of write latency. I wanted the corpus to be sufficiently large to better tease out significant differences in these compressors over a large dataset.

The results for each compressor are listed on their own page, as well as an explanation of the compressor, its origin, and any additional notes. One notable exception to this list is WinRK, which is available in a 64-bit version but contains no command-line interface.

Yet more Javascript benchmarks

Ben — Sun, 21 Jun 2009 06:36:06 +0000

Every so often, it’s nice to take a look at the state of Javascript performance among the various browsers. Though misleading, it’s become something of a truism that “browser performance” is just a nice euphemism for “Javascript performance,” since any website doing anything interesting is basically leveraging Javascript to do it.

What’s come up since the last time I did any sort of Javascript performance comparison? Well, Google Chrome and its JS engine (“V8″), for one. Also, something of a new era in Javascript handling that attempts to optimize how browsers handle it by converting it to bytecode (or, in the case of JavaScriptCore/Squirrelfish Extreme/Nitro, directly to native machine code). In addition, there’s been some new benchmarks arrive on the scene, which allows us to tease out bias from any particular one.

It’s amazing, really, to compare these numbers against the linked benchmark from a mere 1.5 years ago. Opera went from being the top of the heap with 9.5 to being a lazy 3rd or 4th place. And Chrome, of course, decimated the competition (so far). Read on for the testing methodology and the results.

The tests

I chose four different benchmarks, three of which are entirely Javascript and one of which is mostly Javascript.

Dromaeo is a large, comprehensive benchmark that optionally incorporates some of the other benchmarks we’re looking at today. It was created by the Mozilla Corporation. It offers several test suites, but the one used here was the Dromaeo subset of tests
SunSpider is a benchmark created by the folks behind WebKit. It quickly became and remains one of the most popular and venerated Javascript benchmarks available.
V8 Benchmark is a benchmark application designed by Google to test its V8 Javascript engine. It’s actually a very short and simple benchmark that fantastically favors Google’s browser (and Safari).
Peacekeeper is a new browser benchmark by FutureMark. It’s supposed to test not just Javascript performance, but some rendering performance as well. As you will notice below, the numbers seem a little funny, as it appears to favor Safari (WebKit + Nitro) by a significant margin

The Hardware

All tests were run in Safe Mode (where applicable; meaning that no additional plugins or widgets were loaded) on the following hardware:

Intel Core2 Quad @ 2.4GHz
4GB RAM
Windows 7 build 7229, x64 Ultimate Edition

The Data

What follows is the raw data table. Jump down for the summary.

Javascript Benchmark Results
Browser	Dromaeo (higher is better)	SunSpider (lower is better)	V8 (higher is better)	Peacekeeper (higher is better)
Firefox 3.0.11	43.21 runs/s	3106.0 ms	240	1207
Firefox 3.5rc2	51.03	2652.8	283	1785
IE 8.0.7229.0	*	5176.0	96.5	829
Chrome 2.0.178.0	303.92	829.6	3214	2900
Chrome 3.0.189.0	303.14	793.6	3290	2968
Opera 9.64	29.55	3859.4	218	1406
Opera 10.00b1	71.23	3177.2	244	2019
Safari 4.0 (530.17)	221.73	801.2	2258	3238
Arora 0.7.1	88.85	2209.4	595	2344

The Conclusion

So what does this data tell us?

For starters, it means that the much-vaunted but long-developed Tracemonkey javascript engine in Firefox 3.5 is already a distant third-place player, if not fourth-place. Chrome’s V8 engine outperforms it by no less than a factor of 2 on just about every test. In fact, Chrome 3 on Windows 7 is the fastest browser available, period. Now, Chrome is a very simplistic browser (not even a print preview) and it remains to be seen if its competitive edge in terms of rendering and javascript speed will hold true when it’s got other things to worry about (i.e. plugins).

Safari 4 is a combination of the WebKit rendering engine (like Chrome) and JavaScriptCore, which is the umbrella project name for WebKit javascript engine. While the SquirrelFish engine in Safari 3.1 was fast (doing the JIT bytecode compiling that Firefox 3.5 is doing now…), the new SquirrelFish Extreme engine actually compiles to native machine code. It pulls respectable performance across all tests, usually ranking just behind Chrome…. with the notable exception of the Peacekeeper benchmark, where it takes the lead score. Unfortunately for Safari, its greatest accomplish with version 4 is only managing to not look completely horrible on other platforms, so needless to say it isn’t going to be a contender for the Win32 crown anytime soon.

How far Opera has fallen from grace. Whereas a couple of years ago it sported the fastest rendering, some of the greatest features, and a great javascript stack, it was quickly eclipsed by the flurry of development happening in other browsers recently. The beta of version 10.00 improves javascript performance significantly, although Opera’s own efforts toward bytecode-based javascript handling aren’t scheduled to land until well after 10.00.

Internet Explorer is, well, still Internet Explorer. While version 8 is a remarkable improvement over version 7, just as 7 was a spectacular improvement over 6, it is still the last shriveled pickle at the bottom of the jar, limping by in each and every benchmark. Microsoft, in their latest FUD to paint IE8 as a wunder-browser, correctly point out that javascript speed isn’t everything, but both benchmark and simple user testing are enough to point out that Microsoft’s browser team is working hard just to stay in the race; they’re not gaining ground at all.

Tracking LZMA efficiency

Ben — Tue, 10 Feb 2009 04:51:26 +0000

I’m a big fan of 7-Zip. It isn’t the best-looking application ever written, but that could be because its creator, Igor Pavlov, is concerned much more with its compression methods than its interface. 7-Zip has its own container format, but more important is the LZMA compression algorithm that Igor wrote and put into the public domain.

I decided to do some quick and dirty benchmarks to track the progress of LZMA/7-Zip over time. I went back as far as Igor supplied binaries, including one from the very old 3.x series. Rather than test every single release between then and now, I used only “stable” releases, with the exception of version 4.65, which is the latest version of any sort, as well as 4.66, which uses an alpha version of Igor’s new LZMA2 codec (and, as you’ll see, provides definite performance improvement).

I used Igor’s Timer utility to time the process (global time was reported). The corpus in this case was the Linux kernel source, v2.6.28. I conducted these tests on a RAM disk to eliminate hard disk latency issues (especially for decompressions, which improved by about 25% from my initial HDD-based tests). My rig is a Intel Core 2 Quad Q6600 [2.4Ghz], with 4GB of RAM (one dedicated to the RAM disk), running Vista SP1 x64.

The command line setup was an approximation of the 7-Zip GUI’s “ultra” settings: -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on, letting the archiver auto-choose the number of threads to spawn.

The Data

LZMA Efficiency
7-zip version	encoding time (s)	decoding time (s)
3.13	541.271	43.379
4.20	531.457	44.040
4.23	527.871	42.425
4.32	341.290	42.126
4.42	219.451	42.211
4.57	174.064	44.163
4.62	170.973	42.836
4.65	170.917	43.058
4.66 (lzma2)	126.259	46.663

The Analysis

Without conducting a more thorough battery of tests on a variety of different configurations, it’s difficulty to say with certain just where the performance improvements came from, be it better using of threading or multiprocessors, general algorithmic improvements, or something else. I also don’t know if the performance increases we see reside in improvements to LZMA itself as Igor was finalizing it, or just the code quality of 7-Zip, which implements LZMA.

In any case, the improvements since 3.13 are very clear (remember that lower is better), at least for compression, and for “ultra” settings. Decompression remained largely similar, which surprised me. Some of these results might be directly tied to the number and type of files that were compressed in the case: 4.66, for instance, improves decompression speed for uncompressable files, but no such files exist here since it’s source code.

Hats off to Igor Pavlov for his steady improvement on both a really great compression standard and one of my favorite pieces of software for Windows.

FLAC compile benchmarks

Ben — Sun, 21 Dec 2008 16:26:24 +0000

FLAC is a cross-platform codec, but when it comes to Windows, one has a pretty wide range of compiles. Some are more optimized than others.

I first got the idea for this benchmark when I stumbled upon a native 64-bit FLAC executable for Windows. Curious, I did a quick and dirty test against the canonical build for Windows and found that while encoding times were similar, decoding times were considerably faster.

To figure out why this is so (the 64-bitness or something else), I quickly pulled some some additional compiles and benchmarked them against a few different samples.

I have three different samples I used, ripped in WAV format as single files:

Lateralus, by Tool
Californication, by the Red Hot Chili Peppers
Bruckner’s 9th Symphony, by Giulini/Vienna Philharmonic, 1989

I’ve included a table for each sample, split into encode/decode times and compilation.

The builds I included were:

The canonical build from the official FLAC website
MSVC8 (Microsoft) compile from RareWares
ICL 9.1 (Intel) compile from RareWares
64-bit compile (MSVC?) from HardwareHacks 2000

My test system is a Intel Core 2 Quad Q6600, with 4GB of RAM, running Windows Vista x64 SP1, with all available patches. The timer used was by Igor Pavlov (the developer of 7-zip). Reported times are global (as opposed to kernel, user, or process times).

FLAC build benchmarks (time in seconds)
FLAC build	Action	canonical	MSVC8	ICL9.1	x64
Lateralus	encode	140.198	124.146	121.821	135.206
Lateralus	decode	36.770	17.098	16.708	18.315
Bruckner	encode	115.331	104.708	107.984	121.119
Bruckner	decode	33.384	33.338	30.030	31.387
Californication	encode	106.954	98.283	90.309	100.668
Californication	decode	26.973	25.972	25.943	27.238

It turns out that the x64′s advantage when it comes to decoding isn’t due to its 64-bitness, but probably because it includes some extra optimizations (SSE?) that are inherent ot a 64-bit compilation.

The clear winner in both encoding and decoding is the ICL9.1 compile (I wonder if a 10.1 compile would be even better); at least, it is so on my machine. The x64 compile was sometimes better, sometimes worse than the canonical compile. Clearly, there’s an early benefit from some optimization (probably SSE routines), along with a small margin of benefit from using Intel’s superior compiler. 64-bitness is beneficial for FLAC processes only insofar as it necessarily encompasses these routines as part of its instruction set.

If you’re looking to use a particular build of FLAC, I suggest you mosey on over to RareWares and john33′s ICL build: the resulting files are a few KB larger, but the processes itself is noticeably faster.

Javascript engines in 32-bit and 64-bit browsers

Ben — Mon, 24 Mar 2008 01:07:19 +0000

I recently performed some cursory Javascript benchmarks with the new version of Firefox and Safari; curious about performance, I decided to do some testing of 32-bit browsers against their 64-bit counterparts. On Windows Vista x64, the only two browsers so available to me are Internet Explorer 7 and a recent nightly build of Firefox 3. The 64-bit comes from the Mozilla x86-64 project, specifically the build from 21 March 2008. The 32-bit build is a proper nightly from the same date from the official Mozilla FTP.

My hypothesis, before performing the tests, was that the 64-bit compilation would have little or no effect on the Javascript engine performance. It’s so difficult to optimize Javascript rendering, which is inherently single-threaded, and it seems likely to only benefit from a faster CPU clock than bigger memory registers. Afterward, I felt vindicated: Internet Explorer is likely the best test, and the difference was not statistically significant. In Firefox’s case, the 64-bit build was actually significantly worse, though this could easily be due to some other factor I have not taken into account; I have assumed that the source was compiled on the same date. See below for more details.

The Data Tables

Internet Explorer 7.0.6.001.18000 32-bit and 64-bit comparison
Benchmark	32-bit (ms)	64-bit (ms)
Total	21972.4	19955.4
3d: cube	346.6	262.0
3d: morph	415.2	311.8
3d: raytrace	471.2	364.8
access: binary-trees	443.0	321.2
access: fannkuch	739.4	586.6
access: nbody	346.4	258.6
access: nsieve	305.8	284.2
bitops: 3bit-bits-in-byte	409.0	290.0
bitops: bits-in-byte	405.4	290.0
bitops: bitwise-and	446.2	284.0
bitops: nsieve-bits	349.4	290.0
control: recursive	493.0	349.4
crypto: aes	374.2	296.4
crypto: md5	330.8	243.2
crypto: sha1	346.4	246.6
date: format-tofte	433.6	315.0
date: format-xparb	440.0	349.6
math: cordic	465.0	340.0
math: partial-sums	327.6	228.0
math: spectral-norm	393.4	277.6
regexp: dna	387.0	302.6
string: base64	7298.0	8527.2
string: fasta	464.6	334.2
string: tagcloud	1391.4	1145.0
string: unpack-code	461.8	346.6
string: validate-input	3688.0	3110.8

Firefox 3.0b5pre (20080321 nightly) 32-bit and 64-bit comparison
Benchmark	32-bit (ms)	64-bit (ms)
Total	3750.2	7421.4
3d: cube	147.2	312.6
3d: morph	137.8	230.8
3d: raytrace	147.8	264.4
access: binary-trees	60.6	126.6
access: fannkuch	261.6	540.6
access: nbody	173.2	282.8
access: nsieve	103.4	185.0
bitops: 3bit-bits-in-byte	85.6	213.6
bitops: bits-in-byte	123.0	271.0
bitops: bitwise-and	136.4	246.2
bitops: nsieve-bits	148.6	310.0
control: recursive	57.0	140.8
crypto: aes	105.0	195.0
crypto: md5	78.4	175.2
crypto: sha1	88.0	163.6
date: format-tofte	202.2	406.4
date: format-xparb	129.2	258.2
math: cordic	186.2	367.4
math: partial-sums	149.2	257.2
math: spectral-norm	96.8	194.2
regexp: dna	255.6	652.8
string: base64	119.2	220.6
string: fasta	205.2	388.8
string: tagcloud	152.2	294.8
string: unpack-code	272.8	489.4
string: validate-input	128.0	233.4

The Raw Data

Internet Explorer x86 7.0.6001.18000
============================================
RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total:                  21972.4ms +/- 6.2%
--------------------------------------------

  3d:                    1233.0ms +/- 1.1%
    cube:                 346.6ms +/- 4.7%
    morph:                415.2ms +/- 2.7%
    raytrace:             471.2ms +/- 1.9%

  access:                1834.6ms +/- 2.5%
    binary-trees:         443.0ms +/- 3.9%
    fannkuch:             739.4ms +/- 1.5%
    nbody:                346.4ms +/- 7.3%
    nsieve:               305.8ms +/- 5.8%

  bitops:                1610.0ms +/- 3.7%
    3bit-bits-in-byte:    409.0ms +/- 2.0%
    bits-in-byte:         405.4ms +/- 4.8%
    bitwise-and:          446.2ms +/- 5.0%
    nsieve-bits:          349.4ms +/- 6.3%

  controlflow:            493.0ms +/- 2.2%
    recursive:            493.0ms +/- 2.2%

  crypto:                1051.4ms +/- 4.6%
    aes:                  374.2ms +/- 5.2%
    md5:                  330.8ms +/- 4.9%
    sha1:                 346.4ms +/- 6.2%

  date:                   873.6ms +/- 2.3%
    format-tofte:         433.6ms +/- 2.0%
    format-xparb:         440.0ms +/- 3.7%

  math:                  1186.0ms +/- 1.6%
    cordic:               465.0ms +/- 4.6%
    partial-sums:         327.6ms +/- 0.2%
    spectral-norm:        393.4ms +/- 4.1%

  regexp:                 387.0ms +/- 4.2%
    dna:                  387.0ms +/- 4.2%

  string:               13303.8ms +/- 10.4%
    base64:              7298.0ms +/- 6.8%
    fasta:                464.6ms +/- 3.5%
    tagcloud:            1391.4ms +/- 4.0%
    unpack-code:          461.8ms +/- 3.7%
    validate-input:      3688.0ms +/- 25.4%

Internet Explorer x64
============================================
RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total:                  19955.4ms +/- 1.2%
--------------------------------------------

  3d:                     938.6ms +/- 2.3%
    cube:                 262.0ms +/- 6.2%
    morph:                311.8ms +/- 6.3%
    raytrace:             364.8ms +/- 2.9%

  access:                1450.6ms +/- 2.8%
    binary-trees:         321.2ms +/- 3.3%
    fannkuch:             586.6ms +/- 3.7%
    nbody:                258.6ms +/- 6.6%
    nsieve:               284.2ms +/- 3.1%

  bitops:                1154.0ms +/- 1.7%
    3bit-bits-in-byte:    290.0ms +/- 3.7%
    bits-in-byte:         290.0ms +/- 7.6%
    bitwise-and:          284.0ms +/- 5.8%
    nsieve-bits:          290.0ms +/- 6.0%

  controlflow:            349.4ms +/- 3.0%
    recursive:            349.4ms +/- 3.0%

  crypto:                 786.2ms +/- 2.8%
    aes:                  296.4ms +/- 4.6%
    md5:                  243.2ms +/- 7.1%
    sha1:                 246.6ms +/- 10.3%

  date:                   664.6ms +/- 6.0%
    format-tofte:         315.0ms +/- 11.6%
    format-xparb:         349.6ms +/- 3.1%

  math:                   845.6ms +/- 2.5%
    cordic:               340.0ms +/- 2.5%
    partial-sums:         228.0ms +/- 4.5%
    spectral-norm:        277.6ms +/- 6.0%

  regexp:                 302.6ms +/- 3.5%
    dna:                  302.6ms +/- 3.5%

  string:               13463.8ms +/- 1.4%
    base64:              8527.2ms +/- 0.9%
    fasta:                334.2ms +/- 5.2%
    tagcloud:            1145.0ms +/- 2.6%
    unpack-code:          346.6ms +/- 6.2%
    validate-input:      3110.8ms +/- 2.4%

Firefox x86 Firefox 3.0b5pre (20080322 nightly)
============================================
RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total:                 3750.2ms +/- 1.4%
--------------------------------------------

  3d:                   432.8ms +/- 9.8%
    cube:               147.2ms +/- 17.4%
    morph:              137.8ms +/- 1.2%
    raytrace:           147.8ms +/- 22.7%

  access:               598.8ms +/- 2.1%
    binary-trees:        60.6ms +/- 5.5%
    fannkuch:           261.6ms +/- 1.7%
    nbody:              173.2ms +/- 6.0%
    nsieve:             103.4ms +/- 1.4%

  bitops:               493.6ms +/- 5.4%
    3bit-bits-in-byte:   85.6ms +/- 2.2%
    bits-in-byte:       123.0ms +/- 6.4%
    bitwise-and:        136.4ms +/- 11.9%
    nsieve-bits:        148.6ms +/- 9.2%

  controlflow:           57.0ms +/- 3.1%
    recursive:           57.0ms +/- 3.1%

  crypto:               271.4ms +/- 7.3%
    aes:                105.0ms +/- 19.4%
    md5:                 78.4ms +/- 3.8%
    sha1:                88.0ms +/- 2.0%

  date:                 331.4ms +/- 12.3%
    format-tofte:       202.2ms +/- 16.2%
    format-xparb:       129.2ms +/- 8.3%

  math:                 432.2ms +/- 3.9%
    cordic:             186.2ms +/- 7.6%
    partial-sums:       149.2ms +/- 7.0%
    spectral-norm:       96.8ms +/- 1.4%

  regexp:               255.6ms +/- 8.3%
    dna:                255.6ms +/- 8.3%

  string:               877.4ms +/- 4.7%
    base64:             119.2ms +/- 6.0%
    fasta:              205.2ms +/- 10.4%
    tagcloud:           152.2ms +/- 6.2%
    unpack-code:        272.8ms +/- 4.1%
    validate-input:     128.0ms +/- 10.3%

Firefox x64
============================================
RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total:                  7421.4ms +/- 1.2%
--------------------------------------------

  3d:                    807.8ms +/- 2.0%
    cube:                312.6ms +/- 3.5%
    morph:               230.8ms +/- 7.8%
    raytrace:            264.4ms +/- 6.5%

  access:               1135.0ms +/- 1.9%
    binary-trees:        126.6ms +/- 11.7%
    fannkuch:            540.6ms +/- 2.3%
    nbody:               282.8ms +/- 2.9%
    nsieve:              185.0ms +/- 10.4%

  bitops:               1040.8ms +/- 2.3%
    3bit-bits-in-byte:   213.6ms +/- 5.8%
    bits-in-byte:        271.0ms +/- 4.5%
    bitwise-and:         246.2ms +/- 6.8%
    nsieve-bits:         310.0ms +/- 6.1%

  controlflow:           140.8ms +/- 6.6%
    recursive:           140.8ms +/- 6.6%

  crypto:                533.8ms +/- 4.3%
    aes:                 195.0ms +/- 5.8%
    md5:                 175.2ms +/- 11.5%
    sha1:                163.6ms +/- 3.5%

  date:                  664.6ms +/- 4.5%
    format-tofte:        406.4ms +/- 4.6%
    format-xparb:        258.2ms +/- 6.3%

  math:                  818.8ms +/- 3.8%
    cordic:              367.4ms +/- 3.2%
    partial-sums:        257.2ms +/- 10.5%
    spectral-norm:       194.2ms +/- 4.9%

  regexp:                652.8ms +/- 2.7%
    dna:                 652.8ms +/- 2.7%

  string:               1627.0ms +/- 3.1%
    base64:              220.6ms +/- 4.7%
    fasta:               388.8ms +/- 4.4%
    tagcloud:            294.8ms +/- 5.2%
    unpack-code:         489.4ms +/- 2.9%
    validate-input:      233.4ms +/- 11.1%

Linux 2.6 filesystem benchmarks

Ben — Fri, 21 Mar 2008 21:30:33 +0000

A recent Linux filesystem benchmark, using modern filesystems, takes a look at some hard numbers. Looks like JFS usually comes in on top.

Firefox 3.0b4 vs. 2.0.0.12 Javascript benchmark

Ben — Wed, 19 Mar 2008 15:45:56 +0000

Continuing the tradition of my previous post about Safari, I thought I would revisit Firefox’s performance on the SunSpider benchmark.

Clearly, Firefox has also made drastic strides in its Javascript engine, which is an entirely new piece donated by Adobe (and is the same engine, or so I understand, which powers the Actionscript interpreter of Flash). These results put the new Firefox 3 beta at four times the speed of the latest 2.x series. The newly released Safari/Webkit build narrowly outperforms the latest test build of Firefox, but clearly there’s a lot of progress being made (and perhaps more before its final release).

The tests were done on a Windows XP SP2 machine; it has a Pentium 4 and 2GB of RAM.

Firefox 3.0 beta 4 vs. 2.0.0.12
Benchmark	Firefox 3.0 beta4 (ms)	Firefox 2.0.0.12 (ms)
Total	6102.8	23493.8
3d: cube	301.0	812.6
3d: morph	245.8	1562.2
3d: raytrace	295.2	491.0
access: binary-trees	95.6	243.6
access: fannkuch	412.2	571.8
access: nbody	282.2	721.6
access: nsieve	102.4	337.4
bitops: 3bit-bits-in-byte	120.4	362.4
bitops: bits-in-byte	143.0	381.2
bitops: bitwise-and	267.2	3406.2
bitops: nsieve-bits	251.6	559.4
control: recursive	89.0	153.2
crypto: aes	148.8	356.2
crypto: md5	171.4	393.8
crypto: sha1	180.6	400.0
date: format-tofte	329.6	1294.0
date: format-xparb	230.4	3475.0
math: cordic	321.0	940.8
math: partial-sums	310.4	546.6
math: spectral-norm	183.8	381.4
regexp: dna	295.6	1400.4
string: base64	157.8	928.2
string: fasta	280.0	640.8
string: tagcloud	249.6	1025.0
string: unpack-code	412.2	1512.2
string: validate-input	208.0	596.8

Firefox 2.0.0.12
============================================
RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total:                 23493.8ms +/- 6.3%
--------------------------------------------

  3d:                   2865.8ms +/- 1.3%
    cube:                812.6ms +/- 3.0%
    morph:              1562.2ms +/- 2.0%
    raytrace:            491.0ms +/- 4.5%

  access:               1874.4ms +/- 3.0%
    binary-trees:        243.6ms +/- 14.6%
    fannkuch:            571.8ms +/- 1.8%
    nbody:               721.6ms +/- 2.2%
    nsieve:              337.4ms +/- 3.2%

  bitops:               4709.2ms +/- 1.3%
    3bit-bits-in-byte:   362.4ms +/- 7.0%
    bits-in-byte:        381.2ms +/- 2.8%
    bitwise-and:        3406.2ms +/- 0.9%
    nsieve-bits:         559.4ms +/- 1.5%

  controlflow:           153.2ms +/- 13.9%
    recursive:           153.2ms +/- 13.9%

  crypto:               1150.0ms +/- 2.8%
    aes:                 356.2ms +/- 6.0%
    md5:                 393.8ms +/- 5.3%
    sha1:                400.0ms +/- 5.5%

  date:                 4769.0ms +/- 0.9%
    format-tofte:       1294.0ms +/- 1.3%
    format-xparb:       3475.0ms +/- 1.0%

  math:                 1868.8ms +/- 3.0%
    cordic:              940.8ms +/- 4.8%
    partial-sums:        546.6ms +/- 2.4%
    spectral-norm:       381.4ms +/- 8.5%

  regexp:               1400.4ms +/- 81.1%
    dna:                1400.4ms +/- 81.1%

  string:               4703.0ms +/- 9.4%
    base64:              928.2ms +/- 2.3%
    fasta:               640.8ms +/- 3.1%
    tagcloud:           1025.0ms +/- 46.6%
    unpack-code:        1512.2ms +/- 10.5%
    validate-input:      596.8ms +/- 6.7%

Firefox 3.0b4
============================================
RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total:                  6102.8ms +/- 2.3%
--------------------------------------------

  3d:                    842.0ms +/- 4.8%
    cube:                301.0ms +/- 5.3%
    morph:               245.8ms +/- 1.7%
    raytrace:            295.2ms +/- 10.5%

  access:                910.4ms +/- 2.9%
    binary-trees:         95.6ms +/- 2.4%
    fannkuch:            412.2ms +/- 5.6%
    nbody:               282.2ms +/- 3.2%
    nsieve:              120.4ms +/- 8.5%

  bitops:                782.2ms +/- 5.8%
    3bit-bits-in-byte:   120.4ms +/- 22.1%
    bits-in-byte:        143.0ms +/- 21.1%
    bitwise-and:         267.2ms +/- 4.3%
    nsieve-bits:         251.6ms +/- 3.1%

  controlflow:            89.0ms +/- 1.7%
    recursive:            89.0ms +/- 1.7%

  crypto:                500.8ms +/- 5.5%
    aes:                 148.8ms +/- 17.2%
    md5:                 171.4ms +/- 2.0%
    sha1:                180.6ms +/- 1.9%

  date:                  560.0ms +/- 4.2%
    format-tofte:        329.6ms +/- 6.8%
    format-xparb:        230.4ms +/- 2.1%

  math:                  815.2ms +/- 5.5%
    cordic:              321.0ms +/- 1.1%
    partial-sums:        310.4ms +/- 9.2%
    spectral-norm:       183.8ms +/- 16.1%

  regexp:                295.6ms +/- 6.4%
    dna:                 295.6ms +/- 6.4%

  string:               1307.6ms +/- 3.7%
    base64:              157.8ms +/- 16.5%
    fasta:               280.0ms +/- 9.8%
    tagcloud:            249.6ms +/- 3.9%
    unpack-code:         412.2ms +/- 7.9%
    validate-input:      208.0ms +/- 4.5%

Safari 3.04 vs 3.1 Javascript benchmarks

Ben — Tue, 18 Mar 2008 21:23:08 +0000

Safari 3.1 has been released, bringing with it all the latest and great Webkit code. Even though the UI still sucks (at least on Windows; ever hear of native GUIs, Apple?)

I decided to benchmark the Javascript performance of the new Safari against its more immediate predecessor, 3.04. This testing was done on a Windows XP SP2 installation; a HP workstation with a Pentium 4 and 2GB of RAM.

As you can see in the table below, the JS engine has improved considerably since the last build, cutting the total time by more than half.

Safari 3.04 vs. 3.1
Benchmark	Safari 3.1 (ms)	Safari 3.04 (ms)
Total	5865.4	13525.2
3d: cube	218.6	478.0
3d: morph	197.0	818.6
3d: raytrace	287.4	537.8
access: binary-trees	106.4	212.6
access: fannkuch	446.8	977.8
access: nbody	256.4	581.4
access: nsieve	134.2	584.0
bitops: 3bit-bits-in-byte	128.0	515.8
bitops: bits-in-byte	156.2	537.4
bitops: bitwise-and	212.4	450.0
bitops: nsieve-bits	162.6	819.0
control: recursive	146.8	315.2
crypto: aes	175.2	337.6
crypto: md5	128.4	475.0
crypto: sha1	156.0	518.6
date: format-tofte	262.6	390.6
date: format-xparb	415.6	672.0
math: cordic	271.8	847.0
math: partial-sums	287.4	387.6
math: spectral-norm	165.6	365.6
regexp: dna	334.6	531.2
string: base64	190.4	453.2
string: fasta	290.6	628.2
string: tagcloud	203.2	340.8
string: unpack-code	234.2	312.6
string: validate-input	297.0	437.6

Here’s the raw data, if you so desire it.

Safari 3.04

============================================
RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total:                 13525.2ms +/- 0.6%
--------------------------------------------

  3d:                   1834.4ms +/- 4.8%
    cube:                478.0ms +/- 5.5%
    morph:               818.6ms +/- 8.8%
    raytrace:            537.8ms +/- 6.1%

  access:               2355.8ms +/- 2.1%
    binary-trees:        212.6ms +/- 17.9%
    fannkuch:            977.8ms +/- 1.1%
    nbody:               581.4ms +/- 4.4%
    nsieve:              584.0ms +/- 6.0%

  bitops:               2322.2ms +/- 2.4%
    3bit-bits-in-byte:   515.8ms +/- 2.7%
    bits-in-byte:        537.4ms +/- 5.4%
    bitwise-and:         450.0ms +/- 7.1%
    nsieve-bits:         819.0ms +/- 3.1%

  controlflow:           315.2ms +/- 12.5%
    recursive:           315.2ms +/- 12.5%

  crypto:               1331.2ms +/- 2.2%
    aes:                 337.6ms +/- 8.8%
    md5:                 475.0ms +/- 6.3%
    sha1:                518.6ms +/- 3.2%

  date:                 1062.6ms +/- 5.1%
    format-tofte:        390.6ms +/- 7.0%
    format-xparb:        672.0ms +/- 5.4%

  math:                 1600.2ms +/- 3.3%
    cordic:              847.0ms +/- 3.0%
    partial-sums:        387.6ms +/- 4.2%
    spectral-norm:       365.6ms +/- 9.8%

  regexp:                531.2ms +/- 5.8%
    dna:                 531.2ms +/- 5.8%

  string:               2172.4ms +/- 3.1%
    base64:              453.2ms +/- 6.2%
    fasta:               628.2ms +/- 5.1%
    tagcloud:            340.8ms +/- 11.7%
    unpack-code:         312.6ms +/- 10.7%
    validate-input:      437.6ms +/- 5.4%

Safari 3.1
============================================
RESULTS (means and 95% confidence intervals)
--------------------------------------------
Total:                  5865.4ms +/- 1.8%
--------------------------------------------

  3d:                    703.0ms +/- 11.3%
    cube:                218.6ms +/- 15.4%
    morph:               197.0ms +/- 22.7%
    raytrace:            287.4ms +/- 10.1%

  access:                943.8ms +/- 7.0%
    binary-trees:        106.4ms +/- 19.8%
    fannkuch:            446.8ms +/- 10.0%
    nbody:               256.4ms +/- 4.2%
    nsieve:              134.2ms +/- 12.8%

  bitops:                659.2ms +/- 4.4%
    3bit-bits-in-byte:   128.0ms +/- 6.5%
    bits-in-byte:        156.2ms +/- 15.3%
    bitwise-and:         212.4ms +/- 13.8%
    nsieve-bits:         162.6ms +/- 16.1%

  controlflow:           146.8ms +/- 17.9%
    recursive:           146.8ms +/- 17.9%

  crypto:                459.6ms +/- 11.7%
    aes:                 175.2ms +/- 21.4%
    md5:                 128.4ms +/- 16.3%
    sha1:                156.0ms +/- 15.1%

  date:                  678.2ms +/- 4.4%
    format-tofte:        262.6ms +/- 12.1%
    format-xparb:        415.6ms +/- 7.1%

  math:                  724.8ms +/- 7.0%
    cordic:              271.8ms +/- 10.7%
    partial-sums:        287.4ms +/- 9.0%
    spectral-norm:       165.6ms +/- 17.7%

  regexp:                334.6ms +/- 7.7%
    dna:                 334.6ms +/- 7.7%

  string:               1215.4ms +/- 4.5%
    base64:              190.4ms +/- 13.3%
    fasta:               290.6ms +/- 8.9%
    tagcloud:            203.2ms +/- 13.4%
    unpack-code:         234.2ms +/- 10.2%
    validate-input:      297.0ms +/- 9.1%