A Modest Construct » software

Linux Command-Line Compressors Compared

Ben — Tue, 03 May 2011 13:22:31 +0000

A long time ago, I ran a comparison of various command-line compressors in Linux. Recently, intrigued by the rise of parallel computing and the emergence of multi-processor versions of old *nix favorites like gzip and bzip2, I thought I’d give the benchmark another go.

The Setup

My machine is an Intel Core 2 Quad Q6600 [2.4Ghz] on a Gigabyte GA-N680SLI-DQ6 (nVidia 680i), with 4GB of Corsair XMS2(PC2 6400). The tests were run on a vanilla installation of Ubuntu 10.10 x64, fully updated. The compressors themselves were compiled by me from source code, so I would have the latest and greatest at the time of testing.

I did not use a RAM disk, since I wanted all of my available RAM available for compression (some of the tests used aggressive settings). I did use a freshly-formatted SATA disk for the purpose, however. For further information, see the methodology and results.

Timing was done with the GNU time command.

The Compressors

gzip. Perhaps the most well-known compressor on Linux, gzip is a DEFLATE-based compressor which offers moderate compression at a very fast speed. Note: I made the mistake of retrieving the sources from www.gzip.org, which has links to v1.2.4, from 1999. The most recent stable version is 1.4, from GNU’s servers. The version reflected in the benchmark, therefore, is out of date.
bzip2. An old favorite, bzip2 is a basic block-level compressor which is more aggressive than its cousin, gzip, at the cost of speed.
xz. A relatively new compressor, xz arose from the ashes of LZMAUtils, an implementation of the LZMA compression spec from Igor Pavlov’s 7-zip project. xz is is the command-line equivalent, and similar in usage to gzip and bzip2; more to the point, it’s been accepted by the GNU toolchain and now has its own filter in tar (-J).
rzip. Originally written by Andrew Tridgell (of Samba fame, and who indirectly caused git, when you think about it) as a doctoral project. It functions similarly to gzip and bzip2, according to its author, but attempts to take advantage of “long-distance redundancies” in larger files; that is, where bzip2 may only process 900K chunks at a time, rzip will seek to look beyond that in order to eek out additional compression.
lzop. One of the Lempel-Ziv-Oberhumer (LZO) compressors, it focuses on decompression speed rather than pure compression.
compress. Though not really a contender in any real zip, I included the old compress program for comparative purposes. Compress is a fast LZW compressor which creates .Z files, and can be triggered as a filter in tar with -Z. The implementation used is ncompress.
zip. An old standby, zip creates, well, .zip files, using the DEFLATE algorithm. One difference between zip and gzip is that zip creates a structured archive (no tar necessary).
lzip. Another LZMA-based compressor, lzip is heavily-asynchronous, emphasizing small decompression times at the expensive of larger initial compression time.
p7zip. The *nix version of Igor Pavlov’s 7-Zip for Windows, p7zip has a slew of options (including different compressors, such as ppmd), but we’ll be focusing on lzma2.
pigz. A parallel implementation of gzip. Like most parallel implementations, pigz sacrifices a small amount of compression efficiency for large gains in speed.
lbzip2. A parallel implementation of bzip2.
pbzip2. Another parallel implementation of bzip2.
quicklz (qpress). A very fast implementation of LZO, the QuickLZ library was at version 1.5.0 at the time of the benchmark, but its companion compressor program, QPress, was linked against 1.4.1.
lxz. A parallel implementation of the xz compressor. By the same creator as lbzip2. It supports compression only, so the benchmark will not contain decompression numbers for this tool. The author considers it a stopgap until xz gets multithreading support.
lrzip. Long ago, a fork of rzip, Con Kolivas’ lrzip has become a full-fledged compressor with multiple possible algorithms to choose from; the only one tested was LZMA. Note that the version tested here (0.571) was replaced shortly after I ran my benchmarks by v0.600, which was a large rewrite ostensibly improving decompression speed.
plzip. A straightfoward parallelization of the lzip compressor.

A note on options.

Most of the compressors tested follow the same basic pattern for specifying compression strength, modeled after the options of gzip and bzip2. -# specifies compression strength, where # is a number. On some compressors, 0 is the lowest; on others, it’s 1. My approach was to test the lowest strength, the highest strength, and the default strength (either 5 or 6, depending).

On some multithreaded compressors, it was necessary to specify the number of threads or processors to use. Where this was required, I use a value of 4, the number of cores in my processor.

The Methodology

Corpus. I downloaded the 11/3/2009 pages-articles.xml.bz2 from Wikipedia’s mirror an uncompressed it. 23+ GB. Uh oh. Realizing that I didn’t have all the time in the world to spend compressing things, I truncated the file to its first 1’073’741’824 bytes (1GB) using the truncate command.

For each compressor/setting, the compression was run three times; the compressed size was recorded; then the decompression was run three times. Each run’s time was recorded, and the average and standard deviation was calculated for both compression and decompression.

Note that in some cases, operation speed was limited by the speed of the hard drive (I’ll cover this in detail in the next section); there isn’t much to be done about this, other than the note it. Multiple runs should tease out inconsistencies and show where there is a real discrepancy as opposed to a consistent bottleneck.

The Results

The data table itself is far too large to fit in this template, but you can see it as Google Spreadsheet here.

There’s no way to crown a clear winner, since all compressors mean tradeoffs. If I were as smart and capable as Werner Bergman, I could have a formula worked up for compressor efficiency, but I’m more curious to see the differences between single-threaded compressors and their multithreaded variants.

To get it out of the way: xz at maximum compression had the best compression ratio of all tested configurations. At 243’350’864 bytes, it shrank the test corpus to 22.7% of its original size. This came at a heavy cost, though, as compression at this level took an average of 27 minutes, give or take 200 seconds.

The fastest compression, by far, came from QuickLZ, whose lowest setting managed to process the entire 1GB file in an average of 6.3 seconds. Its maximum setting doubled the time to 12 seconds and shaved off an additional 75MB, but the price of such fantastic speed is much less compression: the maximum setting reduced our test corpus to only 44.4% of its original size.

Remember that there is no best compressor; there is only a compressor that is best for your needs. Compression is all about diminishing returns: it takes a mere 12 seconds to get from 100% to 44% with QuickLZ. It takes an additional 27 minutes to get from 44% to 22% with xz.

When looking at the decompression times, one might notice that the scores bottom out at between 12-15 seconds. My guess is this does not reflect the decompression speed at much as the write speed of the hard drive. In theory, one might see even lower numbers with a RAMdisk, but in practice, decompression speeds which outpace storage write speeds are unlikely to be taken advantage of outside a benchmark scenario.

The numbers for bzip2 vary wildly, for reasons I haven’t figured out. I ran a second iteration of three tests for each setting and got similar numbers, so I stuck with my initial measurements. Other compressors don’t have such high standard deviations relative to their runtimes, so I don’t think it was a problem with my setup.

Generally speaking, the parallel implementations of single-threaded compressors fare extremely well. Let’s look at the average times for some of the standard ones. Note that compression ratios are given as the percentage of the original filesize, not the percent reduction, so lower is better.

bzip2
Compressor	Compression Time (s)	Compression Ratio	Decompression Time (s)
bzip2 (min)	186.509	31.59%	63.342
lbzip2 (min)	38.852	31.63%	18.520
pbzip2 (min)	38.072	31.61%	13.533
bzip2 (default)	169.474	28.14%	64.432
lbzip2 (default)	42.085	28.15%	19.140
pbzip2 (default)	40.698	28.33%	14.630
bzip2 (max)	206.307	27.18%	66.460
lbzip2 (max)	52.091	27.19%	23.092
pbzip2 (max)	52.438	27.19%	20.617

gzip
Compressor	Compression Time (s)	Compression Ratio	Decompression Time (s)
gzip (min)	36.146	40.09%	19.195
pigz (min)	8.892	40.06%	12.199
gzip (default)	82.452	34.65%	16.912
pigz (default)	16.673	34.69%	12.852
gzip (max)	134.889	34.24%	16.513
pigz (max)	25.051	34.28%	12.129

xz
Compressor	Compression Time (s)	Compression Ratio	Decompression Time (s)
xz (min)	138.272	33.82%	50.916
lxz (min)	29.890	33.80%
xz (default)	1164.772	24.45%	34.114
lxz (default)	39.997	24.96%
xz (max)	1643.367	22.66%	31.968
lxz (max)	699.268	23.07%

As you can see, most parallel implementations decrease compression and decompression times by a factor of between 3 and 5, while taking, at most, a few tenths of a percent increase in final file size.

GNOME Audio Player Shootout v3.0

Ben — Sun, 29 Aug 2010 17:08:05 +0000

In January 2007 I published the GNOME Audio Player Shootout, a simple comparison of the options available to GNOME users for handling their day-to-day playback needs. It proved to be so popular that in December of 2008 I did a followup, excluding some abandoned players and adding some new ones. Though it hasn’t been quite two years yet, I thought it was time for another look at the state of audio players in the GNOME ecosystem.

This time around, I’ve got a heavy focus on new players, as there have been a number of new arrivals since my last shootout that show a lot of promise. This article will cover (in no particular order):

Rhythmbox (0.12.8)
Exaile (3.2.0)
Banshee (1.7.4)
Quod Libet (2.2.1)
Guayadeque (0.2.6-svn1186)
DeaDBeeF (0.4.1)
aTunes (2.0.1)
xnoise (0.1.10)
GMusicBrowser (1.1.5-git)
Aqualung (0.9~beta11)

All testing was done using an up-to-date Ubuntu Lucid x64 with all necessary repositories added, including some PPAs for the last versions of these players. Considered but not reviewed were Decibel Audio Player (hasn’t changed appreciably since last time), Gejengel (so unstable as to be unusable), and Bluemindo (still too simple to be useful).

Please note that this article necessarily incorporates some of my own biases. I am an avowed foobar2000 fan and you will notice that I tend to favor the utility-minded players over the media centers and iTunes clones. This article should still be useful, even if your own inclinations are different from mine.

Now Reading Reloaded changes

Ben — Wed, 19 Aug 2009 21:15:48 +0000

Starting August 15th, Amazon’s web service API started requiring all requests to be signed—that is, they must include a cryptographically generated key.

This is important, because while the service as always required an ID to run, it was never a secret. In fact, the former developer’s access ID has been embedded in the Now Reading Reloaded script since the very beginning.

The change, however, requires the addition of another key, this one like a password, and it’s not supposed to be given out. Since Now Reading Reloaded is open source, that means anyone who wanted to could use my key.

As a result, I’ve modified the plugin to use two more fields in the options screen, one for the Access ID and one for the Secret Key: both of these are required to add books from Amazon, and you will have to get both yourself.

Actually, it’s easy to do: go to the AWS site and register. Then plug the two keys they give you into the appropriate spaces in the Now Reading options screen. Resume reading.

Now Reading news

Ben — Thu, 02 Jul 2009 05:14:45 +0000

I have providing some light maintenance development for Rob’s Now Reading plugin; since WordPress 2.7 wholly changed its interface, the plugin need some tweaking to make it work.

Up to this point, I’ve been hosting it locally, mostly picking at it whenever time allows.

I just updated it the other day to add a new feature (editable ASIN) and hopefully fix a recurring bug (CDATA error when searching).

In any case, I hope to make a push in the near future to clean it up and submit it the official WordPress plugin site so that its user can benefit from auto-update, etc. etc. My own much-atrophied skills as a PHP developer aside (I deal mostly with Java at work), I think that it will ultimately benefit everybody, assuming I can make it so that the updates don’t override custom templates (perhaps giving preference to Now Reading template files in the theme folder?).

~~Stay tuned.~~ The plugin is now here.

Comments on this post are closed. For support, please use the forum feature of the official plugin repository.

The Ethics of Software Patents

Ben — Wed, 22 Apr 2009 04:54:44 +0000

rev. 15 April 2009; get the PDF

The laws that protect the creation of content are manifold and complicated—even byzantine. America has copyright protection, which applies to concrete expressions of information, trademark protection, which protects distinctive symbols or verbiage associated with a legal entity, and patent protection, which protects “(1) processes, (2) machines, (3) manufactures or (4) compositions of matter” and is perhaps the least understood of all the various kinds of intellectual property protection (Guntersdorfer, 2003).

The explosion of the Internet in the late 20th and early 21st centuries has thrown into stark relief both the legal problems associated with protecting content in a digital age as well as the ethical issues inherent in the existing process for acquiring official intellectual property protection and the rights afforded involved parties in a redress of grievances. Copyright law specifically has come into public consciousness primarily due to the popularity of filesharing: for all intents and purposes, the advent of modern filesharing was the 1999 arrival of Napster, a program which allowed anyone to exchange digital copies of music online, for free. Legal problems eventually forced Napster to shut down (Ante, Brull, Herman , & France, 2000), but its legacy leaves not only alternative modes of filesharing, but a whole host of new web-based content creation engines that toe the lines of fair use.

The politics of copyright is, despite its complexity, relatively straightforward and beyond the purview of this writing. More relevant to the machinations of the world of business are patents, which tend to pertain to the inventions and processes that drive economic engines. While some companies, such as Coca-Cola, eschew the legal—albeit ephemeral—benefit of patents in favor of strict secrecy, the legal mechanism is still quite popular: the United States Patent Office grants between 175,000 and 195,000 patents every year (“U.S. Patent Statistics,” 2009).

Ideally, patent mechanisms act as a function of market efficiency: by protecting corporate or industrial investment in Research & Development, it promotes economic growth, protects ingenuity, and rewards success; most patent law even allows for the eventual “freeing” of the patent after set duration—now usually twenty years (Heisey, King, Rubenstein, & Shoemaker, 2006, pp. iii-v, 4-7). What has come to pass since the 1970s, however, is a new scope of invention—specifically, computer software—that the existing patent protection mechanism is ill-equipped to handle. The critical question this raises is such: to what extent can the granting of patents for software be considered ethical? Conversely, if this is not the case, to what degree is it unethical or antithetical to proposed ideas of market efficiency?

It is vital to illustrate the very important distinctions between copyright and patent when it comes to computer software. Programs consist of written code (called “source code”), usually in a high-level programming language, which is at least partially intelligible to a human reader: this code, like any other written work, is subject to copyright. When portions of the source code to Microsoft’s WindowsÂ® Operating System were stolen and distributed on the Internet (Lemos, 2004), the case was one of unauthorized distribution of copyrighted content, on par with selling bootlegged DVDs. All source code must eventually be “compiled” to create the usable form of software that can be run on personal computers; the compilation process turns source code into a series of 1′s and 0′s, which the computer uses to runs billions of mathematical operations per second. Abstractions aside, all computer programs are little more than large lists of math problems which produce a desired result. Software patents are, at heart, an attempt to gain a “monopoly” on this math (Guntersdorfer, 2003).

“Many politicians [...] don’t understand what software patents do. They often think patents are similar to copyright law [...], which is not the case,” writes noted open source software1 advocate Richard Stallman (2005). He proposes a hypothetical scenario wherein literary concepts can and are patented, and Victor Hugo is unable to write Les Misérables because some lucky entity has acquired a patent for “a communication process that represents, in the mind of a reader, the concept of a character who has been in jail for a long time and subsequently changes his name.” There exist, he insists, patents for such simple things as a progress bar, or accepting online payments via credit card.

To such opponents of software patents, the very nature of patents is inherently unethical in this area of creative content: rather than protect the ingenuity and effort of intellectual investors (so to speak), it rewards those who seek to manipulate the legal system in order to monopolize generic or overly-broad concepts.

In order to examine the ethical ramifications of patents, some working ethical precepts must be established. Specifically:

Economic efficiency is an ethical good insofar as it contributes broadly to the public welfare, within normative moral constraints (Schultz, 2001, pp. 12-14).
Intellectual investment, in the form of corporate research and development or personal ingenuity, can and should be rewarded with an as-yet unspecified right to profit from said effort. This, besides being an ethical good in and of itself, directly contributes to principle #1 (“Are Efforts”, 2001).
The right to personal property, including intellectual property within reasonable bounds, is unambiguous and unabridged.

In addition to the aforementioned points, several more may be offered which are relevant to the discussion of computer software:

The labor efficiency offered by computer software (as well as its attendant effect upon the overall economic good), may be assumed of equal importance to the economic good of content creators—i.e. interoperability, work efficiency, and accessibility of data is at least as important as the right of technologists to individually profit (Lenk, Hoppe, & Adorno, 2007, pp. 91-92).
The unencumbered public availability of processes or information previously under patent protection,whether by design (read: open source software or open specification) or expiration of patent), is considered an unambiguous public good (“Are Efforts”, 2001).
Patents which are overly broad—including those which infringe prior art—have an unnecessarily depressive effect upon both technological progress and the economy as a whole.

As Bessen and Meurer (2008) note, patent litigation has rapidly outpaced other business-related litigation since the 1980s, second only to copyright litigation, and that mostly due to more or less legitimate claims of filesharing-related infringement (p. 127).

Whether software patents are helpful or ethical at all is more to the point. Allison, Mann, & Dunn (2007) quote principal witnesses at a 1994 patent hearing: “Software should not be patented, not because it is difficult to do so, but because it is wrong to do so”; “[t]here is absolutely no evidence whatsoever, not a single iota, that software patents have promoted or will promote progress” (1581). The authors conclude that policy debate now shies away from the common idea that the patent system is broken and unfair: the hue and cry, especially from software companies who ostensibly rely on patents, is to require a technical contribution—that is, implementation aside from a theoretical business method (1621-22).

Early patent law recognized that software seemed inherently unpatentable because it was little more than structured math (Guntersdorfer, 2003; Klemens, 2006, p. 4); mathematical formulae, were they to be subject to such “protection,” would be prohibitive to innovation and damage the public good. Mathematics, fundamentally, isn’t subject to ownership any more than other means of describing reality, such as physics or chemistry. Because software patents can describe not simply a method for obtaining a result, but the very idea of obtaining that result—for instance, Stallman’s example of the patent for credit card payment, irrespective of the code used to construct the process—they have ceased to reward innovation and instead hamper technological progress, deny certain freedoms (to those, like Stallman, who aspire to such principles in software), and have a necessarily depressive effect upon economies and livelihoods, in contravention of principle #6.

Klemens (2006) points out that the very worst patents are often cited as an argument against the ethicality of all software patents: his favorite example is a patent granted in 2002 for three short lines of JavaScript code, a language available for use since 1995 (p. 2). It would be unfair, however, to judge all software patents, or the patentability of software, based upon a portfolio of errors by the United State Patent and Trademark Office. Is there any ethical good that may be derived from software patents broadly?

It is necessary at this point to consider principle #4, which weighs the ambiguous claim of programmers or businesses upon a technology or technological idea against the immeasurable claim of other businesses and the consuming public to interoperability. This piece is being written in a file format called Open Document Format (compare to Microsoft’s “OLE2″ format, used in its popular productivity suite, reflected in “.doc” and “xls” files), in a free program called OpenOffice.org. This program also has the ability to open, edit, or create documents in Microsoft’s proprietary format; were OpenOffice.org not under the protection of the large software vendor Sun Microsystems, it is likely that the developers of the program would be sued for patent infringement, since Microsoft has a number of patents which ostensibly pertain to the OLE2 document format.

The chilling effect that such a legal situation has upon development of third party programs which can interoperate with Microsoft’s file format is well-known in software development circles. It effectively means that Microsoft is remunerated twice: one stream of revenue comes from the sale of its productivity suite (the quality of which directly affected the ubiquity—and market force—of OLE2 files), and the other—due to patents upon the data structures of OLE2—from companies which choose to license the specification rather than risk legal battles.

There is little question that Microsoft deserves the first stream of revenue, which is related to a copyrighted product (a “physical” good) that it sells; that some portion of the sales may have to do with the ubiquity (read: business necessity) of its file format is aside from the point. The fact that the company’s patents effectively prohibit competition by making interoperability cost-inefficient is, while a net economic good for Microsoft, an ethical downfall for everyone else. Lawrence Lessig, a well-known proponent of more “open” practices pertaining to intellectual property, insists that although official case law has yet to demonstrate the validity of the previous statement as measured by favorable judgements, common sense and observation indicate that these kinds of patents primarily serve as a way to keep “early innovators” safe from later, third-party progress (Babcock, 2005).

Assuming some unquantified validity to Microsoft’s patents upon its OLE2 format, how does that compare to a public need for unfettered access to interact with that format? Rawls argues unequivocally that justice “does not allow that the sacrifices imposed on a few are outweighed by the larger sum of advantages enjoyed by many” (1999, p. 3). In the tradition of Omelas, so Rawl’s sense of justice is inflexible (or so he supposes) when it comes to the inalienable rights due every man. Interoperability and the public good, Rawls may argue, are important, but should never be sacrificed if the right of the programmer/inventor to profit from his intellectual labors is to be infringed.

The hypothetical dystopia described has no boundaries: it is essentially a view of a communistic system in which the right to personal property is negated in favor of public benefit from shared resources. Though a different argument entirely, such systems have failed to produce thriving communities achieving a critical mass in the lengths to which it will go to redistribute wealth. However, the argument against software patents is not, at heart, a call for intellectual property to be redistributed to bolster a common good. Even if the dense calculus of utilitarianism was shown to be unarguably true, no ethical principle mandating the forcible forfeiture of IP is likely to be justified by any mainstream ethicists. The argument against software patents is that they do not represent intellectual property at all, but rather a manipulation of the legal system to achieve particular economic ends; it is possible, since the criteria do not require implementation, to patent software which hasn’t been written (Klemens, 2006, pp. 3-5).

The question initially posed was the degree to which patents are ethical or unethical. Assuming that Rawls’ theory of justice as morally paramount—as embodied in principle #3—is significantly correct, and understanding the particular vagaries of software and its algorithmic nature, certain conclusions may be drawn.

First, existing copyright laws do well to cover a content creator’s claim to his or her labor; these laws cover—and are perfectly sufficient for—the assertion of ownership over computer programs, which are themselves physical expressions/implementations of content. In this way, the right to personal property, intellectual or otherwise, is unabridged.
Second, the application of this principle into software written under the auspices of business allows proper remuneration for investment, and contributes to overall economic efficiency and therefore a public, moral good.

Third, within this ecosystem of software, patents are extraneous, serving no apparent purpose other than artificially extending the existing copyright protection to cover content which has not been written, cannot be written, or can be written in different ways (and properly so).

Unambiguously, then, software patents are unethical, serving to abridge the rights of content creators to implement general principles in software code, to enter into competition with existing market constituents and improve the market, and to raise the overall quality of the software ecosystem—and, by extension, allow the public benefit from these improvements.

References

Allison, J. R., Mann, R. J., & Dunn, A. (2007). Software Patents, Incumbents, and Entry [Electronic version]. Texas Law Review, 85(7), 1579-1625. From Business Source Elite (25978301).
Ante, S. E., Brull, S. V., Herman, D. K., & France, M. (2000, August 14). Inside Napster. BusinessWeek, 112-120. Retrieved March 15, 2009, from Business Source Elite (3394425).
Are Efforts to Extend Patent and Copyright Laws Good for Business or Good for Society? (2001, July 4). Retrieved March 17, 2009, http://knowledge.emory.edu/article.cfm?articleid=363
Babcock, C. (2005, April 7). Stanford Law Professor Raps Patents As Barrier To Innovation. InformationWeek. Retrieved March 26, 2009, http://www.informationweek.com/news/software/showArticle.jhtml?articleID=160502321
Bessen, J., & Meurer, M. J. (2008). Patent failure. Princeton: Princeton University Press.
Guntersdorfer, M. (2003, March 21). Software Patent Law: United States and Europe Compared. Retrieved March 17, 2009, http://www.law.duke.edu/journals/dltr/articles/2003dltr0006.html
Heisey, P. W., King, J. L., Rubenstein, K. D., & Shoemaker, R. (2006, March). Government Patenting and Technology Transfer. Retrieved April 13, 2009, http://www.ers.usda.gov/publications/err15/err15.pdf
Klemens, B. (2006). Math you can’t use: patents, copyright, and software. Washington, DC: Brookings Institution Press.
Lemos, R. (2004, February 12). Microsoft probes Windows code leak. Retrieved March 19, 2009, http://news.cnet.com/2100-7349_3-5158496.html
Lenk, C., Hoppe, N., & Adorno, R. (2007). Ethics and law of intellectual property. Burlington: Ashgate.
Patent Asserted Against JPEG Standard Rejected By Patent Office As Result Of PubPat Request: Public Interest Group’s Review Results in Broadest Claims of Forgent Networks Patent Being Ruled Invalid (2006, May 26). Retrieved March 16, 2009, http://www.pubpat.org/Chen672Rejected.htm
Rawls, J. (1999). A theory of justice. Cambridge: Hardvard University Press.
Schultz, W. J. (2001). The Moral Conditions of Economic Efficiency. Cambridge: Cambridge University Press.
Stallman, R. (2005, June 20). Patent absurdity. The Guardian. Retrieved March 17, 2009, http://www.guardian.co.uk/technology/2005/jun/20/comment.comment
U.S. Patent Statistics Chart Calendar Years 1963 – 2008 (2009, March 26). Retrieved March 27, 2009, http://www.uspto.gov/go/taf/us_stat.htm

Tracking LZMA efficiency

Ben — Tue, 10 Feb 2009 04:51:26 +0000

I’m a big fan of 7-Zip. It isn’t the best-looking application ever written, but that could be because its creator, Igor Pavlov, is concerned much more with its compression methods than its interface. 7-Zip has its own container format, but more important is the LZMA compression algorithm that Igor wrote and put into the public domain.

I decided to do some quick and dirty benchmarks to track the progress of LZMA/7-Zip over time. I went back as far as Igor supplied binaries, including one from the very old 3.x series. Rather than test every single release between then and now, I used only “stable” releases, with the exception of version 4.65, which is the latest version of any sort, as well as 4.66, which uses an alpha version of Igor’s new LZMA2 codec (and, as you’ll see, provides definite performance improvement).

I used Igor’s Timer utility to time the process (global time was reported). The corpus in this case was the Linux kernel source, v2.6.28. I conducted these tests on a RAM disk to eliminate hard disk latency issues (especially for decompressions, which improved by about 25% from my initial HDD-based tests). My rig is a Intel Core 2 Quad Q6600 [2.4Ghz], with 4GB of RAM (one dedicated to the RAM disk), running Vista SP1 x64.

The command line setup was an approximation of the 7-Zip GUI’s “ultra” settings: -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on, letting the archiver auto-choose the number of threads to spawn.

The Data

LZMA Efficiency
7-zip version	encoding time (s)	decoding time (s)
3.13	541.271	43.379
4.20	531.457	44.040
4.23	527.871	42.425
4.32	341.290	42.126
4.42	219.451	42.211
4.57	174.064	44.163
4.62	170.973	42.836
4.65	170.917	43.058
4.66 (lzma2)	126.259	46.663

The Analysis

Without conducting a more thorough battery of tests on a variety of different configurations, it’s difficulty to say with certain just where the performance improvements came from, be it better using of threading or multiprocessors, general algorithmic improvements, or something else. I also don’t know if the performance increases we see reside in improvements to LZMA itself as Igor was finalizing it, or just the code quality of 7-Zip, which implements LZMA.

In any case, the improvements since 3.13 are very clear (remember that lower is better), at least for compression, and for “ultra” settings. Decompression remained largely similar, which surprised me. Some of these results might be directly tied to the number and type of files that were compressed in the case: 4.66, for instance, improves decompression speed for uncompressable files, but no such files exist here since it’s source code.

Hats off to Igor Pavlov for his steady improvement on both a really great compression standard and one of my favorite pieces of software for Windows.

FLAC compile benchmarks

Ben — Sun, 21 Dec 2008 16:26:24 +0000

FLAC is a cross-platform codec, but when it comes to Windows, one has a pretty wide range of compiles. Some are more optimized than others.

I first got the idea for this benchmark when I stumbled upon a native 64-bit FLAC executable for Windows. Curious, I did a quick and dirty test against the canonical build for Windows and found that while encoding times were similar, decoding times were considerably faster.

To figure out why this is so (the 64-bitness or something else), I quickly pulled some some additional compiles and benchmarked them against a few different samples.

I have three different samples I used, ripped in WAV format as single files:

Lateralus, by Tool
Californication, by the Red Hot Chili Peppers
Bruckner’s 9th Symphony, by Giulini/Vienna Philharmonic, 1989

I’ve included a table for each sample, split into encode/decode times and compilation.

The builds I included were:

The canonical build from the official FLAC website
MSVC8 (Microsoft) compile from RareWares
ICL 9.1 (Intel) compile from RareWares
64-bit compile (MSVC?) from HardwareHacks 2000

My test system is a Intel Core 2 Quad Q6600, with 4GB of RAM, running Windows Vista x64 SP1, with all available patches. The timer used was by Igor Pavlov (the developer of 7-zip). Reported times are global (as opposed to kernel, user, or process times).

FLAC build benchmarks (time in seconds)
FLAC build	Action	canonical	MSVC8	ICL9.1	x64
Lateralus	encode	140.198	124.146	121.821	135.206
Lateralus	decode	36.770	17.098	16.708	18.315
Bruckner	encode	115.331	104.708	107.984	121.119
Bruckner	decode	33.384	33.338	30.030	31.387
Californication	encode	106.954	98.283	90.309	100.668
Californication	decode	26.973	25.972	25.943	27.238

It turns out that the x64′s advantage when it comes to decoding isn’t due to its 64-bitness, but probably because it includes some extra optimizations (SSE?) that are inherent ot a 64-bit compilation.

The clear winner in both encoding and decoding is the ICL9.1 compile (I wonder if a 10.1 compile would be even better); at least, it is so on my machine. The x64 compile was sometimes better, sometimes worse than the canonical compile. Clearly, there’s an early benefit from some optimization (probably SSE routines), along with a small margin of benefit from using Intel’s superior compiler. 64-bitness is beneficial for FLAC processes only insofar as it necessarily encompasses these routines as part of its instruction set.

If you’re looking to use a particular build of FLAC, I suggest you mosey on over to RareWares and john33′s ICL build: the resulting files are a few KB larger, but the processes itself is noticeably faster.

GNOME Audio Player Shootout Revisited

Ben — Fri, 19 Dec 2008 16:13:56 +0000

It’s been close to two years since I wrote GNOME Audio Player Shootout, a visual and textual comparison of some the best available audio players for the GNOME desktop.

As is usually the case in the world of free software, a lot has happened since then (and yet, in a strange way, things have stayed exactly the same). I decided to revisit some of those players and see how they’ve progressed. Some of them listed last time haven’t seen any appreciable development, and have been left off.

I realize that I am totally ignoring the daemon-based players (read: Music Player Daemon, XMMS2); this is by design, since those players open up a whole new can of worms. Suffice it to say that if you’ve decided on and XMMS2 or MPD-based player and successfully configured it, you probably don’t need any advice on choosing software.

The following programs will be covered in this review (development versions):

BMPx (0.40.14)
Rhythmbox (0.11.6)
Exaile (2.99.1-svn)
Banshee (1.4.1)
Quod Libet (2.0)
Decibel (1.00)
Songbird (1.0)
Listen (0.6~svn1044)

All of the testing was done on a fresh install (and update) of Ubuntu 8.10 in VirtualBox, using a small representative sample of my music collection (some modern, some classical, in Vorbis, MP3, and FLAC).

Now Reading for WordPress 2.7

Ben — Mon, 15 Dec 2008 06:56:50 +0000

This page has been moved to a more permanent spot. Please go there.

In defense of open models

Ben — Wed, 22 Oct 2008 20:50:54 +0000

Andrew Keen has no idea how open models work.

In his latest article, he pontificates that the recent economic downturn is a death knell for community-supported or community-built programs/sites/&c.

So how will today’s brutal economic climate change the Web 2.0 “free” economy? It will result in the rise of online media businesses that reward their contributors with cash; it will mean the success of Knol over Wikipedia, Mahalo over Google, TheAtlantic.com over the HuffingtonPost.com, iTunes over MySpace, Hulu over YouTube Inc., Playboy.com over Voyeurweb.com, TechCrunch over the blogosphere, CNN’s professional journalism over CNN’s iReporter citizen-journalism… The hungry and cold unemployed masses aren’t going to continue giving away their intellectual labor on the Internet in the speculative hope that they might get some “back end” revenue. “Free” doesn’t fill anyone’s belly; it doesn’t warm anyone up.

There are really two broad fallacies that need addressing here. The first is Keen’s use of the word “open source,” which here is a misnomer. He never mentions Linux, Apache, or other open source programs which always have and will continue to have a dedicated base of programmers, most of whom work on it in their spare time, without any remuneration except personal pride and the esteem of their peers. It need hardly be noted that an economic downtown is likely to increase interest in open-source software, as it likely reduces operating costs for businesses.

No, what Keen means when he says “open source” is free-as-in-beer services, often serving liberally-licensed content; Wikipedia’s content is not open source (there’s no source to open), but it is available under the GNU Free Documentation License, which is something like a liberal Creative Commons license. Perhaps Keen has a sheet of words vaguely associated with Web 2.0 and just likes to throw them around in case his readers are too stupid to know better.

But then comes the bigger fallacy—i.e. in an economic depression, the things that motivated people to contribute to social sites and content servers will vanish entirely. Nevermind the fact that most of these services don’t necessarily imply the forfeiture of copyright; or that many already include ways to monetize one’s content. No, Keen fundamentally misunderstands why people contribute to things like Wikipedia. This isn’t a recent phenomenon borne on the largess of the Web 2.0 bubble; people didn’t start contributing to Wikipedia simply because they were so rich from their day jobs that they felt like giving something back. No, people like being a part of something. They like attaching their name to good work, free or not.

This is all a very roundabout way of saying that Keen couldn’t be more wrong; he apparently is crass enough to believe that anything one does can and should be tied to monetary compensation. I imagine he gets paid for his articles for Internet Evolution (if he was doing them pro bono, it would certainly speak volumes about his argument); perhaps he overestimates the value of his labor.