Which is the best low-bitrate audio compression algorithm?
OGG vs. MP3 vs. WMA vs. RA
This article has not been updated for more than 2 years. It contains obsolete information that could mislead the readers. It will be updated again as a completely new blind test during 2005 Q2. I apologize for not keeping up to my last promise, but work was heavy on my during the last months. Thanks for keeping up.
by Panos Stokas panos@ekei.com
Last update in March 2002 adds MP3 pro, new Ogg Vorbis and a Q & A section.
PRIVACY CONCERN:
If Windows Media Player 7 or higher is installed on your system, using this page will reveal personal information to Microsoft while wasting CPU and hard disk resources. If your system serves a critical application, I can take no responsibility for any problems caused by the aforementioned software. This page works best with Winamp.
OSSY AUDIO COMPRESSION can be quickly described as the JPG of music: for a minor loss of quality we have a sound file which is many times smaller to the original and it can be easily broadcasted through the internet or transferred on portable media. But there is one problem: while JPG is a widely accepted standard, audio compression is a new concept and there are still various algorithms contesting for the public approval. The purpose of this study is to find the best algorithm, one that combines high quality at low bitrates and works everywhere.
Test in blind
This comparison is based on the argument that human ear is the final judge of sound quality and it was performed as a "blind test". The test procedures are available in PDF format.
By definition, it is impossible to make an audio format that is both lossy and high fidelity. We thus have stayed away from a high fidelity setup and chose our testing environment to be a typical multimedia system with low cost speakers. It should come to no surprise that our "CD quality" nomination is not that of a good hi-fi system but rather of a CD being played in our system.
We tested upon a classical music sample (copyright by Deutsche Grammophon, no permission to use) altered with Goldwave to apply phase and stereo effects. This formed a very complex wave in matters of frequency, stereo image and dynamic range which proved to be an excellent tool in uncovering weaknesses in the various codecs!
You may also download all the samples in one archive (2.4 MB) for off-line listening. Please, do not hesitate to comment: reader's feedback is taken very seriously and parts of this document (including the scores) were changed considerably after such discussions.
A closer look at the algorithms
MP3 Pro (MP3+SBR)
MP3 Pro is an evolution of the MP3 algorithm at the labs of FhG). By combining a very low bitrate MP3 file with spectral band replication data (heavily compressed high frequencies), we get an MP3 file with full bass and precise treble. The approach is interesting because the MP3 part of the MP3 Pro file is still compatible with all MP3 players. In order to enjoy the full range of MP3 Pro however, the free decoder must be acquired.
How does MP3 Pro stand among the other codecs then? In regards to ISDN-class bitrates, it simply crushes all competition. Neither Windows Media, nor Real Audio can hope to achieve the clarity and lack of artifacts of our 56kbps MP3 Pro clip. Yet, MP3 Pro was unable to compete the new MP3 implementations of the same Institute (the Variable Bitrate FastEnc) at higher bitrates and it performed rather poorly in the phoneline speed arena. Instruments were very clear, but the voice lacked in depth.
MP3 Pro is a great evolution, as it was WMA, as it was Real Audio and MP3. It also focuses on a field that will soon be the defacto connection speed, that of 56kbps. MP3 Pro will never be much talked of, because it is not free but its technical aspects are too strong to be ignored. We will be hearing it, but we shall not be seeing it too much.
MPC and Ogg Vorbis
MPC or MPEGplus is an MPEG1-Layer 2 derivative, parallel to MP3. The encoder is currently free but it will turn to shareware to compensate for its Layer 2 license. The decoder is a free Winamp plugin.
Although it does not work well with low bitrates, this new algorithm seems promising. It was quite capable of analyzing the complexity of the testing sample and would persistently encode at 170kbps unless it were forced to the lowest possible of a "thumbnail" quality that was still good. That leads to believe that the author's claim of highly tuned psychoacoustics is true and that MPC will find a place in high quality formats in less than 3 years.
Parallel to MPEGplus (in algorithm terms) is the unpatented Ogg Vorbis which strives to "replace all proprietary, patented audio formats". Although much improved in the last months, this format has not yet reached maturity and even though the authors are on the final stage of version 1, I think they still have plenty of work ahead. Its low bitrate algorithm cannot compete Windows Media and at higher bitrates it competes with Lame MP3 which is free as well. And in the very high bitrates, audiophiles rave about MPC. In order to gain popularity, Ogg Vorbis must focus its efforts on the phoneline bitrates for internet radio appliances.
New information: Ogg Vorbis now serves as an excellent alternative to patented formats. It has reached maturity and it must be considered as an open source codec that runs virtually anywhere (from mobile phones to linux systems) with better quality than mp3, a unicode comment system and gapless playback. To be analyzed in the next article
Windows Media Audio
You may have encountered Windows Media Audio under the name of "ASF". ASF is Microsoft's specification for streaming -- essentially a wrapper format. WMA is a lossy audio compression algorithm which works only as part of the Windows media player and not as a standard Windows codec (so that you may not convert WMA files to other formats).
In 1998 I applauded the superiority of WMA in the arena of streaming media. With clear stereo music even at 22kbps, WMA is the only way to get an acceptable quality in typical phoneline speeds. Its low CPU requirements and broad availability have turned it, as I predicted, to the Real Audio killer!
Stereo image and voice is not as good however, so Microsoft's claim for FM quality at phoneline speeds is not true. Also, Microsoft's second claim that WMA provides CD quality at 64kbps is entirely wrong. Not only does the 64kbps WMA lack in violin detail, but even at 128kbps (which Microsoft touts as "audiophile") suffers from overbrightness.
Real Audio 8.5
Real Audio is one of the oldest formats for compressed audio; it was first designed for voice applications on the web but later it developed music and video algorithms as well. The free Real Player has been implemented for many platforms but it has become a spyware and its quite demanding on the CPU and therefore it is not available to as many systems as it used to be.
Real Audio is a reliable format and provides acceptable quality in all given bitrates and all kinds of music. Version 8.5 has brought a few changes, some of which are not positive. The good news, Real Audio now provides CD quality bitrates. At phoneline speeds (20 and 34kbps) G2 songs are clear and rich in high frequencies -- indeed very close to FM radio. At 90kbps and above, it is difficult to discern quality loss, though apparent is some overbrightness. I also need to comment on its exceptionaly fast encoding and efficient streaming algorithms -- Real Networks' experience with internet broadcasting issues is obvious.
This is a double-edged knife, however, since many people are prejudiced against G2 in a high quality format role. In a very interesting article about Real Audio, David Fiedler of Homerecording.com concludes: "Why MP3 gets all the publicity, I'll never know! There are far more people with RealAudio players than MP3 players. Give it a shot". Had Real Player not been so CPU intensive, and had it been a standard Windows codec (for saving .wav files) it would have been my favorite high bitrate format.
MPEG Layer III
MP3 is the closest thing to JPG; developed by Fraunhofer Research (the leading MPEG research institute at Germany and also the people who build the MP3 codec for Internet Explorer), the algorithm is open and free for anyone to use. MP3 has become a source of great controversy as it facilitates the transfer of CD quality music on portable media and the internet. Despite the complains of commercial entertainers MP3 has become a standard in music broadcasting. Supported in popular packages like Winamp and Internet Explorer, MP3 only requires a 486 to play on average bitrates.
MP3 is still the top quality standard. It is the only guarantee to get a high quality song that will play everywhere. To all my tests, we have been completely unable to separate the original from the compressed. However, lower bitrate MP3s suffer from a high frequency loss that becomes insufferable on anything less than 44kbps. MP3 files will not play clearly on phone line bitrates.
MP3 still develops. Both Fraunhofer Research and the general public have presented algorithms that create much better results than previous encoders. I tried Fraunhofer's new Fastenc engine (found at Musicmatch Jukebox or Nero) and I was surprised with the quality on an average of 85kbps stream! The Lame Encoder performs almost as well, with CD quality at 113kbps average.
Obsolete codecs
The following algorithms are dead heroes fallen in the field of battle. Either because of lack or support or simply because of their inferiority, these algorithms are no longer discussed in this article. Honours must be payed though: I remind to my readers that the original title of this study were: "MP3 vs VQF".
- MOV
- The QuickTime audio codec, QDesign2 comes at a huge download combining the MVP player and the new Quicktime version. Overcoming the very slow encoding, we enjoyed great clarity and detail on the strings even at low bitrate files but we were dissapointed to find that even at 64kbps human voice was problematic.
- VQF
-
- NNT was the first company to produce a serious MP3 contestant. At the time of its introduction to the public, VQF provided the best quality in the 96kbps range. However, NNT did not support their format with good software. For that reason VQF's popularity has waned almost completely in the 99s and from now on it may be considered a dead case.
- AAC
We've been reading about AAC from the early 1998's. There is even an ISO comparison which proves (with statistics...) AAC's superiority to MP3. However, we have not found a single implementation that did not produce serious amounts of distortion in our testing sample. Nobody seems interested any longer.
There had been great developments on AAC after this article was published. These will be analyzed in the next version of the article
Which is best?
To repeat my introduction, the purpose of this study is to find the best algorithm. It is now obvious that no algorithm is perfect; you must then choose the best one for your application.
Table 2. Best choices of algorithm per appliance
| Appliance | Main requirements | Format of Choice |
|---|
| Share through CDs | Quality, Player Availability | MP3 |
| Share through internet | Quality, Compression | MP3, MP3PRO, WMA |
| Portable players | Compression, Quality | MP3PRO |
| Broadcasting through internet | Player Availability, Compression | WMA, RA-G2 |
If you need to burn some music files on your CDs you should have no problem using a high bitrate. If you need to publish your music, you must make sure that not only you offer CD quality but you are using a popular format as well. MP3 will serve will both needs, with its high quality being available to all platforms.
I now choose MP3PRO over WMA for portable players: quality requirements are not very strict and storage space is quite limited. With its excellent 56kbps implementation, the Fraunhofer codec would provide the best solution if it were not patented and restricted.
Internet broadcasting poses a great challenge in that you must provide FM radio quality on a wide public through slow, phoneline connections. I think ASF/WMA is the best answer here, being available to all new Windows platforms and sounds OK. For voice broadcasting however, Real Audio sounds much better and thus it is the best choice.
Further reading
- The Group Listening at 128 kbps by ff123 is the most recommended link. It treats another popular field of bitrates and it is performed in a more thorough manner than the one you are reading. Also, the sample used in this comparison was also analyzed by ff123 who has published an extensive Discussion of Audio Compression at his site. This is definitely worth looking at.
- The Musepack site and the MPEGplus discussion board are prejudiced against WMA and MP3, but still they are interesting places to find information about the MPC format.
- Andrei Aspidov is bringing forward an Ogg Vorbis vs LAME comparison. Excellent analysis, although as I've stated in the beginning I disagree with the philosophy.
- Real Networks fabricates an Anti-ASF page where they "compare" MS Audio against G2. Apparently they couldn't find one stereo sample that sounds better in Real Audio, so they had to use a monaural sample for their deceitful test.
- The Usenet, and specifically rec.audio.*, is the only serious place to discuss audio issues. Web forums are a waste of time.
Downloads
- Linux is getting more and more attention from music professionals. I have found Sound.condorow.net to be an excellent source of software links and information for this free platform.
- To play MPC and Vorbis files you need appropriate plugins for the Winamp player. MPC plugin can be found at Musepack and the OGG plugin at its official site
- I can't provide a direct link to the free Real Player and Real Producer. You should find them somewhere at Real.com hidden behind screenloads of Real Player Plus orders...
- Fraunhofer's new MP3 encoder 3.1 solves phase and duration problems and provides the best quality in the MP3 arena (whereas Xing generates samples of lower duration while Bladeenc grossly distorts music in low bitrates).
- The WMA encoder and the Media Player is found at Microsoft's Media Tools, but it does not work as a standard windows codec; you can't have Sound Recorder save WMA files.
- Finally, you may wish to try RAR an excellent non-lossy multimedia compression format (the archive of samples on this page was compressed with RAR).
Q&A
- How should I convert WMA to MP3?
- Use an old version of Winamp which allows redirection of WMA output to the hard disk. Microsoft forced Nullsoft to disable this future because Microsoft does not want you to be able to convert your WMA files
- How should I convert Real Audio to WAV?
- If you have a soundcard which allows recording of the digital signal, enable it in the Volume Control and use Goldwave to capture it.
- Any information about ASF video?
- I apologize that your search engine has lead you here. Please use more specific keywords, such as "ASF video" (without the quotes).
- What about extremely low bitrates?
- Take a look at this bibliography and at DVSI. The performance of their IMBE codec is quite impressive at 5kbps!
Notes
- Purpose:
- I have received caustic comments by fans of various formats on whether such a comparison is necessary. PC Magazine, goes even further by proclaiming that their "Labs tests show that even to an audiophile, the differences are virtually imperceptible". Yes, testing high bitrates such as 128kbps shows little difference. However, a few of my dear readers and PC Magazine fail to see the importance of low bitrates in internet broadcasting. Anyway, PC Magazine seems quite determined in awarding Real Audio as the best streaming format so the reasons for such a ridiculous statement are clear.
- High Fidelity:
- You cannot achieve high fidelity on a computer by simply connecting line out to a good hi-fi set. You must use your soundcard's digital output with a separate D/A converter, and set it far away from the noisy PC power supply. Be prepared to get overwhelmed by fidelity loss in low bitrates...
- Phase and stereo effects:
- It has been questioned whether such artificial effects would lead to a precise comparison. As long as music is concerned, I think yes. Such effects are being used in contemporary music for decades. Stereo effects in particular, reveal badly implemented "pseudo-stereo" algorithms, while phase effects would not be encoded correctly by old MP3 encoders.
- CD quality
- The precise definition for CD quality is that of 44.100 Khz / 16bit data played on an ultra expensive high end system. I am using a wrong definition in this article. I started using it when Microsoft proclaimed that 64kbps WMA provides CD quality. I will continue using it as long as Microsoft and other corporations insist on abusing it.
