BBH Central IconBBH Report Home PageSandy and Dave
  CENTRAL home  |   REPORT home About/Contact Us  |   Subscribe  |   Index by Topic  
The February 17, 2004 Issue Provided by System Dynamics Inc.
Table of Contents Print this article Email this article to a friend

Wi-Fi Evaluation Round Two

A lot has changed since our first evaluation of Wi-Fi products eight months ago. Many older products have been upgraded to full compliance with the 802.11g standard. Lots of new products have come to market, some with Wi-Fi certification. With the products stabilizing, we decided it was a good time to resume our testing and to report on all three versions of Wi-Fi.

"Wireless Is Magic" -- Our Evaluation of 802.11g Wi-Fi (BBHR June 2003) described our first Wi-Fi evaluation. That article reported on our tests of four then-new 802.11g devices: two access points and two PC card notebook adaptors, all based on the same Broadcom chip. We measured throughput at nineteen locations around our house while transferring large files between PCs.

802.11g is the newest "flavor" of Wi-Fi. At the time of our earlier tests, the standard had just been approved, and all the tested devices were based on earlier drafts of the standard; none had completed Wi-Fi Alliance certification testing. Now nearly all devices comply with the final standard, and some have passed Wi-Fi tests.

This report covers the first part of our Round Two testing: "baseline" measurements with a wireless notebook adaptor plugged into a notebook PC in the same room as the access point. This provides a good indication of the maximum throughput possible with each combination of an access point and a notebook adaptor. The test results are summarized below; the detailed results are reported in the Round Two Wi-Fi Baseline Tests section of our website.

In an upcoming issue we will report how the throughput is degraded by distance, walls, floors and other obstacles.


Three "Flavors" of Wi-Fi

To remind our readers, "Wi-Fi" wireless networking is based on a set of standards collectively known as "IEEE 802.11". Today's products are based on three "flavors" of 802.11:

  • 802.11a operates in the 5 GHz band and operates at a maximum speed of 54 Mbps
  • 802.11b operates in the 2.4 GHz band and operates at a maximum speed of 11 Mbps
  • 802.11g operates in the 2.4 GHz band and operates at a maximum speed of 54 Mbps

The "11a" and "11b" standards were both published in 1999, while "11g" was published in 2003. 11g products are all back-compatible with 11b. 11a runs in a completely different frequency band so "11a-only" products won't work with 11b or 11g.

Wi-Fi certification logo --> Click for larger pictureTo confuse the customer, these products now come in five different combinations: 11a only, 11b only, 11b/11g, 11a/11b combo and 11a/11b/11g.

The Wi-Fi Alliance has tried to clarify this with its logo program, but it can be difficult for consumers to tell the products apart. The picture shows a Wi-Fi logo for a product which supports all three versions and has been certified for WPA (a new security specification). Some "combo" products support multiple standards but have only been certified for one.


Our Baseline Tests

Five access points on our couch --> Click for larger pictureWe tested six notebook adapters --> Click for larger pictureWe are conducting an extensive series of tests involving five access points and six notebook adapters, covering all three flavors of 802.11. Most of the tested equipment was loaned to us by Linksys and SMC Networks.

When completed, these tests will include three sets of measurements:

  • Maximum throughput with the notebook adapter close to the access point in the same room: this will establish a "baseline" measurement for further tests.
  • Throughput at each of 19 locations in our house, showing how the maximum throughput is degraded by distance, walls, floors and other obstacles.
  • Internet file download speed with and without wireless networking.

We have completed the baseline measurements and summarize them below. We expect to complete the other two test series soon and report on them in an upcoming issue.

Physical rate and throughput

The quoted speeds of 11 Mbps (802.11b) and 54 Mbps (802.11a and 11g) are maximum speeds, representing the maximum bit rate through the air (this is sometimes called the "physical" or "PHY" rate). The maximum rate at which data can be transferred from one computer to another -- called the "throughput" -- would be much more useful, but is not published. Our "baseline" tests were intended to estimate the maximum throughput for each version of Wi-Fi.

Our throughput measurements are quite a lot lower than the quoted physical rate; we calculated the "efficiency" of these products at between 35% and 53%. This did not surprise us, since the 802.11 protocol includes a lot of "overhead" -- airtime consumed by "handshaking" rather than useful data.

Updating firmware and drivers

Although we knew that the vendors had released updated firmware and drivers for many of the devices we were testing, we chose to run a preliminary series of baseline tests with the devices as shipped from the manufacturer. It did not surprise us that some "draft 11g" devices from one vendor did not work properly with "draft 11g" devices from another.

We then tried to update all of the devices, installing new firmware for the access points and new drivers for the notebook adapters. We found this to be a rather difficult process, requiring several steps:

  • Many products have been sold under the same product number with several different versions. There are different downloads for each version and it's important to match the version number with the right download page.
  • Firmware and drivers are usually downloaded in a compressed format. There are different procedures for uncompressing each format.
  • The upgrade procedure may be documented in the manual, on a web page, in a note as part of the download--or not at all.
  • Although Windows XP has a built-in "driver upgrade" feature, some PC cards cannot be upgraded this way.

We succeeded in upgrading all of the access points and most of the PC cards. After updating firmware and drivers, all 11g devices--even from different manufacturers--worked properly together.

Since we sometimes found better results after updating firmware and drivers, we recommend upgrading. This is particularly important for 802.11g devices, many of which were shipped with "pre-standard" firmware or drivers.

SMC2336W-AG with SMC2804WBR router, with firmware 1.02 -- before router firmware upgrade --> Click for larger pictureSeveral of our test runs graphically illustrate the improvement from upgrading firmware.

As an example, we tested the SMC2804WBR router with an SMC2336WAG PC card first with the original router firmware as shipped from the factory and then with the latest firmware after an upgrade.

The "run chart" on the right shows time for each run with the original firmware as shipped from the factory.

The results appear quite "jittery". The throughput is about 14.8 Mbps.

SMC2336W-AG with SMC2804WFR router, after router firmware upgrade --> Click for larger pictureThis shows the same router and PC card after upgrading the router firmware.

The difference is quite dramatic. The throughput is now 19.0 Mbps, an improvement of more than 25%.

In addition, most of the new 802.11 devices support an improved form of security known as "WPA", which is considerably more secure than the WEP security in the original Wi-Fi. Many of the upgrades add WPA support for older devices.

While new production products include support for the final 802.11g standard and WPA, earlier-production products may still be in inventory. These will need upgraded firmware or drivers.


Baseline test results

As in our earlier tests, we measured the throughput while transferring large files between PCs. Unlike our earlier tests at nineteen locations around the house, we made all these baseline measurements with our notebook PC close to the access point in the same room. This provides a good indication of the maximum throughput possible with each combination of an access point and a PC card notebook adapter.

As before, we did these tests in only one house, with one sample of each product and with one particular test tool of our own design. We have no way of knowing how well other samples and other products in other homes will perform.

Our test results are reported in detail in the Home Networking - Wi-Fi Evaluation section of our website. The Round Two Wi-Fi Baseline Tests page includes the measured throughput for each combination of an access point and a network adapter, with many run illustrated by charts.

Here's a summary of what we found:

  • The 802.11b tests averaged 4.5 Mbps throughput, with a range of 3.4 to 5.8.
  • 802.11g averaged 18.9 Mbps, with a range of 16.6 to 21.3.
  • 802.11a averaged 14.7 Mbps with a range of 7.4 to 20.3. The 7.4 Mbps probably indicates a defective PC card - excluding it, the average is 18.4.
  • We calculated the efficiency for each protocol as the ratio of our measured average and best throughput to the maximum physical rate. 802.11b efficiency averaged 41% with a maximum of 53%; 11g averaged 35% with a maximum of 39%; 11a was similar to 11g.
  • Many test runs showed intermittent periods of poor throughput, often lasting 10 seconds or longer during a 1000 second test run. These "spikes" in measured transfer times lowered the measured throughput, sometimes substantially.
  • In some cases, upgrading firmware (for access points) or drivers (for PC cards) made a significant difference in throughput. Sometimes a firmware upgrade greatly reduced the "spikes".

These tests were intended to provide a realistic estimate of Wi-Fi throughput with wireless devices close to access points. The maximum speed of 802.11b is about 5 Mbps, while 802.11g and 802.11a are close to 20 Mbps, about four times faster.

More recent products -- access points and PC cards -- performed better than older ones when running 802.11b. This would indicate that wireless product designers and manufacturers are improving their skills at realizing the potential of the standards.

Many 11b/11g and 11a/11b/11g consumer devices are now on the market. The price difference between these devices and 11b-only devices is small, so we believe consumers should avoid 11b-only devices and favor those with 11b/11g and 11a/11b/11g.

( www.wi-fi.org ) ( www.linksys.com ) ( www.smc.com )