You are here: Central > Broadband Home Labs > Home Networking > Wi-Fi > Round Two Baseline Tests
Round Two Wi-Fi Baseline Tests
This page covers our Round Two testing in February 2004: the "baseline" measurements with a notebook PC close to the access point in the same room. This provides a good indication of the maximum throughput possible with each combination of an access point and a PC card notebook adapter.
[Editor's Note 3/9/2005: We performed our later Round Three testing with a different notebook PC, and did not see the "spikes" we found in Round Two. We now believe that some of the "spikes" on the run charts below appear to have been caused by the notebook PC used in these tests. We plan to re-run this test round with a different notebook PC, and will update this page once we complete these tests.]
These tests were intended to provide a realistic estimate of Wi-Fi throughput with wireless devices close to access points. The maximum throughput of 802.11b is about 5 Mbps, while 802.11g and 802.11a are close to 20 Mbps, about four times faster.
Our conclusions are based on tests in only one house, with one sample of each product. There is no way of knowing how well other products in other homes will perform.
The test procedure, detailed test results, and "run charts" for selected runs are provided below.
Here's a summary of what we found:
We have completed "baseline tests" measuring the throughput of thirty different two-node networks, each consisting of a Wi-Fi access point and a single notebook adapter. Each test run was performed with the access point on top of a couch and a PC card wireless adapter plugged into a notebook PC across the room on a desk.
All other wireless devices in the house were turned off. We do not own a 2.4 MHz cordless phone, and we did not use the microwave oven during the tests.
As soon as we started each test run, we left the room and did not return until the test run (about 18 minutes) was completed.
The diagram at the right shows the main floor of our house (the test locations page shows the other floors and the locations of all our tests). For Round Two, the five tested access points were each at about the same location -- labelled AP on the diagram to the right -- in approximately the center of the main floor of our house. The notebook PC with the wireless adapter under test was placed across the room at location 11; the distance from the access point to the notebook PC card was about nine feet.
We used HONEST to measure the file transfer speed. The test file is a 4.3 MB uncompressible MP3 file. Each test run consisted of transferring the file as many times as possible until the total transfer time exceeded 1000 seconds. In each case, the file was transferred from a hard drive on one of our servers; we measured the time required to read the file on the server and subtracted it from the test time in computing the reported throughput.
We calculated the throughput by averaging the transfer time over all the test passes, subtracting the fixed time taken by the server, and converting the net time to Mbps. We made at least one run for each combination of access point and PC card; when we made multiple runs, we averaged them.
Ethernet Run Chart
For each test run, we created a "run chart" plotting run time against start time for each pass of the test. These charts let us see how consistent the run time was from pass to pass.
Before we started the Wi-Fi tests, we verified our test procedure by conducting the same test over our Ethernet wired network.
The Ethernet run chart on the right shows the results of the test, plotting the run time for each pass against the start time.
Each test pass took a little less than half a second; more than two thousand copies of the file were transferred in our 1000 second test interval.
This test was quite close to a straight line -- most run times ranged between 0.47 and 0.51 seconds, with the longest run at 0.6 seconds.
The average time was 0.48 seconds, giving a calculated throughput of 102 Mbps -- about what we expected as the peak speed of a full-duplex 10/100 switched Ethernet network.
Wi-Fi Run Charts
We created a run chart for each Wi-Fi test run. Since all the runs were made with the wireless adapter in the same room as the access point, we thought that the run chart should be close to a straight line. None of the Wi-Fi tests was quite as straight as we obtained with our Ethernet baseline test, but some came close.
802.11b: For example, the SMC2780W access point with the Linksys WPC51AB adapter comes pretty close to a straight line. The "spike" at the beginning of the run may have been caused by standing between the access point and the PC card at the start of the test.
This test of an 802.11g/b access point with an 802.11a/b PC card shows that 802.11b was the only common standard. The throughput was measured at 5.69 Mbps, about the best we saw for 802.11b.
802.11a: As another example, the Linksys WAP51AB access point with the IBM a/b/g wireless adapter also comes close to a straight line. These devices could both operate with 802.11a, with throughput measured at 20.30 Mbps.
Many test runs showed intermittent periods of poor performance, often lasting 10 seconds or longer during a 1000 second test run. These "spikes" in measured transfer times lowered the throughput, sometimes dramatically. The WAP51AB-IBM a/b/g example shows two spikes.[see Editor's Note above]
The measured throughput of some test combinations was lower than we expected, and we looked more closely at the charts to see the cause.
Sometimes the charts appeared to indicate a stability problem, with significant variation in run time between passes. In some cases, upgrading firmware (for access points) and/or drivers (for PC cards) made a significant difference in performance.
As an example, we tested the SMC2804WBR router with an SMC2336WAG PC card twice--first with the original router firmware as shipped from the factory, and then with the latest firmware after an upgrade.
The "run chart" on the right shows time for each run with the original firmware as shipped from the factory.
The results appear quite "jittery". The average run time was about 2.5 seconds, the throughput about 14.8 Mbps.
This shows the same router and PC card after upgrading the router firmware.
The difference is quite dramatic, with most runs taking nearly the same time. The average run time has improved to about 2.0 seconds, and the throughput to 19.0 Mbps.
In other cases, we saw instability even after upgrading.
For example, the run chart on the right is for the Linksys WAP54G access point with the SMC2835W adapter operating in 802.11g mode.
This chart seems very "jittery" compared to some of the charts shown above.
This chart is the same Linksys WAP54G access point with a different adapter--the SMC2336W-AG.
We do not see the "jitter".
This chart is the same SMC2835W adapter with a different access point--the SMC2870W.
We see the same "jitter" problem.
This would appear to indicate that the instability is probably with the 2835W adapter.
As a final example, another PC card showed what appeared to be a particularly high level of instability, which we saw both in our original tests in June 2003 and in our current round.
The run chart for the Linksys WAP51AB access point and WPC51AB PC card operating in 802.11a mode is shown on the right.
The "Wi-Fi Run Charts" in the first section above showed the WAP51AB access point with a different PC card in 802.11a mode, and the WPC51AB PC card operating with a different access point in 802.11b mode. Since neither indicated any problems, there appears to be a problem with the WPC51AB PC card operating in 802.11a mode.
The following tables show the throughput for each combination of an access point and a PC card network adaptor, expressed in Mbps. Some test results are linked to run charts showing finer details of each test run. [See Editor's Note above]