A question we often get is, “how do you compare with speedtest.net?” We summarize the information here.
update: As of August 2015 the CIRA Internet Performance Test no longer has a Flash dependency and is based solely off web socket technology. This makes the test more cross-browser compatible.
A question we often get is, “how do you compare with speedtest.net?” It is a fair question because speedtest.net, and its parent company Ookla are the 800lb gorilla in the speed test category. We learned a lot from an article published by Steve Bauer from MIT a couple years back that compared speed tests, and wanted to summarize a few key points based on this information.
The referenced article tells us that the Ookla speed test is designed to deliver the maximum speed that you are getting. This is a perfectly legitimate result that is important for Internet users to know. It does not report on the all the detail performance they may want to know for all web browsing sessions. It is the reason Netflix makes such hay out of its reports on the ISP Speed Index where their published number, specific to their streaming video, doesn’t come close to Ookla’s results. (On a side note, Netflix’s analysis can be thought of as another resource to understand speed that just happens to use its own content as the testing tool)
Ookla is about quantity: it uses multiple TCP connections in parallel to maximize throughput measurement through brute force. They use multiple TCP sessions in order to calculate the maximum speed and dismiss 30% of lowest samples and 10% of highest, which essentially weeds out errors or congestion issues, dismisses retransmissions, packet loss, out of sequence packets, and pushes out as much traffic as possible no matter what.
The folks over at M-LAB, looked at this and advised that Ookla appears to have 90+ measurement instances located within the networks of Canada’s Access ISPs. While this is a very impressive number it does also mean that, if the user is using their ISP’s test or a nearby server for an Ookla test, then their tests are often conducted as close to the client as possible. Again, this is the best way to understand the top speed you expect to be getting because it gives the, “last mile” performance (the performance within the Access ISP). However, it is not necessarily how the Internet works, nor reflective of the full path performance or integrity.
On the other hand, the CIRA Internet Performance Test (i.e. the NDT test from M-Lab) measures the maximum capacity/throughput of a single TCP connection in the normal world. It measures the quality of a connection, and takes in account the latency (time/distance between source and destination). These are issues that can impact the quality of your Skype call, your IP phone, streaming video, and other bandwidth intensive exchanges between your PC and another server.
When things get really fast, the browser matters to the CIRA Internet Performance Test
This is an important distinction because many (but not all!) multimedia applications use single TCP session for streaming, one TCP connect to stream a movie, to make a phone/video call, to download/upload a picture or movie. In order to measure the quality of the connection, the browser needs to do a lot more work to monitor TCP sessions, calculate duplicate packets, identify out of sequence packets, calculate packet loss and jitter, and to retransmit lost data. Choosing the right browser to do the NDT test is important to get accurate results since not all browsers are equal.
The NDT application is available in Java and Flash versions, and Chrome is the preferred browser to run the NDT Flash application. CIRA chose to go the Flash route due to known security issues with Java for this type of application. There are advantages and disadvantages to this approach but suffice to say that the community is currently working on developing and launching a websocket version available as a Flash alternative. It is in beta now, but we are keen to continue advancing the Internet Performance Test on the latest versions as time permits. For now, measuring quality is hard work and not all browsers will support this workload executed in Flash. We expect broader browser support with the future release of the websocket NDT client tool.
It’s about Canada’s Internet
Don’t forget what it is all about. The CIRA Internet Performance Test aims to help individual Canadians understand their own Internet connection, but above all, its goal is to help us all create a better & faster Internet in Canada. So what do we want you to do? Bookmark both tests, and run them both as often as you can. Take the time to learn a bit about how the Internet works – we think it is actually pretty cool.
Last updated on March 22, 2017.
Rob a acquis plus de 20 ans d’expérience de la rédaction, de la présentation et du blogage à l’intention de l’industrie des technologies. Il aborde des thèmes aussi variés que les outils de développement de logiciels, l’ingénierie inverse de Silicon, la cybersécurité et le DNS. De fait, Rob est un spécialiste du marketing passionné qui s’adresse aux professionnelles et aux professionnels des TI en leur donnant les renseignements et les précisions dont ils ont besoin pour s’acquitter de leurs tâches.
