Browsing around the papers presented at the latest NOSSDAV workshop, I found “An Empirical Evaluation of VoIP Playout Buffer Dimensioning in Skype, Google Talk, and MSN Messenger”. Having worked extensively with GIPS’ jitter buffer algorithms, and having some knowledge of Google Talk, I was intrigued by the title. The paper had some interesting experiments, but also a few giant leaps to conclusions.

The paper’s authors have created a laboratory test bench for PC soft phones where they emulate different network conditions (delay, jitter and packet losses), and measure objective speech quality (PESQ) and the end-to-end delay. Then they apply a previously proposed hybrid between PESQ and the E-Model to arrive at a score which takes both measured speech quality and delay into account. The idea is that both audio quality and end-to-end delay contribute to the total conversation experience, which is an easily supportable proposition. Finally, they derive an optimal playout buffer delay for each network condition based on this hybrid measure. I will come back to this approach later.

The experimental part of the paper, setting up the lab and examining the three clients, seems all fine to me, even though I’m not sure that their delay estimation algorithm really can cope with the rapid delay changes that modern jitter buffers apply. They also make rather wild assumptions on coding, packetization, and soundcard delays. But those are minor issues. My problem is their use of the objective hybrid model as a guide to optimality. It is widely know that PESQ is rubbish when it comes to assessing agile jitter buffers, simply because it cannot follow the swift delay adaptation. Tagging on a delay impairment factor to obtain a total user experience number frankly doesn’t improve the situation.

The authors wrap up their work by comparing the measured delays of the three clients, with the delay that renders the highest score in their hybrid measure under the same network conditions. The three clients all exhibit different behavior – not very surprising since they have different jitter buffers – but none of them follow what the authors claim to be optimal. Hence, the user experience of all three VoIP clients could be vastly improved, if only the “optimal” delay would be applied, is their conclusion. Allow me to disagree.

Surely these VoIP clients can be improved, but to distrust the man-years of design and implementation, and endless hours of in-house and customer tuning and testing, I need something more than the broken compass that is PESQ.

A while ago I attended a talk by Prof. Henning Schulzrinne, one of the developers behind various VoIP related protocols such as RTP, RTSP, and SIP. While I was expecting to hear a lot about these protocols, I got something completely different. In his talk, he sent the message that the Internet is now a vital infrastructure for millions of people around the world, and that people working within network infrastructure and services – both in industry and in academia – should treat it accordingly.

Comparing the Internet and its services to civil engineering infrastructures might not be that revolutionary at first glance. We all know that the Internet is vital to millions of people in their everyday lives. It’s the way the traditional infrastructures are maintained and evolved, in very long cycles, that is interesting to project onto Internet building. What you once decided to build, you will have to live with for a very long time to come. For instance, it would not make much sense to try to change the rail gauge today – we are stuck with what we have.

I’d like to highlight four conclusions from the talk that I think are worth serious consideration for anyone working on networking applications:

1)      There will be no clean-slate next-generation Internet

For the same reasons we won’t change the electric power sockets in all homes, or the railroad gauge, the current protocols and interfaces for the general networks will be around for quite some time. Standards and protocols won’t be exchanged, they will only be extended and refurbished, and new technology must coexist and work with the old stuff for years to come. 

2)      We need good engineering solutions

Solutions must solve the users’ needs, not only those of the vendors’ needs, and certainly not only developers’ desire to see how much cool stuff they can come up with. Most users must be able to manage their own home equipment. In the presenter’s own words: “You do not need an electrician to install your new toaster.” Setting up a SIP-based VoIP phone at home should be no harder than plugging in my landline phone or my toaster. But still, how should I know what my “secondary STUN server” is? Oh, it’s usually the same as my primary? Gee, thanks…

 3)      We need academic research driven by real problems

There is a very weak connection between network related research in academia and the network industry. For instance, over 13,000 research papers have been authored and published on QoS, but virtually none of them have made it into IETF documents or real products employing some kind of QoS. (I’m not saying that QoS is not happening, just that it did not come from the 13,000 academic research publications.) There is dialouge between research publications and standardization bodies (e.g., the IETF). The standardization people rarely read research literature, and the academic researchers think it’s too much work to push an idea into a standard. 

Professor Schulzrinne also seized the opportunity to dismiss the research efforts of several in the audience by saying that there have been more papers written on “sensor networks” than there are sensor networks out there. Nobody wants it! What we need is research providing usability, reliability and cost savings. 

4)      Network and service availability must become “five nines”

In my mind, this is the most important consequence of treating the Internet and network services as a part of civil infrastructure. 99.999% of the time, it should just work. That means no more than 5.5 minutes down-time per year. Has anyone seen a home internet connection this reliable? I’m still waiting.