Browsing around the papers presented at the latest NOSSDAV workshop, I found “An Empirical Evaluation of VoIP Playout Buffer Dimensioning in Skype, Google Talk, and MSN Messenger”. Having worked extensively with GIPS’ jitter buffer algorithms, and having some knowledge of Google Talk, I was intrigued by the title. The paper had some interesting experiments, but also a few giant leaps to conclusions.

The paper’s authors have created a laboratory test bench for PC soft phones where they emulate different network conditions (delay, jitter and packet losses), and measure objective speech quality (PESQ) and the end-to-end delay. Then they apply a previously proposed hybrid between PESQ and the E-Model to arrive at a score which takes both measured speech quality and delay into account. The idea is that both audio quality and end-to-end delay contribute to the total conversation experience, which is an easily supportable proposition. Finally, they derive an optimal playout buffer delay for each network condition based on this hybrid measure. I will come back to this approach later.

The experimental part of the paper, setting up the lab and examining the three clients, seems all fine to me, even though I’m not sure that their delay estimation algorithm really can cope with the rapid delay changes that modern jitter buffers apply. They also make rather wild assumptions on coding, packetization, and soundcard delays. But those are minor issues. My problem is their use of the objective hybrid model as a guide to optimality. It is widely know that PESQ is rubbish when it comes to assessing agile jitter buffers, simply because it cannot follow the swift delay adaptation. Tagging on a delay impairment factor to obtain a total user experience number frankly doesn’t improve the situation.

The authors wrap up their work by comparing the measured delays of the three clients, with the delay that renders the highest score in their hybrid measure under the same network conditions. The three clients all exhibit different behavior – not very surprising since they have different jitter buffers – but none of them follow what the authors claim to be optimal. Hence, the user experience of all three VoIP clients could be vastly improved, if only the “optimal” delay would be applied, is their conclusion. Allow me to disagree.

Surely these VoIP clients can be improved, but to distrust the man-years of design and implementation, and endless hours of in-house and customer tuning and testing, I need something more than the broken compass that is PESQ.

As long as I have been involved with VoIP the debate whether QoS methods are the solution to providing good voice quality has been ongoing. With QoS methods I refer to protocols that allows for prioritization of packets that have low latency requirements such as VoIP packets. Of course, if from the VoIP applications point of view, the network is perfect you should also expect perfect quality. As a a side point, that is a very reasonable expectation but unfortunately something that is very often not the case. The reasons can be endpoint hardware or software related or a combination of both. I discuss some of the potential issues in a previous blog post.

The reason why QoS methods are not heralded as the savior of VoIP quality (and video for that matter) is that they are often impractical to implement and not as efficient as one might assume. For example, if the amount of data on the network that needs to be prioritized represents a significant portion of the total traffic the scheme will fail completely. Another issue is the impact on the so called background traffic that doesn’t get prioritized and may result in unacceptable behavior of the less prioritized data streams.

QoS methods are successfully used in well managed and controlled networks but because the VoIP traffic often traverses many networks, including the largely unmanaged Internet, rarely can end-to-end prioritization be guaranteed.

because of these limitations of QoS methods it is crucial that any voice or video offering over packet networks deploy endpoints that can compensate for network issues.

So, you may ask, what can I do on my own network? In this article in  ComputerWorld you can learn how to tweak your WiFi router settings to implement QoS on your home network. As I mentioned previously, this will unfortunately only help the performance of the WiFi network and it requires changing the router configuration in a manner most consumers are not aware of or not able to do because of the complexity involved. So, even though I think it is a good idea to make such adjustments they only solve problems on a small portion of the data path for a call (the actual broadband connection is much more often the real culprit) and are unlikely to be done made by most end users. Therefore, as a developer of a VoIP or video over IP product you can never assume that QoS will save you, you have to make sure that your product has been properly designed to mitigate network issues.

The video hype is ever increasing where Cisco buying Tandberg and Logitech buying Lifesize are just two examples. I think the video hype is good news and has written about it a lot.

But, as one of our biggest and most famous customers almost reminds me when I meet them, don’t forget about voice quality. Voice quality is much more important than video quality they say. I totally agree with this!

During a video conference business meeting, the important thing is to have very high quality (HD) voice with robustness. The speech shouldn’t be garbled so that the attendants can’t follow the conversation. Also, a consistent high quality is needed to combat attendant fatigue and increase the effectiveness of the meeting.

Video quality is also very important for the multimedia experience of the attendants. However, if the video sometimes gets a little jerky or freezes, the meeting will still continue as long as you hear what is being said clearly.

I can’t resist pointing out that GIPS is uniquely well positioned to enable our customers to provide the desired end user experience since we started out on the voice side and continuously strive to provide maximum voice quality!