Performance Measures for Multimedia Applications

Randy Bloomfield

Table of Contents

• Slides from this Talk
• Stan Barber's Notes
• June 1997 NANOG Home Page

Stan Barber's Notes

ITS has been involved in performance measurements of telecommunication systems and is now moving to do performance measurements of Internet services.

Internet performance measurements

• Multimedia applications are already happening
• Requires user-oriented performance measures

User-oriented performance measures have three attributes

• technology independent
• perception based
• objectively measured

ITS has implemented measurements that have these attributes, but he did not bring their system today. He offered to demonstrate the ITS multimedia objective quality measurement system at a future NANOG meeting. ITS is the chief research and engineering arm of NTIA. ITS supports the NTIA and its telecommunications objectives.

NTIA is part of the Department of Commerce and is a peer organization with NIST. ITS has a number of products and has recently becoming involved with IETF under its mandate to work with standards bodies in implementing standards.

At this time, ITS has started to be involved Internet performance issues.

ITS proposes to help work on both user-oriented and network-oriented performance measures.

Various impairments exist due to the use of digital codecs. Video suffers with error-blocks, tilling, jerkiness and smearing. Audio can suffer from compression artifacts.

Source dependencies -- Quality may be dependent on the information complexity of the source. In many cases, a specific test signal may not really provide a realistic test of things that users consider important.

To address this, ITS has developed a perception-based measurement system. This system uses feature extraction to provide a more realistic test signal (relative to what the users really care about).

To validate this system, opinions are solicited from users on the performance of multimedia applications. Statistics are used to correlate objective measures of multimedia application performance with the subjective responses (e.g., mean opinion score). The objective measures can then be used to accurately predict the MOS of end-to-end multimedia systems (including codecs and transport).

Also, the ISPs need to be familiar not only with the IETF's performance work but also with other relevant performance standardization activities (e.g., ITU-T, Committee T1).

Questions & Answers

Bill asks: What do you want the ISPs to do for you? I think the IETF is already doing good work. There needs to be more end-to-end effort, though. Also, the ISPs needs to be familiar with this work.

Another person stands to say that he believes that audio and video can only be compressed so far. He askes how far are we from the ceiling? I don't know much about codec design. However, one way to look at the problem is to take the user requirements for quality and see how those needs can be addressed.