RELATED APPLICATIONS  
       [0001]    This application is a continuation-in-part of U.S. Application Ser. No. 09/870336 filed on May 30, 2001, the contents of which are herein incorporated by reference. 
     
    
     
       FIELD OF DISCLOSURE  
         [0002]    This disclosure relates to measurement of quality of service.  
         BACKGROUND  
         [0003]    In a content delivery system, a content-server transmits a media stream to a client over a communication channel. The content-server often transmits media streams at times during which the client system is unlikely to be in use. These media streams are saved in a mass-storage medium for later retrieval and viewing by an audience.  
           [0004]    During transmission, transmission errors are introduced. These errors affect the viewability of the media stream. The extent of these errors is reflected in the “Quality of Service”, or QOS, for that transmission. In many cases, a content-delivery system measures QOS during transmission of the media stream. If the measured QOS indicates excessive transmission errors, then the content-server re-transmits the media stream.  
           [0005]    A conventional content delivery service measures its QOS by collecting network statistics and inferring, on the basis of those network statistics, how good the media stream would appear to the viewing audience. 
       
    
    
     BRIEF DESCRIPTION OF THE FIGURES  
       [0006]    [0006]FIG. 1 shows a content delivery system.  
         [0007]    [0007]FIG. 2 shows the architecture of the content delivery system of FIG. 1. 
     
    
     DETAILED DESCRIPTION  
       [0008]    As shown in the example of FIG. 1, a content delivery system  10  for the delivery of a media stream  12  from a content server  14  to a client  16  first reduces bandwidth needed for transmission by passing the media stream  12  through an encoder  18  executing on the content server  14 . The encoder  18  transforms the media stream  12  into a compressed form, herein referred to as the “encoded media stream  20 ,” suitable for transmission. The encoded media stream  20  then traverses a communication channel  22  until it reaches the client  16 , whereupon it becomes a received encoded media stream  21 . As shown in FIG. 1, the communication channel  22  is a wireless link between the client  16  and the content server  14 . However, the communication channel  22  can also include a portion of a cable distribution network or a computer network. Examples of computer networks include WANs, LANs, private networks, and public networks such as the internet. This received encoded media stream  21  is stored on a storage device  27  for later viewing. In response to a request to view the media stream  26 , or in response to a request for a QOS measurement, a decoder  24  executing on the client  16  retrieves the received encoded media stream  21  from the storage device  27  and transforms it into a decoded media stream  26 .  
         [0009]    In the content delivery system  10  of FIG. 1, there are at least two mechanisms that can impair the quality of the media stream. First, the encoder  18  and decoder  24  can introduce errors. For example, many encoding processes discard high-frequency components of an image in an effort to compress the media stream  12 . As a result, the decoded media stream  26  may not be a replica of the original media stream  12 . Second, errors can be introduced within the communication channel  22  itself. The nature of these errors depends on the type of communication channel. For example, in the case of satellite transmission, ionospheric conditions can degrade the received signal quality. Other mechanisms for introducing errors into the communication channel  22  include multipath reflection and dispersion.  
         [0010]    These two impairment mechanisms, hereafter referred to as encoding error and transmission error, combine to affect the audience&#39;s subjective experience in viewing the media. However, the audience&#39;s subjective experience also depends on one other factor thus far not considered: the content of the media stream  12  itself.  
         [0011]    In many cases, the extent to which a particular error affects an audience&#39;s enjoyment of a decoded media stream  26  depends on the content of the original media-stream  12 . For example, a media stream  12  rich in detail will suffer considerably from loss of sharpness that results from discarding too many high frequency components. In contrast, the same loss of sharpness in a media stream  12  poor in detail, such as one having extensive night-time scenes, will most likely go unnoticed.  
         [0012]    Referring to FIG. 2, a system  28  for measurement of QOS includes a content server  30  in data communication with a client  32 . Communication between the client  32  and the content server  30  can be over a satellite link, as shown in the figure. Alternatively, communication can be established over a cable network, a global computer network such as the internet, or any other data communication network.  
         [0013]    An encoder  38  applies an encoding or compression algorithm to the original media stream  39 , thereby generating an encoded media stream  40 . This encoded media stream  40  is them provided to the content server  30  for transmission to the client  32 . In most cases, particularly in the delivery of video programming, encoding is carried out in advance and the encoded media stream  40  is transmitted to the client  32  at an off-peak time, such as in the middle of the night, when the client  32  is unlikely to be in use by an audience. In such cases, the encoded media stream  40  is stored on a mass-storage system (not shown) associated with the content server  30  to await transmission.  
         [0014]    A variety of encoding processes are available. In many cases, these encoding processes are lossy. For example, certain encoding processes will discard high-frequency components of an image under the assumption that, when the image is later decoded, the absence of those high-frequency components will not be apparent to the viewing audience. Whether this is indeed the case will depend in to part on the features of the image.  
         [0015]    In addition to being transmitted to the client  32 , the encoded media stream  40  at the output of the encoder  38  is also provided to the input of a first decoder  42 . The first decoder  42  generates first decoder output  43  by recovering the original media stream to the extent that the possibly lossy encoding performed by the encoder  38  makes it possible to do so.  
         [0016]    The first decoder output  43  is then provided to a first feature extractor  44 . The first feature extractor  44  generates first feature data  49  by implementing known feature extraction algorithms for extracting temporal or spatial features of the encoded media stream  40 . Known feature extraction methods include the Sarnoff JND (“Just Noticeable Difference”) method and the methods disclosed in ANSI T1.801.03-1996 (“American National Standard for Telecommunications—Digital Transport of One Way Video Signals—Parameters for Objective Performance Specification”) specification.  
         [0017]    The original media stream  39  is also passed through a second feature extractor  46 , that performs feature extraction like the first feature extractor  44 , that generates second feature data  47 . The first and second feature data  49 ,  47  are then compared by a first analyzer  48 . This comparison results in the calculation of an encoding metric. As noted above, a media stream can be degraded through transmission errors and through encoding errors. The encoding metric provides a measure of how badly the media stream is degraded as a result of encoding errors alone.  
         [0018]    An analyzer  48  compares features of two media streams. The output of the analyzer  48  is typically a dimensionless quantity that represents a normalized measure of how different the two media streams would appear to a viewer. In some practices of the invention, the analyzer  48  is configured to re-encode some or all of the original media stream  39  if the encoder metric indicates that the first and second feature data  49 ,  47  are too different from each other.  
         [0019]    The first feature data  49  is also provided to the content server  30 . The content server  30  transmits both the encoded media stream  40  and the first feature data  49  to the client  32  by way of the first antenna  34 . The encoded media stream  40  and the first feature data  49  can be transmitted concurrently or at separate times.  
         [0020]    As it propagates between the first antenna  34  and a second antenna  35  associated with the client  32 , the encoded media stream  40  is subjected to the various difficulties that are commonly encountered on a communication channel. These difficulties are manifested as jitter, packet loss, and packet latency in the encoded media stream  40  received by the client  32 . In one embodiment, statistics on these and other measures of transmission error are collected by a network performance monitor  52  located at the client  32  and made available to a second analyzer  60  located at the client  32 .  
         [0021]    The media stream received by the client  32 , referred to herein as the “received encoded media stream  53 , is then stored on a mass-storage device  57 . In response to a request to view the media stream, a copy of the received encoded media stream  53  is provided to a second decoder  54 . The output of the second decoder  54 , referred to herein as the “second decoder output  56 ,”is provided to a display  55  for viewing by an audience.  
         [0022]    In some cases, transmission error significantly impairs the quality of the received encoded media stream  53 . In cases in which the original media stream  39  is transmitted to the client  32  in advance of when it is viewed, there is an opportunity to correct this by re-transmitting some or all of the original media stream  39 . For this opportunity to be taken advantage of, the client  32  must determine whether the received encoded media stream  53  has been significantly impaired.  
         [0023]    In contrast to conventional systems, the client  32  does not simply examine network statistics during transmission to assess the impairment of the media stream. As noted above, whether the media stream is so impaired as to degrade the viewer&#39;s experience depends, to a great extent, on the content of the media stream. The client  32  instead compares first feature data  49  from the first feature extractor  44  with corresponding third feature data  59  extracted from the second decoder output  56  by a third feature extractor  58  that performs feature extraction like the first and second feature extractors  44 ,  46 . The first and third feature data  49 ,  59  are then provided to the second analyzer  60  for comparison with each other.  
         [0024]    Unlike the second decoder output  56  provided to the third feature extractor  58 , the input to the first feature extractor  44  was never subjected to the vagaries of transmission. Hence, any difference between the first and third feature data  49 ,  59  is attributable to transmission errors alone. This difference is determined by the second analyzer  60  on the basis of a comparison between the first and third feature data  49 ,  59 . On the basis of this difference, and optionally on the basis of network statistics  62  provided by the network monitor  52 , the second analyzer  60  calculates a transmission metric  64  indicative of the extent to which the subjective perception of a viewing audience would be degraded by the transmission error alone. This transmission metric  64  can be sent back to the content server  30 , either using the same channel that was used to transmit the encoded media stream  40 , or through an alternate data communication channel, for example through a telephone line.  
         [0025]    Upon receiving the transmission metric  64 , the content server determines whether the QOS measured during transmission of the encoded media stream  40  is below a threshold. If the transmission metric  64  indicates that QOS during transmission was poor, the content server  30  re-transmits the encoded media stream  40 . Otherwise, the content server  30  need do nothing further.  
         [0026]    In an alternative embodiment, a decision to request retransmission on the media stream is made at the client  32  on the basis of the output of the second analyzer  60 . In this case, the client  32  sends a signal back to the content server  30  to request re-transmission of the media stream. This further shifts the computational burden to the client  32  from the content server  30 .  
         [0027]    The client  32  thus provides an estimate of how a viewing audience is likely to perceive a second decoder output  56  derived from the received encoded media stream  53 . If the received encoded media stream  53  proves to be excessively impaired, the client  32  requests re-transmission of the encoded media stream  40 . In effect, the client  32  previews second decoder output  56  to determine whether it is of adequate quality to present to a viewing audience.  
         [0028]    Instructions for carrying out the method described herein are typically stored on a machine-readable medium for execution by a processing element such as that found in digital computers, PDA&#39;s, and other devices that employ processing elements such as microprocessors and micro-controllers. Such devices can include electronic devices, optical devices, or combinations thereof.
A method of measuring quality of service includes receiving, from a content server, a transmission of a first media stream and comparing that first media stream with a second media stream that corresponds to the first media stream prior to transmission thereof. This comparison provides a basis for determining a quality of service of the transmission.