text:
Conversational protocols for implementing distributed conversational networking architectures and/or distributed conversational applications, as well as real-time conversational computing between network-connected pervasive computing devices and/or servers over a computer network. In one aspect of the invention, a communication stack for use in a real-time distributed conversational network comprises a first layer for generating encoded audio data, a second layer for wrapping encoded audio data in a real-time encoded audio data stream, and a third layer for wrapping control messages in a real-time control data stream for real-time control of conversational distributed functions over the network. Preferably, the communication stack comprises extensions of RTP (Real Time Protocol) and RTCP (Real Time Control Protocol) to respectively transmit, in real-time, the necessary encoded audio data and control data for executing and controlling distributed conversational functions, wherein meta information associated with the real-time encoded audio stream is added as an extension of header of an RTP packet, and wherein meta information associated with the real-time control data stream is added as an extension of a header of an packet.
1. A communication system, comprising:
a communication stack comprising:
a first layer for generating encoded audio data, wherein the audio data comprises compressed feature vectors representative of speech, wherein the encoded audio data comprises a file format that enables transmission of segments of speech and decompression of the segments of speech in a random order;
a second layer for generating a data stream comprising the encoded audio data;
a third layer for generating a transmission control data stream, wherein the transmission control data stream comprises meta information for coding scheme notifications; and
a fourth layer for transporting the data stream and the transmission control data stream.
2. The system of
3. The system of
4. The system of
5. The system of
6. The system of
7. The system of
8. The system of
9. The system of
10. The system of
11. The system of
12. The system of
13. The system of
14. The system of
15. The system of
16. The system of
17. The system of
18. The system of
19. The system of
20. The system of
21. The system of
22. A communication system for use in a real-time distributed conversational network, the communication system comprising:
a communication stack comprising:
a first layer for generating encoded audio data;
a second layer for wrapping encoded audio data in a real-time encoded audio data stream;
a third layer for wrapping control messages in a real-time control data stream
comprising control data that enables real-time control and coordination of
conversational distributed functions over the network; and
a fourth layer for generating a real-time transmission control data stream, wherein the real-time transmission control data stream comprises meta information for providing coding scheme notifications.
23. The system of
24. The system of
25. The system of
26. The system of
27. The system of
28. The system of
29. The system of
30. A method for providing real-time distributed conversational computing, comprising the steps of:
generating encoded audio data;
wrapping encoded audio data in a real-time encoded audio data stream; and
wrapping control messages in a real-time control data stream comprising control data that enables real-time control and coordination of conversational distributed functions over the network,
wherein the audio data comprises compressed feature vectors representative of speech and wherein the encoded audio data comprises a file format that enables transmission of segments of speech and decompression of the segments of speech in a random order.
1. Technical Field
The present application relates generally to systems and method for providing conversational networking and, more particularly, to conversational protocols for implementing distributed conversational applications over a computer network.
2. Description of Related Art
The computing world is evolving towards an era where billions of interconnected pervasive clients communicate with powerful information servers. Indeed, this millennium will be characterized by the availability of multiple information devices that make ubiquitous information access an accepted fact of life. The evolution of the computer world towards billions of pervasive devices interconnected via the Internet, wireless networks or spontaneous networks (such as Bluetooth and Jini) will revolutionize the principles underlying man-machine interaction. In the near future, personal information devices will offer ubiquitous access, bringing with them the ability to create, manipulate and exchange any information anywhere and anytime using interaction modalities (e.g., speech and/or GUI) most suited to the user's current needs and abilities. Such devices will include familiar access devices such as conventional telephones, cell phones, smart phones, pocket organizers, PDAs and PCs, which vary widely in the interface peripherals they use to communicate with the user.
The information being manipulated via such devices may reside on the local device or be accessed from a remote server via a communications network using open, interoperable protocols and standards. The implementation of such open standards also leads to a seamless integration across multiple networks and multiple information sources such as an individual's personal information, corporate information available on private networks, and public information accessible via the global Internet. The availability of a unified information source will define productivity applications and tools of the future. Indeed, users will increasingly interact with electronic information, as opposed to interacting with platform-specific software applications as is currently done in the world of the desktop PC.
With the pervasiveness of computing causing information appliances to merge into the users environment, the user's mental model of these devices is likely to undergo a dramatic shift. Today, users regard computing as an activity that is performed at a single device like the PC. As information appliances abound, user interaction with these multiple devices will be grounded on a different set of abstractions. The most intuitive and effective user model for such interaction will be based on what users are already familiar with in today's world of human-intermediated information interchange, where information transactions are modeled as a conversation amongst the various participants in the conversation.
Indeed, it is expected that information-centric computing carried out over a plethora of multi-modal information devices will be essentially conversational in nature and will foster an explosion of conversational devices and applications. It is to be noted that the term "conversational" is used to mean more than speech interaction—it encompasses all forms of information interchange, where such interchange is typically embodied by one participant posing a request that is fulfilled by one or more participants in the conversational interaction. The core principle behind the conversational interaction is that any interaction between the user and the machine be handled as a dialog similar to human—human dialog. Accordingly, the increasing availability of information available over a communications network, along with the rise in the computational power available to each user to manipulate this information, brings with it a concomitant need to increase the bandwidth of man-machine communication so that the increased human-machine interaction that will result from the pervasive use of such information devices will be as natural and simple as if the user was having a conversation with another individual.
With the increased deployment of conversational systems, however, new technical challenges and limitations must be addressed. For instance, currently available pervasive clients typically do not have the required memory and/or processing power to support complex conversational tasks such as recognition and presentation. Indeed, even with the rapid evolution of the embedded processor capabilities (low power or regular processors), one can not expect that all the processing power or memory is available for executing complex conversational tasks such as, for example, speech recognition (especially when the vocabulary size is large or specialized or when domain-specific/application-specific language models or grammars are needed), NLU (natural language understanding), NLG (natural language generation), TTS(text-to-speech synthesis), audio capture and compression/decompression, playback, dialog generation, dialog management, speaker recognition, topic recognition, and audio/multimedia indexing and searching, etc.
Moreover, even if a networked device is "powerful" enough (in terms of CPU and memory) to execute all these conversational tasks, the device may not have access to the appropriate domain-specific and application-specific data files or appropriate algorithms (e.g., engines) to adequately execute such tasks. Indeed, vendors and service providers typically do not allow for open exchange of the algorithms (conversational engines) for executing conversational tasks and/or the data files (conversational arguments) utilized by such algorithms (e.g., grammars, language models, vocabulary files, parsing, tags, voiceprints, TTS rules, etc.) to execute such tasks, which they consider intellectual, business logic and technology crown jewels. Indeed, some conversational functions may be too specific to a given service, thereby requiring back end information that is only available from other devices or machines on the network.
Furthermore, the network infrastructure may not provide adequate bandwidth for rapidly exchanging data files needed by conversational engines for executing conversational tasks. For example, NLU and NLG services on a client device typically require server-side assistance since the complete set of conversational arguments or functions needed to generate the dialog (e.g., parser, tagger, translator, etc.) may be too extensive (in terms of communication bandwidth) for transmission from the server to the client over the network connection. In addition, even if such data files can be transmitted over the network, such transmission may introduce long delays before the client device is able to commence an application or process an input, thereby preventing real-time interactions.
These problems may be solved through implementation of distributed architectures, assuming that such architectures are implemented in appropriately managed networks to guarantee quality of service for each active dialog. Indeed, the problems associated with a distributed architecture and distributed processing between client and servers require new methods for conversational networking. Such methods comprise management of traffic and resources distributed across the network to guarantee appropriate dialog flow of for each user engaged in a conversational interaction across the network.
Accordingly, systems and methods that provide conversational networking through implementation of, e.g., distributed speech recognition (DSR), distributed conversational architectures and conversational protocols for transport, coding and control, are highly desirable. Indeed, it would be advantageous to allow network devices with limited resources to perform complex conversational tasks (preferably in real-time) using networked resources in a manner which is automatic and transparent to the users of such devices.
The present invention is directed to conversational protocols for implementing distributed conversational networking architectures and/or distributed conversational applications, as well as real-time conversational computing between network-connected pervasive computing devices and/or servers over a computer network. The implementation of distributed conversational systems/applications according to the present invention is based, in part, on a suitably defined conversational coding, transport and control protocols.
In one aspect of the invention, a communication stack for use in a real-time distributed conversational network comprises a first layer for generating encoded audio data, a second layer for wrapping encoded audio data in a real-time encoded audio data stream, and a third layer for wrapping control messages in a real-time control data stream for real-time control of conversational distributed functions over the network.
Preferably, the communication stack comprises extensions of RTP (Real Time Protocol) and RTCP (Real Time Control Protocol) to respectively transmit, in real-time, the necessary encoded audio data and control data for executing and controlling distributed conversational functions. Meta information associated with the real-time encoded audio stream is added as an extension of a header of an RTP packet, wherein the RTP header extension comprises one field indicating an encoding type of the encoded audio data in the RTP packet and a second field comprising header information associated with a file of the encoded audio data. Further, meta information associated with the real-time control data stream is added as an extension of a header of an RTCP (Real Time Control Protocol) packet, wherein the RTCP header extension comprises a first field indicating a type of conversational distributed protocol and a second field comprising a corresponding control message. The control data may comprise any type of application-specific control data that may be utilized for implementing a specific distributed framework.
In another aspect, the communication stack comprises a fourth layer for generating a real-time transmission control data stream, wherein the real-time transmission control data stream comprises meta information for providing coding scheme notifications. Meta information associated with the real-time transmission control data stream is preferably added as an extension of a header of an RTCP (Real Time Control Protocol) packet.
These and other aspects, features and advantages of the present invention will be described and become apparent from the following detailed description of preferred embodiments, which is to be read in connection with the accompanying drawings.
The present invention may be implemented in various forms of hardware, software, firmware, special purpose processors, or a combination thereof. Preferably, the invention is implemented in software as an application comprising program instructions that are tangibly embodied on one or more program storage devices (e.g., magnetic floppy disk, RAM, CD ROM, ROM and Flash memory) and executable by any device, machine or platform comprising suitable architecture. Since the invention is preferably implemented in software, the system architectures and method steps may differ depending upon the manner in which the invention is programmed. Given the teachings herein, one of ordinary skill in the related art will be able to contemplate these and similar implementations or configurations.
I. Overview
The present invention is directed to conversational protocols for implementing distributed conversational networking architectures and/or distributed conversational applications, as well as real-time conversational computing between network-connected pervasive computing devices and/or servers over a computer network. More specifically, the implementation of distributed conversational systems/applications according to the present invention is based in part on a suitably defined conversational coding, transport and control protocols.
There are three basic configurations for implementing conversational protocols in accordance with the present invention and the format of such conversational protocols will vary based on the underlying transport layers and desired application. In a preferred embodiment, conversational protocols for providing distributed conversational networking are implemented on top of RTP (Real Time Protocol). For example, as described in detail below, a conversational coding protocol according to one aspect of the present invention is implemented by extending RTP to produce what is referred to herein as RTCCP (real time conversational coding protocol).
As is known in the art, the real time protocol is a method for providing real time transmission of data over a network. RTP does not have all the functions of a transport protocol and is typically used on top of a network level protocol such as TCP (transmission control protocol) or UDP (User Datagram Protocol). TCP is known as a transport level protocol which controls the transmission and the flow of data between two hosts on a network. The TCP protocol may not be ideal in real-time implementations due to its data flow control and reliability mechanisms which can halt/interrupt the flow of data transmission over a network. More specifically, TCP provides reliability through a mechanism that ensures that every single datagram (or packet) is delivered. This mechanism involves assigning each datagram a sequence number and sequentially transmitting each datagram to a receiver. For a given transmitted datagram, if the receiver successfully receives the datagram (i.e., receives an undamaged datagram), the receiver will transmit an acknowledgment message (ACK) to inform the source that it has successfully received the datagram and to send the next sequential datagram. If, after transmitting a given datagram, the source does not receive an ACK message for the datagram, the source will continue to transmit the datagram until an ACK message is returned. Consequently, the flow of datagrams may be temporarily interrupted during transmission as the receiver waits for an undamaged datagram.
UDP is alternative protocol to TCP for use with RTP. The UDP does not utilize the reliability and flow control mechanisms of TCP but rather transmits the datagrams to the receiver in a continuous stream. Consequently, UDP is a preferred protocol for use with RTP for real-time implementations since it delivers a constant stream of datagrams without any significant delay (other than connection bandwidth and network congestion). Despite its lack of a reliably delivery mechanism, the implementation of conversational protocols on top of RTP using UDP in accordance with the present invention preferably employs a reliable delivery mechanism (usually at a relatively low bit rate) similar to TCP/IP, but not as restrictive as TCP. Indeed, as described in detail below, in the absence of TCP/IP, reliable packet delivery is emulated by providing a packet delivery confirmation and re-sending mechanism. Advantageously, the implementation of conversational protocols on top of RTP and UDP (with a reliability mechanism) affords real-time interaction when needed (e.g., when immediate user recognition is expected by the dialog or when the input must directly impact the state of the dialog). The present invention will provide a detailed embodiment of implementing conversational protocols using real time protocols.
It is to be understood that the conversational protocols described herein may be implemented as extensions to other transport mechanisms. For instance, the conversational protocols may be implemented on top of TCP/IP. This presents the advantage to be the most common type of transport protocol that is employed—It is the Internet transport protocol. While TCP/IP is the simplest default mechanism to transport data and control from one device to another using, e.g., FTP (file transfer protocol), RMI (remote method invocation), RPC (remote procedure call), etc., as explained above, it does not guarantee real-time interaction. Indeed, missing or damages packets are systematically waited for or re-sent. This may not be an issue for, e.g., deferred recognition tasks. For example, a form filling process in VoiceXML (http://www.voicexml.com), may not expect immediate speech recognition for each field that is filled, but only recognition upon submission of the entire form to the server. In any event, it is to be understood that a preferred embodiment of implementing RTP-based conversational protocols may utilize TCP/IP if the given application calls for guaranteed, but non-real time, transmission of the associated data packets.
In another aspect of the present invention, the conversational protocols may be implemented on top of HTTP (hypertext transfer protocol) (or WAP (wireless application protocol). HTTP is the main protocol used on the Internet for hypertext transfer (Web programming model), i.e., transferring data on the World Wide Web. The implementation of conversational protocols on top of HTTP allows direct integration of the engine distribution within browser solutions (e.g. http://www.voiceXML.com) with no major change required in the infrastructure. WAP is an equivalent lightweight transport protocol to use on wireless networks (devices with limited wireless bandwidth connections and limited GUI capabilities). Since HTTP is implemented on TCP/IP and has a significant amount of overhead associated therewith (e.g., most of the remote commands result in multiple exchanges various headers) and because WAP provides a reliable delivery mechanism, the implementation of RTP-based conversational protocols on top of HTTP and WAP is preferable for non-real time applications.
It is to be appreciated that the RTP mechanism is preferred when real-time interactions are required. Regardless of the implementation choice, however, the following design principles (definitions) are preferably considered herein for implementing a distributed network using conversational protocols according to the present invention. Conversational protocols according to one aspect of the present invention are preferably defined based on the following criteria:
As indicated above, conversational protocols are preferably implemented on top of RTP so as to minimize the dialog delays introduced by distributed processing. A preferred embodiment for implementing conversational protocols on top of RTP based on the above criteria will be explained in detail below. First, an overview of a preferred embodiment utilizing RTP-based conversational protocols in accordance with such criteria will now be given.
With respect to audio encoding and decoding mechanism and a file format for encoded audio data, the present invention preferably employs a well-defined conversational coding protocol comprising (1) a CODEC for encoding/decoding speech/audio data, which minimizes the distortion of the acoustic front-end features and allows reconstruction of intelligible waveforms and (2) a file format associated with the encoded speech/audio data (which is transmitted between network-connected devices/servers using a conversational transport mechanism discussed below). In a preferred embodiment, a conversational CODEC preferably compresses a cepstral feature stream while minimizing the distortion of the reconstructed features. In particular, any CODEC employing a compression scheme that minimizes the error rates of associated conversational engines and which allows for reconstruction/playback of the waveform in an intelligible manner (preferably in a perceptually acceptable manner) may be employed. For instance, any CODEC that compresses MEL cepstra feature vectors and adds pitch information is preferably employed.
A preferred CODEC is the Recognition-Compatible VoCoder (RECOVC) which is discussed in greater detail below with reference to
Advantageously, when an audio subsystem of a client device employs an audio CODEC having the specific, well defined characteristics (as described above) for capturing and processing speech prior to transmission to a remote server for server-side processing, the main factors that affect the audio characteristics are related to the source and its acoustic environment. This minimizes the degradation of server-side audio processing, thereby providing increased accuracy of complex conversational tasks such as speech recognition and speaker recognition.
In addition, any file format for the encoded audio data that comprises a header which defines information such as the compression algorithm, the size of the file, the audio parameters (feature type and dimension, sampling frequency, compression scheme), as well as other meta-information, if needed, such as language type and ancillary transformation information may be employed herein. In a preferred embodiment described in detail below with reference to
Furthermore, with respect to a streaming mechanism for minimizing the dialog delays introduced by remote processing, the present invention preferably employs RTP by extending the RTP header to enclose the CODEC file format. The resulting stream is referred to herein as RTCCP (Real Time Conversational Coding Protocol). This streaming mechanism is discussed in greater detail below with reference to, e.g.,
Next, with respect to a mechanism for selecting the coding schemes, the present invention preferably utilizes the H.245 control standard by extending H.245 to include any supported conversational protocols. It is to be understood, however, that other protocols similar to H.323 (e.g., SIP) may be utilized.
Moreover, with respect to a control mechanism, a preferred embodiment comprises extending RTCP (Real Time Control Protocol) to produce what is referred to herein as RTCCtP (Real Time Conversational Control Protocol). In particular, RTCCtP extends the functions of RTCP to provide a mechanism for selecting/switching the coding scheme in the middle of a stream transmission and for notification and confirmation. A preferred embodiment of RTCCtP is discussed below with reference to
Furthermore, with respect to a mechanism for sending control data between the client and the speech server, the present invention preferably employs an extension of RTCP (i.e. an extension of RTCCtP) to add the extra information, to produce a control stream that is referred to herein as RTCDP (Real Time Conversational Distributed Protocol). Preferably, the control stream comprises any one or combination of the following: information about the data file (e.g., what data file to use and where to get it from); a description of the type of processing to apply (e.g., algorithm string—sequence of actions to perform on the input or output by the conversational engines); the expected type and format of the results; an address where to return the results; exception handling mechanisms; I/O event notifications (e.g. for a distributed multi-modal browser); and/or modality specific view updates (e.g. ML (markup language) pushes to the modality specific viewing browsers in the multi-modal browser case).
It is to be understood that in a Voice over IP environment comprising RSVP (Resource Reservation Protocol), the RSVP can be employed to allow pre-reservation of specific bandwidth and quality of service between two locations on the network so as to provide extra capability of traffic management.
Referring to
Furthermore, the implementation of such conversational protocols in a distributed environment to provide automatic and coordinated sharing of conversational functions and resources between local and remote applications/devices/servers (without implementing a CVM platform) is described in detail, for example, in International Application No. PCT/US99/22925, filed on Oct. 1, 1999, entitled "System and Method For Providing Network Coordinated Conversational Services," which is commonly assigned and incorporated herein by reference.
Briefly, referring to
The conversational distributed protocols 101 allow networked (distributed) conversational applications 105, 105 a and network-connected devices (local client and other networked devices such as a server) to, e.g., register their current conversational state, arguments (data files) and context, share local and distributed conversational engines 108, 109 between network connected devices (e.g., client/server), and otherwise exchange information to coordinate a "conversation" involving multiple devices or applications including master/salve conversational network, peer conversational network, and silent partners.
The information that may be exchanged between networked devices using the conversational distributed protocols 101 comprises pointers to data files (arguments), transfer (if needed) of data files and other conversational arguments, notification for input, output events and recognition results, conversational engine API calls and results, notification of state and context changes and other system events, registration updates: handshake for registration, negotiation updates: handshake for negotiation, and discovery updates when a requested resource is lost.
Preferably, the conversational distributed protocols 101 also comprise dialog management (DM) protocols that provide a mechanism for exchanging information between dialog managers (DMs) of networked devices. For example, in a distributed environment, dialog management protocols are used for exchanging information to determine which dialog manager will execute a given function. Typically, different devices, CVMs or different applications will have their own dialog manager and context stack. Through the exchange of information via DM protocols, the different dialog managers involved in a dialog session will negotiate a topology with a master dialog manager and slave or peer dialog managers, wherein the active master dialog manager will be responsible for managing the flow of I/O to the different managers to decide the active dialog and appropriately execute a query and update the context and/or history. For instance, the following information can be exchanged: (1) DM architecture registration (e.g., each DM can be a collection of locals DMs); (2) pointers to associated meta-information (user, device capabilities, application needs, etc.); (3) negotiation of DM network topology (e.g., master/slave, peer-to-peer); (4) data files (conversational arguments) if applicable (e.g., if engines are used that are controlled by a master DM); (5) notification of I/O events such as user input, outputs to users for transfer to engines and/or addition to contexts; (6) notification of recognition events; (7) transfer of processed input from engines to a master DM; (8) transfer of responsibility of master DM to registered DMs; (9) DM processing result events; (10) DM exceptions; (11) transfer of confidence and ambiguity results, proposed feedback and output, proposed expectation state, proposed action, proposed context changes, proposed new dialog state; (12) decision notification, context update, action update, state update, etc.; (13) notification of completed, failed or interrupted action; (14) notification of context changes; and/or (15) data files, context and state updates due to action.
In a preferred embodiment of the present invention, the distributed conversational protocols 101 are implemented via extensions of RTP/RTCP (as described below). In another aspect, the distributed conversational protocols may be implemented on top of TCP via RMI (remote method invocation) or RPC (remote procedure call) system calls to implement the calls between the applications and the different conversational engines over the network. As is known in the art, RPC is a protocol that allows one application to request a service from another application across the network. Similarly, RMI is a method by which objects can interact in a distributed network. RMI allows one or more objects to be passed along with the request.
Although the distributed conversational protocols may be implemented via RMI/RPC (as well as DCOM/ActiveX, Cobra, etc.), RTP is preferred because, e.g., RTP (i) takes advantage of the existing/emerging framework of Voice over IP (land and wireless), (ii) provides an open standard approach, (iii) does not make any assumptions on the OS/platform of the different entities, (iv) does not make any assumptions on the engines or APIs used by the different entities, and (v) can take advantage of the functions and services offered in the Voice Over IP framework and (vi) allows (when not encrypted) a third party and intermediary to appropriately modify and/or prepare the RTP stream to increase or improve the user experience.
The speech transmission protocol 103 (or conversational coding protocol) are used by speech transmission clients 107, 107 a to transmit compressed speech (compressed speech file format 104 discussed below) to other networked devices, systems or applications for processing. The speech transmission clients 107, 107 a operate in conjunction with compression, decompression and reconstruction engines 110, 110 a (preferably using the CODEC techniques described below) and suitable compression hardware 111, 111 a for processing the speech (e.g., speech file 104) transmitted over the network. As described below, the speech coders 110, 110 a provide perceptually acceptable or intelligible reconstruction of the compressed speech and optimized conversational performance (e.g., word error rate). The speech is captured (and transformed into features) on the respective networked devices using acoustic signal processing engines (audio subsystems) 112, 112 a and suitable audio hardware 113, 113 a.
In addition, a compressed speech file format 104 can be transmitted/streamed between devices for distributed speech processing using one of the real-time streaming methods described herein in accordance with the present invention. More specifically, the speech transmission protocol 104 allow the devices to transmit compressed speech or local processing results to other devices and applications on the network. In a preferred embodiment, after the handshake process between a source device and a receiver device, a data stream (packet based) is sent to the receiver. The packet headers preferably specify the coding scheme and coding arguments (i.e. sampling frequency, feature characteristics, vector dimensions, feature transformation/family, etc. In addition, error correcting information can also be introduced (e.g. last feature vector of the previous packet to correct the differential decoders if the previous packet is lost or delayed), or appropriate messaging to recover (re-send) lost packets.
The conversational protocols further comprise conversational discovery (detection), registration, and negotiation protocols (or methods) 102. The registration protocols allow networked devices or applications to exchange and register information regarding their conversational capabilities, state/context and arguments, so as to limit data transfer between the devices to relevant information and negotiate the master/slave or peer networking. By way of example, the registration protocols allow the following information to be exchanged: (1) capabilities and load messages including definition and update events; (2) engine resources (whether a given device includes NLU, DM, NLG, TTS, speaker recognition, speech recognition compression, coding, storage, etc.); (3) I/O capabilities (e.g., GUI, Voice, HTML, etc.); (4) CPU, memory, and load capabilities; (5) data file types (domain specific, dictionary, language models, languages, etc.); (6) network addresses and features; (7) information about a user (definition and update events); (8) user preferences for the device, application or dialog; (9) customization; (10) user experience; (11) help; (12) capability requirements per application (and application state) (definition and update events); (13) meta information for CUI services and behaviors (help files, categories, conversational priorities, etc.) (definition and update events, typically via pointer to table); (14) protocol handshakes; and/or (15) topology negotiation.
Registration may be performed using a traditional communication protocol such as TCP/IP, TCP/IP 29, JINI, T-Space, X-10 or CEBus, and socket communication between devices. The devices use a distributed conversational architecture to exchange information such as their conversational arguments (e.g., active vocabulary, grammars and language models, parsing and translation/tagging models, voice prints, synthesis rules, baseforms (pronunciation rules) and voice fonts). This information is either passed as files or streams to, e.g., a CVM controller and the conversational engines, or as URLs. In one embodiment for implementing the registration protocols, upon connection, the devices can exchange information about their conversational capabilities with a prearranged protocol (e.g., TTS English, any text, Speech recognition, 500 words and FSG grammar, no speaker recognition, etc.) by exchanging a set of flags or a device property object. Likewise, applications can exchange engine requirement lists. With a master/slave network configuration, the master dialog manager can compile all the lists and match the functions and needs with conversational capabilities. In addition, context information may be transmitted by indicating passing or pointing to the context stack/history of the device or application that the controller can access and add to its context stack. Devices can also pass information about their multi-modal I/O and UI capabilities (screen/no screen, audio in and out capabilities, keyboard, etc.) The conversational arguments allow a dialog engine to estimate the relevance of a new query by the NLU engine, based on the current state and context.
The conversational discovery protocols 102 are utilized by spontaneously networked conversational clients 106, 106 a of the devices to automatically discover local or network conversationally aware systems and dynamically and spontaneously network-connect such conversationally aware systems. The information that is exchanged via the discovery protocols comprises the following: (1) broadcast requests for handshake or listening for requests; (2) exchange of device identifiers; (3) exchange of handles/pointer for first registration; and (4) exchange of handles for first negotiation. Discovery may also be implemented by accessing a central repository that comprises a description of the registered devices (via, e.g., LDAP (lightweight directory access protocol) or a home page/server that lists the registered devices).
Furthermore, the negotiation protocols 102 allow the negotiation between master/slave or peer networking so as to provide the appropriate coordination between, e.g., multiple CVM systems in dynamic master-slave and peer-to-peer interactions. More specifically, multiple CVM devices when registering will add to the conversational registration capability, information pertaining to, e.g., their controlling capability, the conversational engines that they have access to, and applications and devices that have registered with them and that they control. Based on their UI, I/O capabilities and active I/O, one CVM controller becomes the master and the other CVM controllers act as slaves, which is equivalent relatively to the master as being registered applications until a new negotiation occurs. The role of master and slave can be dynamically switched based on the active I/O modality or device or based on the active application.
II. Conversational Codec
As indicated above, one component of conversational protocols for implementing for distributed conversational networking comprises a suitable audio coding/decoding (Codec) protocol. Referring now to
The MFCC feature vectors are preferably compressed via MFCC compression module (208) using any technique known to those skilled in the art that provides compression without effecting the performance of the speech recognition system. Preferably, the compression module 208 preferably implements the compression scheme disclosed in the above-incorporated U.S. Pat. No. 6,009,387 (although other suitable compression schemes may be utilized). The compression scheme disclosed in this patent utilizes a first order prediction, multistage split VQ technique. Preferably, the bit rates are in the range 4–6.4 kbps, depending on the size of the MFCC feature vector. It is to be appreciated that the preferred compression approach is flexible in terms of acoustic feature characteristics such as dimensions or sampling rates. It is to be further appreciated that when used in combination of robust front-ends, the features may be compressed prior to transformation. The transformations are transmitted separately as described in detail below. On the receiving end, the transformations are applied after decompression.
To provide speech reconstruction and playback using the MFCC feature vectors, an additional pitch frequency information (including voiced/unvoiced decisions) is extracted for every frame of speech data via a voice decision and pitch detection module (207) together with the respective MFCC feature vector. It is to be appreciated that the pitch data is efficiently calculated from the STFT module 205 using a spectral peak detection process. It is to be understood that for some speech recognition systems, especially for tonal languages (e.g. Mandarin Chinese), the pitch information that is used for recognition and pitch detection is already implemented as a part of the front-end process.
The pitch period values are compressed at bit rates of 300–500 bps via a pitch compression module 209. The streams of compressed MFCC feature vectors and the compressed pitch are multiplexed via MUX 210 to form an output bitstream (of coded cepstra and pitch) for storage and/or transmission.
Referring now to
It is to be appreciated that the speech for playback is reconstructed from the decoded MFCC feature vectors and the decoded pitch values via a sinusoidal speech synthesis module 216, which preferably employs a novel, low complexity, frequency domain reconstruction method described in detail in the above-incorporated patent application U.S. Ser. No. 09/410,085. The reconstruction is performed using a sinusoidal speech model (such as described by R. Mc Aulay et al., Sinusoidal Coding, Speech Coding and Synthesis, Chapter 4, pages 121–170, Elsevier, 1995.) The values of the model parameters are determined such that the reconstructed speech has an MFCC feature vector similar to the decoded MFCC feature vector, and a pitch similar to the decoded pitch. This is sufficient to reconstruct natural sounding, good quality, intelligible speech with the voice of the original speaker.
It is to be appreciated that the RECOVC system described above using a cepstral feature compression scheme minimizes the level of degradation of the performances of a conversational task performed on the decompressed feature stream. The preferred compression scheme is a key basic element of conversational networking. It is to be understood, however, that any suitable coding scheme that compresses the cepstral feature stream while minimizing the distortion of the reconstructed features may be used herein. In addition, for practical purposes, a preferred coding scheme for use in conversational distributed environment is one that supports reconstruction of intelligible waveforms. Indeed, this reconstruction is useful for later playback from the server or playback from the client (if stored locally) or for subsequently proofreading the transcription, error correction, or human monitoring of the process. Accordingly, any conversational CODEC that minimizes the distortion of the acoustic front-end features and allows reconstruction of intelligible waveforms may be employed herein. For example, any conventional CODEC combined with an acoustic feature error correction/minimization scheme would fit the definition. Preferably, such coding schemes should provide data rates as low as between 4 kbits/s and 5 kbit/s with no degradation of the recognition performances. As a result, interactive exchanges can be performed in real time with the back-end (server) resources even over wireless modems or wireless data links.
It is to be understood that although a preferred CODEC system and method is described above, it is to be appreciated that the transmission of speech from the local client to a remote network-connected server (or vice versa) can be performed using other techniques depending on the circumstances and desired results. For instance, there can be direct transmission of the waveform as a file, a stream or a stream of packets. In addition, a compressed waveform may be transmitted using conventional methods such as ADPCM and APC. Furthermore, a stream of features can be transmitted in accordance with the method disclosed in "Compression Of Acoustic Features For Speech Recognition In Network Environments," by G. Ramaswamy et al., Vol. 2, pp. 977–980, Proc. ICASSP, 1998, which is incorporated herein by reference. This method allows recognition (speech recognition, speaker recognition or NLU) on the receiver side but no reconstruction of the signal.
III. Conversational Coding Protocols
(A) File Format
As indicated above, one component for defining a conversational coding protocol comprises a definition of the file format that is associated with the encoded data. In a preferred embodiment, the CODEC system and method (and feature compression scheme) described above (i.e., RECOVC) is used for generating an internal file format that can be utilized for real-time distributed conversational interactions. Referring now to
A preferred format of the RECOVC file comprises a plurality of Blocks, each comprising compressed feature vectors of several successive 10 msec audio frames, for example. More specifically, in a preferred embodiment, each Block comprises a single IntraFrame (comprising uncompressed or losslessly compressed) speech features and one or more InterFrames having speech data coded using RECOVC. More specifically, an IntraFrame is the first frame of a Block that is preferably non-encoded or, alternatively encoded by different schemes that guarantees that the IntraFrame can be recovered/reconstructed, even if previous blocks or frames have been corrupted. Moreover, an InterFrame is a frame between IntraFrames. The InterFrames may be coded differently than the IntraFrames, as it may be less critical to have them corrupted (since the stream will be recovered at the next IntraFrame. Robust encoding, including error correcting codes may be used for the InterFrames.
The (maximum) number of frames N1 for each Block is specified in the File Header. The feature vectors are stored in Blocks in such a way that each Block can be decompressed on its own. It is to be appreciated that this allows decompression to be performed at any portion (e.g., the middle) of the RECOVC File, as well as skipping damaged or missing data.
The RECOVC File further comprises one or more Segments, comprising, e.g., speech and silence segments, all of which are preceded by a corresponding Segment Header. For instance, each speech Segment comprises several Blocks and a Speech Segment Header indicating the type of content (e.g., speech). The Speech Segment Header specifies the number of frames (N2, N4) per speech Segment. The RECOVC file further comprises one or more of Silence Segments and EOS Segments (end-of-stream), as well as ancillary data segments that may be defined depending on the application.
Referring now to
A Cepstra Dimension field (
Referring to
The Number of Frames field comprises a value that indicates the total number of frames of the corresponding Segment. As further illustrated in
(B) Conversational Streaming and Control Mechanisms
As indicated above, a suitably defined streaming mechanism is implemented to transmit the RECOVC file for distributed conversational applications. Packetization of the RECOVC file format is preferably achieved by buffering the data stream block by block (and initially sending the header). Typically with 300 ms packets, the data rate can be as low as 4 kbit/s (4.5 kbit/s when reconstruction of the waveform is not required). This is sufficient for real-time low bit rate transmission even over wireless modem and real-time interaction. Packetization will be discussed in detail below.
In a preferred embodiment, packet transmission of a RECOVC data stream (as shown in
A brief discussion of a standard RTP protocol will now be provided with reference to the diagram of
The RTP header 700 comprises a version number field 701 (2 bits) which identifies the version of RTP. The most current version of RTP is version "2". A padding (P) field 702 comprises a 1 bit field, whereby if the padding bit is set, this indicates that the packet contains one or more additional padding bytes at the end which are not part of the payload. The last byte of the padding contains a count of the number of padding bytes that should be ignored. This padding (bytes of value 0) is added to the end of the payload of an RTP packet so as to maintain the 32-bit fields aligned at offsets divisible by four.
An extension (X) field 703 is a one bit field that is set to indicate that a variable-length header extension is appended to the RTP header, following the CSRC list 710 (if present). A CSRC count (CC) field is a 4 bit field that indicates the number of CSRC identifiers that follow the fixed header (i.e., the first 12 bytes). A marker (M) field 705 is a 1 bit field that carries profile-specific information. A profile specifies a default static mapping of payload type codes to payload formations. The marker is intended to allow significant events such as frame boundaries to be marked in the packet stream. A profile may define additional maker bits or specify that there is no marker bit by changing the number of bits in the payload type field 706.
The payload type field 706 is a 7 bit field that identifies the format of the RTP payload and determines its interpretation by the application. The RTP payload is the data transported by RTP in a packet. As indicated above, a profile specifies a default static mapping of payload type codes to payload formats.
A sequence number field 707 is a 16 bit field that comprises a sequence number of the RTP packet. The sequence numbers allows the receiver to reconstruct the sender's packet sequence. The sequence numbers may also be used to determine the proper location of a packet, for example in audio decoding, without necessarily decoding packets in sequence. The sequence number increments by one for each RTP data packet that is sent, and may be used by the receiver to detect packet loss and to restore packet sequence.
A time stamp field 708 is a 32 bit field that indicates the time of sampling of the first byte in the RTP data packet. The time stamp may be derived via NTP (network time protocol) or other clocking methods known to those skilled in the art for providing synchronization depending on the application.
A synchronization source (SSRC) identifiers field 709 is a 32 bit field that indicates the synchronization source of a stream of RTP packets. This identifier is chosen randomly and is identified by a 32-bit numeric SSRC identifier carried in the RTP header so as not to be dependent upon the network address.
The CSRC identifiers 710 field is a 32 bit field that identifies the contributing sources, if any, for the payload contained in the packet. A source of stream of RTP packets that has contributed to the combined stream produced by an RTP mixer. The mixer inserts a list of the SSRC identifiers of the sources that contributed to the generation of the particular packet into the RTP header of that packet. An example application is audio conferencing where a mixer indicates all the persons who speech was combined to produce the outgoing packet so that the receiver can determine the current talker, even though all the audio packets contain the same SSRC identifier (i.e., the SSRC identifier of the mixer).
In accordance with a preferred embodiment of the present invention, the RTP format described above with reference to
The RTP header extension length field 802 is a 16 bit field that counts the number of 32-bit words in the extension, excluding the 4-bytes comprising fields 801 and 802 of the RTP header extension. Moreover, in a preferred embodiment, the codec header field 803 comprises the RECOVC header (
Referring now to
If desired, error correction can be performed on a block by block basis. Preferably, a data Segment, defined by the RECOVC file formation, can be included which contains the error recovery information. More specifically, as shown in
It is to be appreciated that, as discussed in detail above, the RTCCP can run on top of an unreliable protocol such as UDP for real-time applications. When real-time is not an issue, RTCCP can be implemented on top of a reliable transport layer that will guarantee appropriate packet ordering and delivery such as TCP (transmission control protocol). This is illustrated in
As indicated above, because multiple conversational codecs can be utilized (e.g. RECOVC with different settings), a protocol/mechanism should be defined to select a coding scheme. For instance, the endpoints, e.g., source and receiver, must negotiate to determine compatible settings before the audio data and/or data communication links can be established. The present invention preferably utilizes the control functions defined by the H.245 standard (which is known to those skilled in the art), which specifies messages for opening and closing channels for media streams, and other commands, requests and indications to provide such control functions. More specifically, an initial connection between a source and receiver starts with a preliminary handshake, similar to H.245, except that it incorporates all the different conversational codecs (e.g., RECOVC) that are employed in the given application. The extension of the H.245 control and handshake protocol is referred to herein as H.245.RTCCP.
More specifically, the default for transmission is set to RECOVC.101 (rather than G.711 (audio codec, 3.1 Khz at 48, 56, and 64 Kbps (normal telephony) or G.723 (Audio codec, for 5.3 and 6.3 Kbps modes) as currently prescribed by H.245) which is supported by all end points in the network. Aurora DSR or other schemes may also be supported. In real-time mode, RECOVC.101 is a preferred default codec that is initially enabled/selected upon system connect unless an agreement is reached to select another coding scheme before completion of the handshake, in which case the agreed upon coding scheme will be implemented.
As further indicated above, a control protocol/mechanism should be defined for switch a coding scheme in the middle of a RTCCP stream transmission. In accordance with a preferred embodiment, notification and confirmation messages are transmitted as control extensions to the RTCP (Real Time Control protocol), resulting in what is referred to herein as RTCCtP (Real time Conversational Control Protocol). This architecture is illustrated in
As is known in the art, RTCP is based on a periodic transmission of control packets to all participants in a session, using the same distribution mechanism as the RTP packets. The underlying transport protocol must provide multiplexing of the data and control packets, for example, using separate port numbers with UDP. As is further known in the art, the RTCP specification defines several RTCP packet types to carry a variety of control information, where each RTCP packet type is allocated a unique identification code. For instance, the RTCP packet types include sender reports (SR) (code 200) for transmission and reception statistics from participants that are active senders, as well as receiver reports (RR) (code 201) for reception statistics from participants that are not active senders. RTP receivers provide reception quality feedback using RTCP report packets which may be SR or RR reports, depending on whether or not the receiver is also a sender. The only difference between the sender report (SR) and the receiver report (RR) forms, besides the packet type code, is that the SR includes a 20-byte sender information section for use by active senders. The SR is issued if a site has sent any data packets during the interval since issuing the last report or the previous one, otherwise the RR is issued. Other packet types include source description (SDES) (code 202) packets comprising source description items such as CNAME (canocial end-point identifier), BYE packets (code 203) to indicate end of participation, and APP packets (code 204) for application specific functions.
As is known in the art, each RTCP packet begins with a fixed header similar to that of RTP data packets, followed by structure elements that may be of variable length according to the packet type, but which always end on a 32-bit boundary (so as to allow RTCP packets to be "stackable" or concatenated to form a compound RTCP packet that is sent in a single packet of the lower layer protocol, e.g., UDP).
In accordance with a preferred embodiment of the present invention, in addition to the conventional RTCP functions, RTCP is extended to RTCCtP to include application specific functions for conversational distributed functions. More specifically, in addition to the conventional RTCP functions, RTCP sender and receiver reports, for example, can be extended with suitable profile-specific extensions to support coding scheme notifications (signal/agree on changes of coding schemes). Other application specific extensions for conversational distributed functions include, e.g., RTCCtP identifiers, header extension length, code bits for RTCCtP functions, packet receipt request and acknowledgments, and codec change notification/request for confirmation, etc. These messages are propagated through the RTCP layer associated with the RTP stream.
By way of example, for purposes of error correction, RTCCtP messages can require packet repetition and provide the packet sequence number of the packets to be repeated. In one embodiment, the RECOVC header of the RTCCP packet is repeated based on the receivers report (RR) in the RTCCtP stream. In a default case, the RECOVC header is repeated until confirmation is obtained from the receiver. The receiver must confirm to the sender the receipt of an X=1 packet and provide the packet ID/sequence number.
It is to be understood that in the absence of RTP/RTCP, to provide control, the source may transmit the RECOVC header until confirmation is received by the source for all registered receivers. Moreover, in the absence of RTP support by the transport layer, similar functions must be emulated between clients and servers. Furthermore, in the absence of RTCP support by the transport layer, similar functions must be emulated between the clients and servers.
It is to be appreciated that in accordance with another aspect of the present invention, RTCCtP may be further extended to transmit other application-specific control data between, e.g., a client (source) and a speech server (receiver) for providing conversational distributed functions. In a preferred embodiment, when additional RTCP packets are needed for immediate information transfer, the APP RTCP packet type (code 204) noted above is preferably utilized to provide an application-specific extension for implementing the conversational distributed functions. A preferred complete control protocol is referred to herein as RTCDP (Real-Time Conversational Distributed Protocols). This is illustrated in
Referring to
It is to be appreciated that depending on the application, any suitable application-specific control data can be transmitted between, e.g., a source and a receiver using RTCCtP for providing conversational distributed functions. For example, the protocols and APIs described above in connection with the above-incorporated International Appl. Nos. PCT/US99/22927, filed on Oct. 1, 1999, entitled: "Conversational Computing Via Conversational Virtual Machine," and International Application No. PCT/US99/22925, filed on Oct. 1, 1999, entitled "System and Method For Providing Network Coordinated Conversational Services," may be implemented to transmit control parameters and messages to support remote control of a speech engine (e.g., start/stop recognition), determine type of recognition to perform (e.g., speech, TTS, speaker recognition, NL parsing, NL tagging, Dialog Management, etc.), what data files to use (e.g., grammar files, acoustic models, language models, tagger data files, parser data file, dialog information, etc.), where and how results of, e.g., a recognition, should be sent, as well as messages that are needed to register, negotiate, and synchronize different engines.
Furthermore, with Voice Browsers and Multi-Modal Browsers (as described below) and other applications, the control messages of RTCCtP may be transmitted as XML data (e.g., URLs pointing to particular algorithms, data files, and engines to be implemented) or byte code representation of XML tags (preferably, XML name space convention according to CML) and values associated with necessary control information. Such control information comprises: field identifiers and/or browser event identifiers (when also sent to Multi-modal shell (described below); argument data file(s) for the engines; format of the result/output to be specified (e.g., audio format (e.g., RTP stream) or text (ASCII, XML, attribute value pairs) or function call), with extra tag information and address of browser to push data; address and method/protocol to send results (back to browser or content server); identifier for the results, and commands to execute. Furthermore, when the stream is sent to a speech server, the XML tags associated with the active input are sent: field information for a directed dialog, active forms (or URLs of forms) for mixed initiative NL, etc. It is to be noted that the packets of streamed XML and protocols may be implemented using SOAP (simple object access protocol). In summary, RTCCtP may be used to transmit all types of control messages depending on the implementation.
Referring now to
Advantageously, this framework allows the engines 1431 and browser application 1421 to disregard the fact that the other component is local, remote, or distributed. Between the proxies, the RTCDP protocols assure real-time exchange of the control parameters. Again, the RTCDP control stream exchanged between the proxies 1420, 1430 may comprise information such as argument data file(s) for the server engines, additional feature transformations, addresses where to send the results (back to browser or to content server), format of result (text, XML or Audio RTP stream), extra tag information and address of browser or server where to push data, identifier for the results, commands to execute, data file: what data file to use and whereto get it from; description of the type of processing to apply, e.g. algorithm string—sequence of actions to perform on the input; expected type and format of the results; address where to return the results; exception handling mechanisms; I/O event notifications (e.g. for a distributed multi-modal browser like DOM (document object model) level 2 events); modality specific view updates (e.g. ML pushes to the modality specific viewing browsers in the multi-modal browser case), etc.
For this application, the proxies operate as described above with reference to
Indeed, in alternate embodiments of the invention, RTCDP control of, e.g., remote conversational engines can be implemented via remote APIs (e.g., RMI (preferably JSAPI (java speech API with extensions) or RPC) to the engines which precedes argument audio streams, although higher level control is still preferably performed via RTCCtP. The remote calls preferably use TCP (over IP) or any other transport mechanism that reliably guarantees message delivery.
The overall conversational protocol architecture (or umbrella stack) according to a preferred embodiment of the present invention is illustrated by the diagram of
Advantageously, the use of RTP-based conversational protocols as described herein guarantees that the conversational protocols are compatible with, and can be extended to, any network (existing or future) that supports streamed data and Voice over IP or packet voice communications. For example, well-known protocols such as H.323 and SIP (session initiation protocol), which rely on RTP/RTCP can be readily extended to implement the conversational protocols described herein. Moreover, other types of wireless networks can use similar designs adapted to the peculiarity of the underlying communication protocol layers.
Further, as indicated above, it is to be understood that the above-described functions could be directly supported on top of TCP, HTTP or other transport protocols, depending on the important of real-time versus guaranteed packet delivery, using the same conversational protocols and header extensions.
Referring now to
More specifically, in
Preferably, the RTSCDP protocol is used when control of the conversational engines 1602 is performed by the controller 1603 and not the source 1601. In such a case, it is preferable to ship the audio from the source 1601 directly to the server 1602 engines, instead of shipping audio from the source 1601 to the controller 1603 (browser), and then having the controller 1603 ship the audio and control data to the server engines 1602.
If the audio is not shipped from the controller 1603, it does not utilize the RTCCtP layer. But in a Voice over IP environment, for example, the RTSP protocol has been explicitly developed to act as a remote control of an appliance/service (i.e., controller 1602) acting on a RTP stream with appropriate synchronization features with the RTP stream when needed. Therefore, given the current VoIP framework, it is advantageous to extend RTSP to add the conversational control messages (transmitted between the controller 1603 and server 1602) on top of RTSP to control the conversational engines that act on the RTCCP/RTCCtP stream received by the source 1601.
IV. Distributed Conversational Networking Examples
Referring now to
Conventional implementations of speech browsers assume local processing of speech. This is true for browsers that are local on pervasive clients or remote on servers (e.g. telephony servers). It is to be appreciated, however, that the speech recognition engine (and other conversational engines) can be remotely located from the client device, machine, or platform that captures the speech. Indeed, within the Voice XML 0.9 specifications, this can be artificially implemented through a grammar specification.
By way of example, as illustrated in
The client device 1701 may be, for example, a desktop PC (personal computer), a PDA (personal digital assistant), an automobile computer, a smart phone or a conventional telephone. The client 1701 may also comprise one or more speech-enabled local applications 1706 (and a database 1707) running on the client 1701. The client utilizes using conversational protocols described herein to communicate with the speech browser 1705. For example, the local application may be a car navigation application in which a "Speech Navigation Application" interacts with computer mapping software and a GPS (Global Positioning System) device to provide conversational driving directions. In addition, the local application may be a local speech browser, wherein the functions between the local speech browser and speech browser 1705 are distributed. In addition, functions between a local speech engine and remote speech engine may be distributed.
The browser server 1702 can access any one of a plurality of server systems S1, S2, and S3 over network (e.g., the Internet) using a standard network protocol (e.g., HTTP, TCP/IP) to access VoiceXML pages on behalf of the client device 1701/local application 1706 and parse and process the page/transaction via the speech browser 1705. For example, the speech browser 1705 can connect to server S1 to access existing HTML information via a transcoding proxy that transcodes, e.g., legacy HTML documents to Q VoiceXML documents. In addition, the speech browser 1705 can connect to a specialized web server application (S2) such as Lotus Domino server to access Notes data (e.g., Notes e-mail) via a CGI application. In particular, the Domino server can be configured to generate VoiceXML pages and transmit pages using HTTP. In another example, the speech browser 1705 can connect to a web server application (S3), using a CGI application or Java Servlet to access an legacy database of an enterprise, wherein the web application generates and transmits the information in VoiceXML.
In the exemplary distributed system of
It is to be appreciated that the system of
Indeed, rather than redirecting the RTCCP sent from the client 1701, it is advantageous to send the RTCCP stream to the browser 1705 which redirects or multi-casts the RTCCP stream appropriately (this is different from the method described above with reference to
Alternatively, the conversational protocols described herein (e.g., RTCCtP/RTCDP) provide a mechanism whereby the speech browser 1705 can communicate with the client 1701 to advise the client 1701 where to direct the RTCCP stream for remote processing. For instance, as shown in
In general, as described in the above-incorporated applications, a multi-modal browser comprises a multi-modal shell that parses and interprets CML (multi-modal) documents and mediates among, and coordinates synchronized information exchange between, multiple modality specific browser components (e.g., a visual browser and a speech browser). In one embodiment, content pages and applications are implemented in a gesture-based single authoring CML format, wherein conversational gestures are elementary dialog components that characterize the dialog interaction with the user and provide abstract representation of the dialog independently of the characteristics and UI offered by the device or application rendering the presentation material. The multi-modal browser processes a gesture-based CML document using specific predefined rules to automatically transcode the gesture-based CML document to any supported presentation modality or modalities of the particular browser or device (e.g., transcoded to the appropriate declarative language such as HTML, XHTML, or XML (for automated business-to-business exchanges), WML for wireless portals and VoiceXML for speech applications and IVR systems, etc.), as well as provide tight synchronization between the different views supported by the multi-modal browser.
In another embodiment, CML may be implemented by incorporating a plurality of visual and aural markup languages (i.e., a CML document that comprises sub-documents from different interaction modalities). For example, a CML document may be implemented by embedding markup elements from each of a plurality of represented/supported modalities (e.g., VoiceXML and HTML tags) in the same file using synchronizing tags to synchronize the different ML content (i.e., to synchronize an action of a given command in one modality with corresponding actions in the other supported modalities) on an element-by-element basis using, for example, the techniques described in the above-incorporated application International Appl. No. PCT/US99/23008, as well as U.S. Ser. No. 09/507,526.
As shown in
It is to be appreciated that conversational protocols described herein (e.g., RTCCP, RTCCtP/RTCDP) may be used to implement low-bandwidth Voice over IP. For instance, using RECOVC described herein, the H.323 protocol stack (which is a standard is known in the art for a set of protocols providing voice, video and data conferencing over packet-based networks) can be readily extended to encompass RECOVC (i.e., H.323.RTCCP) and add conversational networking as a basic Voice over IP feature. Indeed, all other Voice over IP protocols such as H.323 that implement RTP can be extended using the conversational protocols described herein to allow direct two way voice communications between a regular device (e.g., telephone) and a device connected on a low bandwidth network, while also preserving capabilities to offer conversational functions.
By way of example,
AS further illustrated in
It is to be understood that when conversational application are widely distributed across a network, mechanism should be employed to mitigate traffic and delay and some quality of service must be guaranteed and accordingly the network must be managed to provide this quality of service. This is implemented with conventional methods, however new consideration must be added to the cost functions to optimize. Indeed, the conversational distributed systems described herein require:
This impacts the management (network and server systems) and renders even more acute the problem of intelligent network caching (not only of the content/business logic but also of the data files), storage, traffic routing, etc. Again all this is done using conventional method the novelty of the invention is that the optimization criteria has changed.
Although illustrative embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the present system and method is not limited to those precise embodiments, and that various other changes and modifications may be affected therein by one skilled in the art without departing from the scope or spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as defined by the appended claims.
Patent Citations
Referenced by
Legal Events
Original Image


text:
Monarch Teaching Technologies (MTT), the makers of VizZle®, educational software for visual learners, and n2y, the makers of the popular special education curriculum Unique Learning System®, the News-2-You adapted weekly newspaper and SymbolStix™ dynamic symbol set, yesterday announced a partnership. Subscribers to both the Unique curriculum and VizZle can now create interactive lessons using SymbolStix within VizZle to support the Unique curriculum content. The resulting lessons will be available to all VizZle subscribers at no cost.
"This is the future of education," said Terry Murphy, CEO of MTT, "Providing common core aligned curriculum with touch technology and multimedia will allow Unique and VizZle subscribers to engage the most challenged students. Teachers can customize educational content in VizZle, add interactivity, fun factors and multimedia, all proven to reach these students. They can then share those resources with the community, saving time and increasing effectiveness across the board."
VizZle subscribers will have free access to collections of lessons that have been moderated and approved by both VizZle and Unique to support the Unique curriculum. Lessons will be added continuously throughout each month as users share the content they create. Subscribers to both Unique and VizZle can import SymbolStix Online images for use in VizZle lessons, and freely share them within the VizZle Library. Future plans for the partnership include direct access to SymbolStix within VizZle.
"We are excited about sharing our content with yet another company with such a unique service," said Jacquie Clark, President and CEO of n2y. "Unique Learning System provides the foundation for standards-based learning in Preschool through Transition special education classrooms. We are excited to support those teachers looking to extend the monthly lessons using VizZle software. And now Unique subscribers will also be able to directly access at least four VizZle supports per grade level band every month in the Supporting Files section of the [Unique Learning] dashboard."
About Monarch Teaching Technologies, Inc. (MTT)
MTT (monarchtt.com) and its VizZle visual learning software system was originally conceived of at of the Monarch Center for Autism in Shaker Heights, OH, a not for profit agency which provides a full array of education and residential services for people with autism across the lifespan. After a four year R&D collaboration with researchers funded in part by NIH and US
Department of Education grants, MTT introduced VizZle, a web based visual learning resource, in 2009. Now teachers, therapists, students and parents across the country collectively spend over 15,000 hrs/month on VizZle using any of the almost 8,000 lessons from the moderated shared library, creating lessons from in program templates and over 14,000 images, audio and video clips, and tracking outcomes by IEP goals or Common Core Standards.
About N2Y
n2y is a family‐run, online education publishing company started by Jacquie Clark, a speech‐language pathologist with over 30 years experience in and out of the classroom. n2y has grown to include a weekly newspaper, online curriculum, and dynamic symbol set. The New‐2‐You current events newspaper utilizes concise, symbol‐supported text to deliver current events news to a population previously denied this information. Unique Learning System is the only comprehensive web‐based and standards‐based curriculum specifically designed for special learners. SymbolStix ONLINE offers subscribers the newest and most dynamic symbol set ever developed. The set is used exclusively throughout n2y products. For additional information, please visit us at www.n2y.com. We can also be reached by e‐mail at email@example.com or call us at (800) 697‐6575.

subset: commoncrawl


text:
The fairgrounds are requesting all exhibitors and spectators to enter and exit thru the gate by the new bridge (by the horseshoe pitch) You are not to enter thru the main gate and go by the grandstands.
Also the harness horses will be permitted on the track until 8:50am, so please becareful when entering and crossing the harness track.
Welcome to Silver Saddles Horse Club!! We hold several horse shows a year at the Washington County Fairgrounds, Washington, PA. We offer classes for everyone! Leadline, walk-trot 12 & under, walk-trot 13 & over, junior & open pleasure, trail and equitation classes. Plus contest classes for leadline, walk-trot 12 & under, pony and horse. Our shows also have some jackpot classes. You are welcome to join our club and attend our annual awards banquet. We offer family and single memberships.
Please check out the calendar for show dates and show bills (event page). We will gladly post your show info. Please send information to:


text:
Don't underestimate the power of teamwork.
A survey's best performing companies reported only a 1.6% annual cost increase.
Wellness programs delivered just a 50 cents on the dollar return, while disease management programs returned $3.80 for each dollar spent.
Employers had better start getting used to accommodating all those baby boomers who, according to a Gallup poll, plan on working past the normal retirement age.
Keas survey of 100 HR professionals find many ready for new PPACA mandates. Aflac survey finds few say they are "very prepared."
Oregon benefits attorney: "Really is about the bottom line."
While employee benefits managers worry over health packages in the light of Obamacare, CFOs fret over how to pay for those same benefits.
There are lessons to be learned about the ongoing debate that are applicable to all of us.
Exchanges will provide opportunities for credit unions to better anticipate healthcare costs, says Brad Pricer, CUNA Mutual human resources process leader.
As if it weren't enough that some baby boomers are adjusting to not being able to retire when they would like to, many of them are now dealing with con artists targeting them for all kinds of fraudulent schemes.


text:
Emcee Melvin HO!
Hello! My name is Melvin Ho and I'm your Professional Emcee based in Singapore. Drop me a text to check my Availability by clicking the button below.
Let's work together to make your event a Great Success!
Contact Melvin
Client Testimonials
Excerpts from our client testimonials. For full testimonials, please visit Here
"Thanks for the GOOD job done!"
- April Lee, Committee member for DND, HBO Asia
"IBC Asia will definitely engage Melvin again"
– Sukumar Verma, General Manager, IBC Asia
"Melvin was totally awesome and brought down the house!"
- Victor, org comm member for DND, SingTel mio TV
"Melvin really brought up the atmosphere!"
– Jassemine Seah, Executive Secretary to CEO, ThaiExpress


text:
A Answers (2)
Stacy Wiegman, PharmD, Pharmacy, answeredThe gymnema in glucose balance supplements has been known to assist in weight loss. The reason is that gymnema decreases sweet cravings, which may be effective in discouraging the consumption of sweets while dieting. So while glucose balance supplements are not technically part of an official weight loss plan, they can be helpful when taken in combination with a healthy diet and regular exercise.
Julia Havey, Nutrition & Dietetics, answered
Dr. Oz has had PGX featured on his show at least 4 times. Dr. Mark Hyman discussing it most recently. Why? Because PGX is showing significant ability to help stabilize blood sugar levels. When that happens, a person doesn't have the dramatic ups and downs in fluctuation and that helps reduce cravings. Another effect of PGX is that it fills the stomach up with fiber, a sticky, semi-solid mass of fiber that makes the stomach stop sending appetite hormones to the brain and signals satiety or fullness. This feeling of fullness lasts for hours after a healthy sized meal and thus makes appetite control real and possible, and that helps with weight loss.
I personally have lost 40 lbs. using PGX, and have heard from people losing over 100 with it!


text:
Updated: 03/15/2013 3:26 PM KSTP.com By: Leslie Dyste
A construction project is underway along Highway 36 in Maplewood.
According to MnDOT, during the construction crews will remove a stoplight on a busy stretch of Highway 36.
The project will permanently close the right turn lane from eastbound Highway 36 to southbound English.
A portion of Highway 36 will close for two months in mid-June during construction.
A nearby trail will also temporarily close during the construction.
MnDOT says the construction will improve safety and efficiency along the Highway 36 corridor in Maplewood.
Click here to learn more about the project.


text:
Series: Saturday Science Links (27 entries)
The biggest science stories of the week are reviewed.
The Resource Finder has powerful filtering features — check out the tutorial to learn more.
The biggest science stories of the week are reviewed.
Is Christianity anti-science? Can Christians be good scientists? Is science the only source of true knowledge? Pastor Dave Gustavsen of Jacksonville Chapel addresses these issues.
A short guide to where BioLogos fits in the origins debate.
President Haarsma reflects on our big interview with Bill Nye, and explains how it relates to our mission.
By getting rid of the miracle stories in the Bible, Bultmann and his followers hoped to make the Christian story more palatable to modern man. Although I recognize the emotional weight of this sentiment, I am not convinced that it is an intellectually coherent approach, mainly for reasons of self-consistency.
In the early 1700s, Isaac Newton and Gottfried Wilhelm Leibniz famously disagreed about whether God periodically "corrected" the orbits of the planets. Why their disagreement matters for today's debates about God and evolution.
Evangelicals are skeptical of space exploration—but can pastors change this from the pulpit?
The former chair of the Harvard University physics department muses on how faith gives meaning to his scientific work.
The story of the Genesis Rock reminds us of how biblical and scientific accounts, despite their different purviews and purposes, remain inexorably linked in our understanding of origins.
Theology needs science, but science needs theology; there can be no two-state solution.
In this video originally featured in March of 2012, Dr. David Finch, a biologist at New York University, discusses his thoughts on both Creationism and the effects of "new atheists" like Richard Dawkins.
Here are some of the best responses from around the web to David's Barash's controversial editorial about God and Evolution in the New York Times.
Are black holes in trouble? Could life have come from space? What do we know about the religious lives of Indian scientists? Find the answers to these questions and more.
New BioLogos content editor Brad Kramer reviews Chapter 1 of "Surprised by Scripture" by NT Wright, and sees Wright's work as a signpost for the future of the science/faith debate.
This week's news features volcanoes, skeletons, and stars, as well as a thought-provoking new book from InterVarsity press.
From science and religion blogs: quantum uncertainty and God, the remarkable fact that we have come to understand our place in the created order, and the role of theology in making wise choices about the use of technology.
News about the Ebola outbreak and initiatives in genomic medicine lead our news roundup this week, with a collection of research and learning opportunities in biology and at the science/faith border to follow.
The purpose of Jesus's art was to give verbal, visual, and dramatic forms to those complicated and confounding relationships, symmetries, and harmonies between himself, the father and spirit, and between the triune God and the world… Such creative expressions did not and do not make everything clear, but rather resist simple clarity, forcing their hearers to come at the whole complicated truth from a position of intellectual and spiritual humility.
When the boxer and the biologist collided that November evening, they both had a substantial following, and they presented a sharp contrast to the audience: a pugilistic, self-educated fundamentalist evangelist against a suave, sophisticated science writer. When it comes right down to it, not all that different from Ken Ham versus Bill Nye, except that Ham has a couple of earned degrees where Rimmer had none.


text:
August 4, 2015
Firefighters say the brush fire was put out shortly before 4 PM. The evacuation order for Riley Road has been lifted.
A fire burning in the northeast part of Brazos County has forced officials to order an evacuation for Riley Road.
According to Brazos County dispatch, the ten acre fire is burning on Riley Road, heading north east toward F-M 2038.
Officials are asking people to avoid the area. Anyone who lives there is asked to evacuate and find a safe place to stay.
Right now firefighters from three different agencies are on scene fighting the fire -- men and women from Brazos County Precincts Two, Three and Four.
They've also asked for assistance from a helicopter, to drop water over the fire.
We have a crew on scene and will be bring you the latest information as it becomes available, on air and online.
To comment, the following rules must be followed:
Comments may be monitored for inappropriate content, but the station is under no legal obligation to do so.
If you believe a comment violates the above rules, please use the Flagging Tool to alert a Moderator.
Flagging does not guarantee removal.
Multiple violations may result in account suspension.
Decisions to suspend or unsuspend accounts are made by Station Moderators.
Questions may be sent to email@example.com. Please provide detailed information.
Click Here to access the online Public Inspection File
Viewers with disabilities can get assistance accessing this station's FCC Public Inspection File by contacting the station with the information listed below. Questions or concerns relating to the accessibility of the FCC's online public file system should be directed to the FCC at 888-225-5322, 888-835-5322 (TTY), or email@example.com.


text:
Just want the recipe? Click Here.
Steamed potatoes. Don't they sound, I don't know, kind of boring? Truth is, steamed potatoes are not something I'd made--or thought about--until recently.
For my friend's birthday party, her husband made an incredible clam steam. Along with the clams, sausages, and corn, there was, you guessed it, steamed potatoes. At first I thought I'd give the steamed potatoes a pass. I mean, they were just potatoes! Better to save tummy room for those clams, right?
Well, after I'd eaten my share of clams, I still had a wee bit of room--perfect for half a potato. Since these were large white potatoes, I asked my husband to share with me. (Spoiler: This does not end well for Greg.) I cut the potato in half, poured some melted butter on it and gave it a generous sprinkle of salt. Oh my! This potato was so good! It's flesh was creamy while the skin remained almost "snappy" without being tough. When Greg decided to nab his share of the potato, I, um, decided I didn't feel like sharing!
Oh my! This potato was so good! It's flesh was creamy while the skin remained almost "snappy" without being tough. When Greg decided to nab his share of the potato, I, um, decided I didn't feel like sharing!
I wondered if I loved the potatoes so much because they steamed above clams and sausages or if perhaps steamed potatoes were just that good. After making several batches this summer, I can affirm: they are just that good.
Here's how to make them. (You'll need a steamer basket to make the potatoes. That's it!)
Wash potatoes. Although the first steamed potatoes I had were big white potatoes, I found I prefer steaming little potatoes, like these baby yukon golds I picked up at the farmers' market.
Add about one-inch of water to the bottom of a large pot.
Look for the smiling potato. (Okay, you don't have to do this. But, seriously, how cute is that potato?!?)
Place potatoes in the steamer basket, arranging the largest potatoes on the bottom of the basket and the tiny guys on top.
Cover. This is important! You can't steam with an open pot!
After about 30 minutes of cooking over medium heat, it's time to check the potatoes.
A knife should slide easily into the potatoes. (I always check a few just to be sure they are all ready.)
Melt two tablespoons of butter.
Pour over potatoes.
Add a generous sprinkling of kosher salt.
And herbs. You can use dried or fresh. I love (LOVE, LOVE, LOVE) Penzey's Parisien Bonnes Herbes. But use whatever you love. Dill, basil, and chives are especially nice.
And serve. The finished potatoes are creamy with an intense and lovely potato flavor.
Click Here for the Recipe!


text:
The Southern Maryland Agricultural Development Commission (SMADC) is offering grants to aid in the purchase of wine grape vines to local farms in Anne Arundel, Calvert, Charles, Prince George's and St. Mary's counties. These funds are being made available as a continuation of SMADC's Growing Grapes for Wine Program which was established to encourage the development of a competitive wine industry in Southern Maryland.
The grant program requires matching funds supplied by the farm owner for the purchase of grape vines of recommended grape varieties compatible with the region. The Grapes for Wine Program is offered together with the University of Maryland Extension Agency which will provide ongoing training and production expertise.
Next entry: Gov. O'Malley Proclaims Statewide Celebration for SMCM Sailing Team
Previous entry: Solar energy may come to Frederick County


text:
Angelina's animosity towards Jake (James Kyson) reaches a wild boiling point, and an angry Miss Lolly (Dot Jones) is forced to intervene. Undeterred by the memory of that mess, Angelina tries to give Joe a brief lesson in "redemption" but when she meets another weird date she proves that she herself knows nothing at all about redemption. She takes matters into her own hands and succeeds only in further infuriating Miss Lolly.
Thirty-something "widow" ANGELINA JONES, works as a fame-struck Hollywood gossip reporter hell bent on finding a father figure for her gay teenage son JOE. But Angelina is really Joe's father, who lost his mind after the mysterious disappearance of his wife.


text:
Genetic testing now allows personalized treatment. Find out more.
Search for a doctor by name or specialty. Search now.
Our patients have amazing accounts of hope and resilience. Read their stories.
Your gifts help us push the boundaries of cancer treatment. Ways to Give
Click on this figure to see the full screen.
Official Web Site of the University of California, San Diego.


text:
Tag: Artist work in Glamour
Artist Gets Artwork in Glamour
Have you ever dreamed of opening up a magazine and seeing something you created inside? Well that dream came true for one County resident. NewsSource 8 reporter Kelly O'Mara caught up with her and has the story. And you can find Littlefields artwork on her website at [email protected]


text:
Narrow your search
Polished graphics and sound design along with excellent puzzle-solving gameplay make this sequel well worth the price tag.
Beyond: Two Souls undoubtedly deserves credit for its great story and unique take on interactive software, even if it conflicts with what many believe constitutes a video game.
The sequel to its runaway OnePlus One hit improves on design and hardware to give top-end flagships a run for their money.
Pricing not available
Nostalgia overload isn't enough to make the many flaws of Epic Mickey 2: The Power of Two bearable.
Silent Circle has announced a sequel to its privacy-focused Blackphone, as well as a secure tablet.
Pricing not available
Although the YotaPhone 2's e-ink display is wonderfully novel, and can result in much better battery life, the phone needs to be considerably cheaper if it's going to be a sensible purchase over other Android phones.
A low-profile, weather-resistant full-HD action camera, the Drift Stealth 2 isn't without compromises, but makes up for them in size and usability.
The JBL Charge 2+ is a wireless Bluetooth speaker that sounds flat-out great -- and can double as a backup battery for your phone, too.
Wasteland 2 is genuine fun, packed with enough lore to make you want to keep exploring and adventuring, well past your bedtime.
The Polar Loop 2 can track daily activities and display notifications from your iPhone or Android device.


text: http://ubuntuforums.org – I'm rather new to Linux/Ubuntu and about to lose my religion figuring this one out. I am unable to connect to the internet with ethernet in 10.04 64bit. Everything was working fine this morning, but when I booted up this evening, no connection. In Network Connections, under the Wired tab, Auto eth0 reads Last Used: never. Also, Network Manager had created a new network, Wired Connection 1. I deleted Wired Connection 1. This morning the system attempted to run a check, which I cancelled. So I rebooted and allowed the check to run. Now, no connections appear in the Wired Network b (HowTos)


text:
Tunggadewa Mattangkilang
Samarinda, East Kalimantan. Officials from East Kalimantan and Malaysia have agreed to boost cross-border bartering of goods and produce to give residents of the remote border region easier access to items they would otherwise have difficulty sourcing.
Yaya Sabian Noor, head of the provincial investment coordinating body (BPPMD), said on Monday that the program was one of several agreed to at a meeting of the Malaysia-Indonesia Social and Economic Working Group in Bali earlier this month.
"Part of the agreement was that we would boost the barter trade between the two countries along the shared border between East Kalimantan and the Malaysian state of Sabah," he said.
Yaya, who also chairs the working group, said that under the program, residents of Nunukan district would be able to trade their agricultural produce and fish catches to their Malaysian neighbors in exchange for basic goods.
The agreement will also allow residents of Nunukan's Sebatik subdistrict to buy newly hatched chicks from Sabah, then raise them and sell them back to the Malaysians as broiler chickens.
"There were a lot of points that were agreed to in our meeting, but the most important is the emphasis on barter trade," Yadi said. "It is this that will allow our citizens on the border a market for their agricultural and maritime produce."
He said that in an effort to ensure the policy was not abused by people seeking to trade illicit or prohibited goods, Malaysian officials would build border posts.
"Under the barter trade program, citizens of both countries won't need a passport to travel across the border or be subject to customs and excise duties for the goods that they trade," he said.
"So we're allowing them to barter freely, while at the same time still keeping an eye on the trade."
The program is part of a wider effort to boost economic welfare in Indonesia's largely underdeveloped border region with Malaysia.
Residents of Krayan subdistrict in Nunukan have been particularly affected by the isolation, forced to depend heavily on trade and food from across the border to meet their needs.
The lone road from the subdistrict on the border with Malaysia to the district capital can only be traversed during the dry season. In the rainy season, it remains flooded for months.
The terrain is more forgiving on the 15-kilometer road to the border, meaning that 90 percent of the goods coming into Krayan are from Malaysia, residents say.


text:
BlackBerry Messenger v22.214.171.124 Coming Soon to the Public
Brace yourself, because BlackBerry Messenger version 22.214.171.124 will soon be arriving at the BlackBerry Test Center. This can only mean one thing: all of us will finally be able to get our hands on the redesigned BlackBerry Messenger (BBM) and get to try out the new features. BBM got an unexpected popularity boost during the FIFA 2010 World Cup from reports of how many fans used to the service to talk about the big plays and it looks like Research In Motion took notice and wants to propel BBM towards a future as a main feature on its BlackBerry line of smartphones.
So what is new in BBM 18.104.22.168? Take a look:
- Improved contacts menu
- Better picture posting features
- More intuitive menu options
- Any SMS contact can be set to have their texts show up in BBM
- Chat and group chat have been improved
- A whole new user interface
For those of you who regularly use BBM, these features will probably be a godsend. For the rest, it will be worth taking a look to see a cool, BlackBerry only feature that might actually make your messaging life easier.
No word yet as to whether or not the new BBM will keep pseudo-celebrities like Snooki from MTV's Jersey Shore from behaving like, well pseudo-celebrities, and doing foolish things like posting their BlackBerry PINs on Twitter. The poor woman's BlackBerry probably started smoking and melted after some of her 300,000+ Twitter followers tried using BBM to message her.
Will you be looking forward to getting your hands on the new BBM?
Angry that your friends have jumped the BlackBerry ship and climbed aboard the iPhone and Android OS bandwagon?
Should Snooki do a photo shoot for RIM to promote BBM?
Tell me what you are thinking. I really do want to know.BBM, BlackBerry Messenger v220.127.116.11, BlackBerry Messenger, twitter