
Distributed hypertext systems like the World Wide Web have become so pervasive and popular that the goal of a true hypermedia environment encompassing text, video, graphics, and sound seems within our grasp. There is a catch, however. The data models used for multimedia rely on stream-based, unstructured media representations bearing little or no relationship to the content and structure of the multimedia data themselves.
To understand this problem, we need to distinguish between a data model and its representation. A data model defines the nature of elemental components and the relationships between them, while its representation dictates how those elemental components are encoded.
The data model for hypertext consists of a body of text that is chunked into a set of nodes, i.e., semantic information units, which can be related to one another by means of hyperlinks. Resulting from this model is a three-tiered hypertext system architecture made up of a presentation layer (how the text is presented and browsed), a link layer (how the nodes are managed and related), and a node layer (the structured textual data themselves). This data model works quite well for text-only systems; however, the need for multimedia extensions forced designers to graft unstructured bit-stream models of continuous media onto the existing hypertext systems, resulting in compromised data access. For instance, audio and video data can only be accessed sequentially in a "playback" mode. Other multimedia deficiencies lie in the CCITT/ISO standard for visual multimedia (JPEG and MPEG-1), which are great for compression but poor from an information-management perspective.
The problem is how to create a true hypermedia environment where all forms of media can be managed as semantic or conceptual units using the same general data model. All media forms should have the same functionality as hypertext, namely, they should be navigable within and across media types, support random and associative access, contain editable representations, have a node and link structure independent of presentation issues, support Publisher Item Identifier S 0018-9219(97) 05615-6. arbitrary restructuring of data nodes, allow cutting and pasting between multimedia documents, allow individually labeled and accessible nodes, and be fully searchable and indexible.
To achieve this end, we might resort to knowledgebased coding schemes that approach objects in semantic fashion. Human cognition, however, works in such a way that semantic associations occur only after exposure to and interaction with a hitherto unknown object. Rather than resorting to burdensome, a priori artificial-intelligence techniques as a means of recognizing objects semantically, a hypermedia system more appropriately should perform a syntactical analysis first. To put it another way, we need a system that can identify objects and object characteristics without necessarily "knowing" what they are.
To understand this syntactical approach to hypermedia fully, we need to review salient issues in cognition, semiotics, and perceptual psychology. The goal of hypermedia is to design an associative, linked system that works in the same way as our minds do. Cognition research teaches us that mental representations of reality are formed and manipulated in a hierarchical network, starting with simple features as detected by our perceptual faculties, which are then transformed into meaningful conceptual units. To understand something, you need to know it either in terms of something else (recognition) or in terms of itself (syntactic understanding). The study of semiotics then shows us how information comes to be represented and communicated through signs. Semiotic theory has three dimensions:
• signs and their associated syntax;
• designatum (what the sign points to) and semantics;
• the interpretant (the user) and pragmatics. There are also metasigns, which are formed by grouping signs in the same way that signs are first formed from constituent elements (subsigns). Semiotics can show us much about how signs are formed from syntactic units, but how are these syntactic units formed into something semantically meaningful? The author answers this question by means of Gestalt theory, a theory of perception based 0018-9219/97$10.00 © 1997 IEEE on the idea that the whole has a meaning greater than the sum of its parts. The whole in fact influences the meaning of each part. In short, context influences how we perceive external stimuli.
Based on this detour through cognition, semiosis, and Gestalt theory, the author develops a new paradigm for hypermedia data models and representations. It identifies cognitively based semiotic articulations for multimedia data and the three dimensions of their defining, characteristic, and relational attributes. Syntactical data models are proposed for image, video, and audio supporting direct and random content-based access. Preliminary encoding schemes based on these data models provide concise representations and direct access to statistical and syntactic information that may be used to infer semantic information.
-Jim Esch

{'paper_id': '10943951', 'title': 'Hypermedia Data Modeling, Coding, And Semiotics', 's2_url': 'https://api.semanticscholar.org/CorpusID:10943951', 'source': 'S2ORC'}