title: Storing data packets in a database
body: <p><strong>Problem description:</strong> 
  In my application, I have to present the contents of data packets with a certain format. An example: </p>
  
  <p><strong>An example</strong>
  Any packed binary data, for example: 4 byte header, 4 byte type (type codes having pre-defined meanings), then source address, destination address, and so on.</p>
  
  <p>Previously, I made home cooked implementations that stored the data in a binary file (fixed record length allowed fast lookup), but with time I'm realized I'm inventing some kind of a database. For example, I'm implementing my own efficient binary storage format for very large data files. I'm also implementing my own indexing to rapidly run searches on some fields. I think a real DB (even the simple SQLite) can make this stuff transparently simple.</p>
  
  <p><strong>Question #1:</strong> are DBs useful for storing such data, and how should it be done? Note that there are no 1-to-many, many-to-many mappings here and other advanced things, it's just a plain sequence of packets with a certain internal structure I want to display to the user and let him interact with (i.e. search by a certain field).</p>
  
  <p><strong>Question #2:</strong> Now suppose the user himself can specify the format of his packets, i.e. in a configuration file: the length of each field, its type, what its values mean (in case of an enumeration) and so on. How do I extend a DB-backed implementation for this? Should the user define DB schemas? Should the configuration file be auto-translated into this schemas? ORM? </p>
  
  <p><strong>Question #3:</strong> Even more advanced... Now suppose the data packages can be varying in length and contents. I.e., for type #2 packages, there are some field, for type #3, some other fields, and so on. But I'd still like my app to handle it, displaying everything nicely and also allowing users to specify the formats in config files. How is it done?</p>
  
  <p>Thanks in advance.</p>