   File formats for games.
  Hello there! How are you people?I was thinking, there are a lot of different formats for maps/areas and players for example md5, md3, md2, smd, mdl and etc, and maps like bsp is the only one I know. Also some engines like unreal engine use a self made extensions like utx (for textures) and more.Every single extensions designed specially for the specific game like bsp of quake will have the quake "magic number", quake lump and will be totally different (or maybe not, depends on map i think) from half-life's "magic number" and number of lumps.The perfect thing to do for a beginner game programmer IMHO, is to take some of these extensions learn their format and render them (as i did), but again IMHO its not suitable for a commercial (or not) game that you are going to develop.So I want to ask what do you prefer for a game that you develop and think that it will have potential to be published and to be known by some amount of people (don't must to be best seller or etc) what file format they should use? Should they use already developed formats? Or develop their own? Are there some tutorials or explanation how to design data files like map storage? player storage and animation? I mean IMHO a project that have potential to be published should develop their own format (unless they use some ready engine, again like unreal). What do you think?Well I hate when I cant describe my question in small amount of lines -.-Hope you got me, have a good weekend :)   
  This isn't a direct answer to your question, but it is hopefully something to think about. From where I am standing, there are 3 requirements for each file format that I use:It must be easily created.I don't want to spend all my time writing and debugging exporters and conversion tools, so it is best if it is an existing (and widely used) format.It must be easily loaded.I don't want to spend all my time writing loaders either, so it must either be a simple, well documented format, or there must be a library that already loads this for me.It must store data efficiently.Disk space, and even more importantly, download size, is at a premium. I could store all my 3d models in xml (like Collada), but the files will be at least an order of magnitude larger than an efficient binary format.However, there is another aspect to this. For instance, the Collada format passes #1 and #2 with flying colours, but fails miserably at #3 - which indicates that I should use Collada for my asset pipeline, but I will need to convert it to a binary format before shipping a product.   ;swiftcoder Thanks a lot for your comment :)I would like to hear other opinions as well.   ;  I actually started my app to create and load my custom data file format. Here's what I shot for when writing my format specification.Make it flexibleI made a stand alone tool to create my game data files. It has a class for each file type that handles loading the original input files (currently only .ac, .bmp and .wav) and also saving the data in the new format. If I want to add another file format, I just add another class.Make it fastStore the data in a format that is closest to how it is stored in memory. Strive to be able to stream the data directly from the file to the object in memory that you're going to use. The less processing after loading the data, the better. In my opinion, drive space is cheap and no one likes long load times.I also preprocess the data and then make a header that has a list of the object ids, types, and exactly where the object starts in the file so I can jump to any object in the file without sifting through everything before it. And like swiftcoder said, make a nice easy to read and easy to update specification for the format.I'm no expert but those are my opinions. [smile]   ;  Ok I see.You two told that you made your own formats. May I ask how? I mean I see one possibility, to model the character/level in 3ds max for example and then write your own exporter that will export it to your format.If its possible is there any tutorials how to create custom formats? Or things that will help me? I mean I don't need an algorithm like, "take this code, compile, run and you done", just things that will explain me about data storage maybe some compression algorithms, or ways of data organization connected to storage of 3D models, I don't need compression algorithms to compress music for example, not yet :). Thanks a lot.   ;  heyI think designing a file format comes naturally when you understand the data and routines used to processs/use the data.So firstly you should learn and understand how to render a model with all the features you would like. Well maybe you can start off with something simple, and gradually make things more advanced.So for starters, maybe learn to render just a model without textures or material properties, and then design a format to suite these requirements. Well for testing your routines and such you may want to make a model loader using something basic/simple, such as the WaveFront .OBJ format.If your arn't sure about this, then you may want to design your routines based on an existing format. Hmm, or even just looking at other file format descriptions/specifications may help.Anyhows I advise you to lookup the WaveFront .OBJ file format description/specifications (just google such a term), and learn from them :). It's a very basic ASCII format, so you can just open the file up in a text editor and see the data :).Otherwise it might even be better to start off by making your own texture loader and/or format, and learn to draw this on a quad or whatever.Anyhows later on you may want to look into more advanced formats, and get to understand the data and such involved. The DirectX .X format is quite nice, as it supports alot of things (such as skeletal animation), plus it's also an ASCII format. Hmm, if you like I could probably dig up some docs for ya, well I've posted links to such docs in other threads on these forums :).BTW, I think there is a binary version of the DirectX .X format, but I don't think it's very well supported.Quote:You two told that you made your own formats. May I ask how? I mean I see one possibility, to model the character/level in 3ds max for example and then write your own exporter that will export it to your format.You missed another possibility. That is to write a converter from one format (which should be quite well supported, and support whatever things you need) to your own format ;).The advantage of this is that you can get to your model format or whatever from many different applications. Well I think most 3d software now days can import/export to many different formats anyhows, so this may not be such a big issue.Also, this is actually what I originally done.Well I started doing such things when I designed my own image format, which had a little compression scheme :). I think I converted to it from the Bitmap .BMP format.Now days in the world of 3d, I use my own texture, model, skeleton, and animation formats. I convert to my texture format from the Truevision .TGA format, and I export (from blender) to my model/skeleton/animation formats. I originally converted to my model/skeleton/animation formats from the DirectX .X format.Well actually, before doing any of these I had made loaders for the WaveFront .OBJ, and Autodesk .3DS formats. Anyhows I think that's about it. Hmm, somewhere along the line I also figured out the THPS4/THUG/THUG2 (Tony Hawks Pro Skater) image/texture/model formats, and made some converters and such.Anyhows I think the first 3d format I ever looked into was the WaveFront .OBJ format :), which I would recommend you to firstly look into also.cya   ;Quote:Original post by PsychoPumpkinMake it fastStore the data in a format that is closest to how it is stored in memory. Strive to be able to stream the data directly from the file to the object in memory that you're going to use. The less processing after loading the data, the better. In my opinion, drive space is cheap and no one likes long load times.Agreed that you should make it fast, but depending on what you mean by 'fast', you should take different routes.  Real-time loading of resources would likely benefit from the method you stated, being something you could just dump straight into use.  Typical 'loading screen' type of loading though could often times benefit from some compression, since disk reads are time consuming and your processor would be otherwise useless.  Performing real-time loading means you're actively using the processor for more pressing matters, and the heavy processing load associated with unraveling a tight compression would be problematic.  Also, you can just throw the disk read into another thread and let the huge read block for who cares how many frames.If you have files with a very large read latency [large files, or files that will be transfered over a network on demand], then you might want to try to make them useful earlier in the loading process if you're shooting for real-time loading.  For example, storing meshes as a number of levels of detail, so loading can be started when the object is far away [and can be rendered in the lower detail], and shift to the higher detail as loading completes, which will be needed as the object moves closer.  Seems silly, but makes real time loading of complex data much more tolerable in certain instances where you can't easily predict what resources will be required.Ask yourself : When do you intend to load your resources.How soon will you need a resource to be used from when you identify it will be needed.What information will you want to be included that isn't 'typically' included in a given type of file [typically only really important with animating files, since everything else will likely be either fully standard or fully proprietary].Also, as far as exporting your resources, it is often times easier to just write a quick converter from an easy to use but insufficient format into your own format than it is to write an actual export script... just something to consider.;  I'm not sure which is the most hazardous and painstaking pastime: Making video games, making stable nitroglycerine, and making homemade lasers. One of the main problems is that you either have to take off the shelf components and make the rest of your creation fit them, or you have to make your own, whereby they will be bulkier, harder to transport from A to B safely, and not work as efficiently. For images, I use .tga format. Not perfect, took a while to find a decent library to load it, but well supported. I can photoshop my textures, they support RGBA colour, and I now have libraries to load them. For static geometry I used the proprietry file format from the application AC3D, which has a public domain file format, and is easier to read. I then convert it into octrees in memory at load time.If my game was going commercial I would convert the code to use my own file format for textures and models. But its not, so I For GUI stuff I considered using CEGUI, or Guichan but they both would have increased my applications dependencies, and were bloated for my purposes. (worms type control panel, and the inventory for a sandbox game).  So, I'm now spending a month writing my own. (nearly finished.) Because I wrote it to fit the application, it fits perfectly. I'm using STL containers, and my development time has actually been far shorter than I was warned about.So here is the point of decision: If you take an off the shelf component will you reduce development time/cost/hair loss or will you be better off making your own component that fits the bill perfectly. The ability to make this decision applies to most disciplines, not only software development.   ;  You could always do what was done in a few of the commercial games i've edited use, which is explained (hopefully with examples below);eg. .tga, .dds, .jpg, .jpeg just simply with a different file extension at the end, to edit them, its simply, change them back to the respective type then save it and change the extension again. Or even more simple, just use those file extensions, makes it really easy. (i use jpeg's and tga's usually).Maps are just matrices of values so a text document like a .txt should be sufficient.   ;yosh64Thanks for a detailed explanation.As I already told I loaded some formats so I do have experience with various formats.DrigovasTotally agree.speciesUnknownWell many people tell that better to take a ready product and to use it, and its true, the cost of development decreases, also the time of development, time of debugging (cause well you will use a ready loader of gif/tga/jpeg or whatever that probably has been checked and debugged), and well it also help you to focus on your goal (a game? a rocket control system? or whatever) instead of wasting time on rewriting algorithms that already exists in x,y,z.dmandnDon't see point in it. Its a possibility but I have no idea why to do it.Thanks a lot to all who posted :)Other opinions welcome as well :)Edit:I thought about it, why big companies develop their own format for every game? I mean its not that they want to "have experience in compression algorithms or data structures", like me for example, why not to use a ready formats?[Edited by - s.kwee on February 4, 2008 7:13:47 AM]