   Optimizing file loads
  Been optimizing under the guidance of a profiler lately to great effect. DevPartner is working better than I thought it would, and it is fun to get such massive results (we're talking about the GUI being frozen for 10 seconds or so, now its ultra responsive). I am using TinyXML to output to an XML file but when serializing/deserializing the list the GUI just hangs for a second (timed it) or so, and I have a decent computer. The XML file is about 40k or so.Questions:1. Should I try switching ReadFile/WriteFile for fread/fwrite inside TinyXML?2. Would memory-mapped files be of use here?Edit: an algorithmic improvement would be to only write out things that have changed since the last run (e.g. a dirty flag). But that doesn't help reading.Edit 2: increasing the size of the buffer used by fread didn't help a thing, it appears to be the actual interpretation of the data.[edited by - antareus on February 3, 2004 12:44:00 AM][edited by - antareus on February 3, 2004 12:54:17 AM]
  I can think of two ways too keep your GUI responsive while big loads are happening... Though, neither way I''m talking about has to do with optimizing (well, a little).You can either use a callback mechanism... From your load function you use a pointer to a function that can check on GUI events, update render system, and do some quick things, then return back to load function.Option two:Use threads...just my 2 cents ;  Even if I use threads or callbacks my program will still seem slow because the GUI is useless without the data (a contact list).Then again Trillian takes like 10 seconds to load, now I see why.  I think I need to switch formats. ;  threading this sort of thing just might help youin one of my projects I have a HUGE loader/saver function that watches certain semaphores and dynamically loads whatever it expects will be needed soon, and saves information that it predicts will not be used for a while.It is surprisingly efficient. But you gotta have the right semaphores.   George D. FiliotisI am a signature virus. Please add me to your signature so that I may multiply.;quote:Original post by antareus  1. Should I try switching ReadFile/WriteFile for fread/fwrite inside TinyXML?do this. i implemented my base file class of mine with the windows file i/o (so i could switch bewteen file i/o functions for kicks) and i got a massive improvement (party due to cache i think, but the initial load was still much faster). it might not be so good for small files (i''m talking 20-30+ meg archive files), but there might still be an improvement. ;  TinyXML is a DOM-like parser, which means it has to allocate a bit of memory for each XML node it comes across and store all this in a big tree. Needless to say, this is very convenient but may also be pretty slow. As much as I love TinyXML and have contributed several bug fixes to it, you may consider moving to a SAX (eg. callback-based) parser if speed is what you need. However, that''s more for reading than writing; I''m not sure what the write bottleneck would be.   [ MSVC Fixes | STL Docs | SDL | Game AI | Sockets | C++ Faq Lite | Boost  Asking Questions | Organising code files | My stuff | Tiny XML | STLPort] ;  The Windows routines allow you to specify that you''ll be reading sequentially, and using them directly would cut out the stdio buffer; they also make possible asynchronous reads, which are necessary for maximum throughput. None of those are important for smaller files; Kylotan is correct about the bottleneck. We use XML (Xerces) for our GUI; I was stepping through the code recently with BoundsChecker, and it reports TONS of allocations, even with an insignificant test file. *sigh*Might not a simple, ''dumb'' text file be better for your needs? Or you you care for buzzwords and the mythical advantages of XML more than reasonable load times? ;  Hi Jan:I cannot ship my product without it using at least three TLA''s (three letter acronyms). So I will have to look for another technology to take its place.I''ll try the text files and doing algorithmic-type changes such as only storing data if necessary. ;  hehe 