   dynamic streaming of assets?
  I'm making a 2d game in C++ with SDL. I put all my images into giant 7z archives, and use PhysFS for all the file access, which works great, except for one problem: it usually takes 100-200ms per image to load, which produces unacceptable lag at certain points.I figured that if I tried to preload some of them in the background, it would reduce the lag, but I have been unable to find much information about how you'd actually do this, despite searching.I guess first off, I need a multithreading library. The main hard parts as I understand it is*Coordinate all of the loading threads, including priorities (ranging from might need-it-eventually to being-rendered-this-frame), and making sure that the same image isn't loaded twice, and images are available when needed.*Creating a system to decide when to preload something and in what order.*Make sure everything is done in a bug free manner without race conditions, deadlocks, etc.   
  I'm not an expert on the matter, and what you're gonna get from me is just some intuition on how to go about it.With a 2d game, I'd probably start from generating some sort of spatial partitioning out of quads the size of the screen (even very simple will do), so with which I can tell easily and pretty quickly what part of my level is going to be exposed. Then all I'd do is always keep the assets needed to render what is an area about 9 times the size of the viewport under the camera, and depending what your maximum camera velocity, or if the camera can be zoomed out quickly, you can bring that preloaded area bigger.In fact, if you just give your resource loader class some maximum amount of memory it can take up, and make the loading logic to always cap that memory no matter what, so with that you can get the pre-emptive loading you want. (The loader starts from the center point of your screen, and starts working outward of that point until it caps it's memory limit)Now regarding technical issues, I'd say what you want is:1. Reference counting class for storing your assets, so when you load your level - you create an instance of every possible asset that stores it's own file path and has the loading logic ready to be launched - this sorts out the problem of loading duplicates.2. Multi-threaded loading logic, that invokes the load-functions of your assets pre-emptively on a logic you want. 3. Generic memory management, if an asset has 0 references, it should be loaded out.4. Some common sense - Every asset should count how many times it's been drawn, the higher that value is, the more priority it can get when loaded in, and the lower priority it gets when it's taken out of memory. 5. (Optional, but maybe very unnecessary for a 2d game) Include some data to your level-files, about how many times a given asset is in a level, and use that to determine what should always be in memory, and what is worth streaming in and out.Finally:Only run your asset de-loading logic, when the cap-memory amount is overrun, then check your assets in this priority order:1. if Reference count == 0, Unload from memory2. Find asset with the smallest rendercount && biggest distance from camera && lowest occurrence value in this level and unload it3. Repeat until you go under your memory capThat's just me bantering tho, you can find some good multi-threading from Boost-libraries, and also it should have some good stuff to be used for those safe reference counting pointers.Also check out Enginuity for some ideas on making that memory management work for you :)   ;  My game is already divided into rooms, each with a separate camera system, so there's no need for a spatial partitioning system.I guess the priority would be more likeLoad everything that is being drawnLoad everything else that might be needed by the current roomLoad everything in a room that can be reached from the current room.Also, I'd estimate that the total size of all my images is under 100mb even after conversion to 32bit bitmaps, so I don't think I need to worry to much about running out of memory. I think it would simplify the system if just kept everything around after loading, instead of trying to unload stuff also.   ;  I'm using the exact strategy that you named (load what's needed, load what might be needed, load what can be reached from here) with a pool of 10 threads controlled by an IOCP and using overlapped reads (NOT using the no-buffer-flag!).Also, I always read more than needed, rounding the beginning down and the end up (to the next 128 kB). I've tried communicating success back to the main thread with APC, which works ok but is a bit awkward to incorporate in OOP. A simple single-producer-single-consumer queue turned out faster and easier to incorporate. The main thread simply polls once whether something has become available after swapping buffers. Thus, every texture (or whatever) is either present in a frame being drawn, or not. No other synchronisation needed.Apart from that, what Shinpou said... reference counting etc.The ideas behind this are:- loading is asynchronous most of the time on most systems- loading will always block on some old systems and on systems with compression enabled etc., but it will only block a worker thread- IOCP takes care of the thread sync crap, most threads will be blocked most of the time, at most num_cpu threads will be fighting for time slices- loading will sometimes be "instanteous", which also means "block"- even though it's "block" for about 0.2-0.4 ms (IMO a long time for little more than a memcpy!), the "instanteous" read from the buffer cache is much, much preferrable- seek times are by order of magnitude higher than read delays, or in other words: reading an additional 128 kB on the average costs pretty much exactly the same, but possibly saving 1-2 seeks with that is big win- modern computers have sheer abundant RAM, so one should use the page cache -- using unbuffered reads does have uses for some rare cases, but for most general purpose uses, it's silly- many systems still run XP 32 bit, so your address space is limited, having the OS cache the pages is often better than caching them in your own address space   ;  I just thought of another problem.Suppose I implement a threadpool, and create a thread to load image x. Now, another request comes in to load image x at top priority while the previous thread is still going.If I create another thread, they'll try to access the same file simultaneously, leading to horrible race conditions.So I need some way of determining whether there is an existing thread associated with loading image x, and have a way of joining it.   ;  Oh god.You shouldn't make a thread per-item. Make one loading thread.In the main thread: Lock a mutex, add new requests to a priority queue, unlock the mutex.In the loading thread:1) Lock the mutex, pop the top priority job, unlock the mutex.2) Check your resource pool for a resource slot.Check the pool to see if the resource is already loaded. If so signal that the resource is loaded.Check the pool to see if the resource is reserved and already loading. If so, continue to the load step.Check the pool for a empty slot of the correct size. If so, continue to the load step.Check if there are any items that are of lower priority that haven't been used recently that take a large enough slot. If so, evict the item and continue to the load step.Access to the resource pool needs to be mutex synchronized too so that the main thread doesn't use a resource that just got evicted.3) Check the resource slot for load progress. Issue a file read of some amount (like 32K) into the resource buffer, update the load progress.If that was the last read, run any code needed to process the file, but be careful. This step happens in a different thread. It can be a PITA to create a texture from a thread that isn't the same as the one that created the render context.  But you can do any other parsing you might want.Then signal the resource is loaded.If that wasn't the last read, lock the mutext, push the job back into the priority queue, unlock the mutex.So, as new jobs come in, they queue in priority.  The resource loaded manages the most urgent first, leaves less urgent ones partially complete (may even evict them if it runs out of room), and redundant requests are looked up in the resource pool before anything is done.  Since you only have one loading thread, you don't have to worry about synchronizing anything but the job queue and the resource pool.You can get some better performance by making the loader issue asynchronous read calls.  This complicates the loader even more though. Since now you have to hold onto resource pool items as long as they have outstanding asynchronous reads happening. On the other hand, you can issue much larger reads this way.   ;  I don't think PhysFS allows any of the things you mentioned.Here's my newest attempt. I haven't actually tested it yet because I'm still trying to figure out how to build the boost thread library.#pragma once#include <vector>#include "BOOST/shared_ptr.hpp"#include "SDL/SDL.h"#include "imagedatastruct.h"#include "utilityheaders/rectfuncts.h"#include "utilityheaders/debug.h"struct ImageLoader;class imageManager2{    protected:    imageManager2(); //Apparently, it's a good idea to make a Singleton constructor protected    public:    static imageManager2& Instance();    typedef Uint32 ImageID;    static ImageID NullID() {return 0xFFFFFFFF;}    static bool IsValid(ImageID h);    bool mIsValid(ImageID h) const;    //bool IsValid(ImageID h) {return h!=NullID() && h>=0 && h<loadedimages.size();}    ImageID GetImageID(const ImgDataStruct& data);    void draw_perSurfaceAlpha(ImageID id, SDL_Surface *screen, int x, int y, SDL_Rect* clip = NULL, int alpha = 255);    private:    //Manager has sole ownership over the pointers but I made them shared so the stl could copy them around    std::vector<boost::shared_ptr<ImageLoader> > loadedimages;    ImageID AddImage(const ImgDataStruct& data);    SDL_Surface* DereferenceID(ImageID h);};#pragma once#include "BOOST/array.hpp"#include "BOOST/thread/thread.hpp"#include "BOOST/thread/condition_variable.hpp"#include "BOOST/thread/future.hpp"//PhysFS is thread safe, except for individual file access//As long as each file is only accessed from one thread, everything is fine//WARNING! May leak memory if destructed while loading an imageclass FutureStruct //Not copyconstructable because myfuture isn't copyconstructible{    boost::unique_future<SDL_Surface*> myfuture;    bool ownsptr;    public:    FutureStruct(): ownsptr(false) {}    FutureStruct(const ImgDataStruct& data): ownsptr(true)    {   assert(!ownsptr);   boost::packaged_task<SDL_Surface*> newtask(boost::bind(&ImgDataStruct::Load, data));   myfuture = newtask.get_future();   boost::thread task(boost::move(newtask)); // launch task on a thread    }    SDL_Surface* GetValue()    {   if(!ownsptr)   {  return NULL;   }   else   {  SDL_Surface* returnedval = NULL;  myfuture.wait();  returnedval = myfuture.get();  ownsptr = false;  return returnedval;   }    }    void swap(FutureStruct& other)    {   myfuture.swap(other.myfuture);   bool temp = other.ownsptr;   other.ownsptr = ownsptr;   ownsptr = temp;    }};#include "imagemanager2.h"#include "asynch_loader.h"//#include "utilityheaders/timer.h"//WARNING! May leak memory if destructed while loading an imageclass ImageLoader{    ImgDataStruct mydata;    SDL_Surface* myptr;    FutureStruct myloader;    public:    ImageLoader(const ImgDataStruct& data): mydata(data), myptr(NULL), myloader(data) {}    ~ImageLoader()    {   SDL_FreeSurface(myptr); myptr = NULL;    }    bool DataMatch(const ImgDataStruct& data) const {return mydata.Equals(data);}    SDL_Surface* Surface()    {   if (!myptr)   {  myptr = myloader.GetValue(); //Blocks until load is finished, and then takes ownership from the loader   }   return myptr;    }};typedef Uint32 ImageID;imageManager2& imageManager2::Instance(){    static imageManager2 singleton;    return singleton;}imageManager2::imageManager2(): loadedimages() {}bool imageManager2::IsValid(ImageID h){    Uint32 tempsize = imageManager2::Instance().loadedimages.size();    return h != imageManager2::NullID()  && h >=0  && h < tempsize;}bool imageManager2::mIsValid(ImageID h) const{    return h != imageManager2::NullID()  && h >=0  && h < loadedimages.size();}ImageID imageManager2::GetImageID(const ImgDataStruct& data){    //Debug("GetImageID called");    for (Uint32 i=0;i<loadedimages.size();++i)    {   if (loadedimages.at(i)->DataMatch(data))   {  return i;   }    }    //The image has not already been loaded    return AddImage(data);}void imageManager2::draw_perSurfaceAlpha(ImageID id, SDL_Surface *screen, int x, int y, SDL_Rect* clip, int alpha){    //Debug("IM rendering");    assert(IsValid(id));    assert(alpha>=0);    if (alpha==0) {return;}    SDL_Rect offsets;    offsets.x=x;    offsets.y=y;    if (alpha<255) {SDL_SetAlpha( DereferenceID(id) , SDL_SRCALPHA, alpha);}    SDL_BlitSurface( DereferenceID(id) , clip, screen, &offsets);    if (alpha<255) {SDL_SetAlpha( DereferenceID(id) , SDL_SRCALPHA, SDL_ALPHA_OPAQUE);}}ImageID imageManager2::AddImage(const ImgDataStruct& data){    assert(loadedimages.size() < 0xFFFFFFFF); //This value is reserved for a null id    loadedimages.push_back( boost::shared_ptr<ImageLoader>( new ImageLoader(data) ) );    return loadedimages.size() - 1;}SDL_Surface* imageManager2::DereferenceID(ImageID h){    assert(IsValid(h));    return loadedimages.at(h)->Surface();};  Similar to what the other have mentioned, my method for solving it relies on constructing a thread/agent to handle the loading. I use the PPL from VS2010 along with agents. When a loadImage is called, it sends a asynchronous message to the agent, and return immediately with a Concurrency::event which will be flag upon task completion.If the image is being loaded, it will return the same Concurrency::event as the previous ongoing loadImage call.The Concurrency library that comes with VS2010, has quite a number of helper classes that allows easier multi-threading and cooperative blockingConcurrency runtime: http://msdn.microsoft.com/en-us/library/dd504870.aspxConcurrency async agents: http://msdn.microsoft.com/en-us/library/dd492627(v=VS.100).aspxOn the other hand, @Storyyeller, if you do not want to restrict a single thread just for loading, meaning you want to use available free threads for it, you may consider looking into:Concurrency::task_groupConcurrency::structured_task_groupeach time you wish to load a image, you wrap the loader in a task handle and send it to the task_group, using Concurrency:Scheduler you can control the number of threads that is dedicated to the task_grouphope this helps you   