   MMO Network Design
  Im working on little MMO project and can use some advice on the networking part.First of all, im not really creating a game (yet), im more trying to make some base MMO design more or less to have a challange to see if i can do it and my interest in MMO design.so the idea behind the project:- Master Server (handle accounts/worlds)- Character Server (handling character info)- World Server (server running a small game world)People can login, choose a world server, create/select a character, travel around, chat, interact with npc, fight some mobs ...The way im building the networking is using TCP with select() method for multiple connections.now my FD_SETSIZE is set to 1024, so limiting me to 1024 connections at the same time. Now i have a few ideas in mind to solve to connection limit issue.1) Limited zone usersas the game world is split up in zones, it would be possible to have 1 fd_set / zone, so allowing 1024 players in same zone at same time. I dont know how acceptable this would be for real MMO games.2) Linked fd_setsa second solution would be having a linked list containing a 1024 array fd_setso when the first fd_set contain 1024 sockets and a new socket is added, a new node to the linked list is added with a new 1024 array fd_set.When a connection drops a socket is moved from the last fd_set to the first available fd_set till the last one is empty and can be cleanup up again.3) Single socket selectNot sure if this would be a good idea and how it would affect server performance, network performance. It would be possible to let every socket have its own fd_set and so called socket->select() method on each single socket.now in case the 2the method would be the best method, i wonder about the fd_set size. Will having small fd_set (for example 64) have better performance as largde fg_sets (for example 1024).note: all networking is done in separated thread, when a socket receive data, the data is parsed, validated and pushed in the the MessageInQue. The main loop thread check for incomming message trough that same Que (mutex locks) and handle the message. Sending data is done same way, if main loop thread must send data, it pushed a new message in the MessageOutQue, so the network thread can deque the message and send it.   
  you should look into epoll instead of select.http://blog.kovyrin.net/2006/04/13/epoll-asynchronous-network-programming   ;  While _Kami_ makes a good point, it should be noted that epoll is specific to Linux. That means that it *cannot* be used under windows (or any other Unix-like system, for that matter) without wrappers.   ;  Setting FD_SETSIZE to something bigger than 64 on Windows is not recommended, because the Windows kernel doesn't like blocking on more than 64 handles at the same time. If you find that you need more than 64 handles in the same wait operation, it is recommended to go with IOCP.If you want all that handled for you, I suggest using the boost::asio library, which provides asynchronous I/O for Linux and Windows (on sockets as well as files) using one cohesive API. Let someone else sweat the small stuff :-);  Currently looking into IOCP but its bit hard to learn with limited info and sample being so different from eachother its hard to track down the main steps.As for 3the party libraries such as boost, as i mentioned in the main topic, im doing this project to have a new challange, so prefer to write myself and learn from my mistakes.Now about the IOCP i looked into some samples but all of them work so different.I guess the way of implement depend a lot from application to application.Now in my case it might get bit complicated. First of all i have more as 1 listener.as for the master server i have the main master server port where clients connect, execute some message (login, select server, select/create char) and disconnect again.then i have another connection where multiple char/world servers can connect to (connection stay alive all time)After the little bit i learned about IOCP so far i could use some advice to see if im going the right directions.1) Should all sockets from both listeners be put in same completion port?2) Should i add my listeners to the completion port or run them in 2 extra threads with blocking accept?now i have 1 more question about: CreateIoCompletionPort() methodWhen u call it the first time, u can set the last parameter 'NumberOfConcurrentThreads' for the number of threadsi notice most samples set this value to 0 and create theyre own set of worker threads calling 'GetQueuedCompletionStatus()'now this confuses me a bit.for example, u make 2 worker threads calling 'GetQueuedCompletionStatus()'these are 2 visible threads. But if u set the thread parameter also to 2 when making the IOCP, does it mean there 2 more threads running on background (non visible threads)If so ... what those threads do?   ;Quote:Original post by FallonsnestCurrently looking into IOCP but its bit hard to learn with limited info and sample being so different from eachother its hard to track down the main steps.There is a very recent discussion about just that.;  Wrote a first IOCP test today but wondering how good my code is.the test is bit based on the IOCP sample of the windows SDK and some stuff i read about IOCP dont make sense with it.1 thing i read is that an overlapped structure should never be used again and a new overlapped struct should be made for each operation. As u can see in code bellow (same as the sdk iocp sample) the overlapped structure is being re-used.As u also notice in the code is that i create 2 overlapped structures, 1 for read, 1 for write, so both can happen same time. (im running dual core pc, so this code will use 2 IOCP threads and 4 worker threads)Now my send code is bit dirty, its bit based on my first design where every frame the code check if there is pending data to send. I know i should only start write actions once there is data and stop send events as soon all data is send.Question1: Can i re-use the overlapped structure like in code bellow or should i made a new one?Worker thread code ...Question2: Can i use a read/write IO operation at same time or can i only do send if a read operation completed?I know the code below is bad as there should be no send event (without passing any data) all time if there is no data to send//-------------------------------------------------------------------------------// Included header files//-------------------------------------------------------------------------------#include "StdAfx.h"#define DEFAULT_BUFFER_SIZE1024//-------------------------------------------------------------------------------// Name: // Desc: //-------------------------------------------------------------------------------struct IO_CONTEXT{WSAOVERLAPPEDOverlapped;BYTEOperation;charBuffer[DEFAULT_BUFFER_SIZE];WSABUFWSABuffer;};//-------------------------------------------------------------------------------// Name: // Desc: //-------------------------------------------------------------------------------DWORD WINAPI WorkerThread( LPVOID lpParam ){LPWSAOVERLAPPED lpOverlapped = NULL;SOCKET hSocket = INVALID_SOCKET;DWORD dwNumBytes = 0;DWORD dwFlags = 0;BOOL bResult = FALSE;// Get the completion port handleHANDLE hCompletionPort = (HANDLE)lpParam;// Run the thread loopwhile( true ){// Check the completion port for pending io operationsbResult = GetQueuedCompletionStatus( hCompletionPort, &dwNumBytes, (PULONG_PTR)&hSocket, (LPOVERLAPPED*)&lpOverlapped, INFINITE );// Check if we have a valid socketif( hSocket == 0 )break;// Get the contextIO_CONTEXT* pContext = CONTAINING_RECORD( lpOverlapped, IO_CONTEXT, Overlapped );// Check what kinda io operation we dealing withif( pContext->Operation == 0 ){// Check for dropped connetionsif( 0 == dwNumBytes ){OutputDebugString( L"Connection dropped ...\n" );closesocket( hSocket );}else{//TODO: Process the bufferOutputDebugStringA( pContext->WSABuffer.buf );// Begin receiving the next bufferZeroMemory( pContext->WSABuffer.buf, pContext->WSABuffer.len );WSARecv( hSocket, &pContext->WSABuffer, 1, NULL, &dwFlags, &pContext->Overlapped, NULL );}}else if( pContext->Operation == 1 ){//TODO: Check for outgoing data and put in the WSA buffer// Begin sending the next bufferZeroMemory( pContext->WSABuffer.buf, pContext->WSABuffer.len );WSASend( hSocket, &pContext->WSABuffer, 1, NULL, dwFlags, &pContext->Overlapped, NULL );}}// Exit the threadreturn 0;}//-------------------------------------------------------------------------------// Name: // Desc: //-------------------------------------------------------------------------------int wmain( int argc, WCHAR* argv[] ){#if defined(DEBUG) | defined(_DEBUG)// Enable run-time memory check for debug builds.    _CrtSetDbgFlag( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF );#endif// Initialize WinSockWSAData wsaData;WSAStartup( MAKEWORD( 2, 2 ), &wsaData );//--> Begin IOCP code// Get the system informationSYSTEM_INFO systemInfo;GetSystemInfo(&systemInfo);// Calculate the number of threadsDWORD dwThreadCount = systemInfo.dwNumberOfProcessors * 2;// Create the completion portHANDLE hCompletionPort = CreateIoCompletionPort( INVALID_HANDLE_VALUE, NULL, 0, systemInfo.dwNumberOfProcessors );if( NULL == hCompletionPort )goto App_Cleanup;// Create the worker threadsHANDLE* hThread = new HANDLE[dwThreadCount];for( UINT32 iThread = 0; iThread < (UINT32)dwThreadCount; iThread++ )hThread[iThread] = CreateThread( NULL, 0, WorkerThread, hCompletionPort, 0, NULL );//--> End IOCP code//--> Begin socket code// Create the listen socketSOCKET hListener = WSASocket( AF_INET, SOCK_STREAM, IPPROTO_IP, NULL, 0, WSA_FLAG_OVERLAPPED );if( INVALID_SOCKET == hListener )goto App_Cleanup;// Bind the socketsockaddr_in service;service.sin_family = AF_INET;service.sin_addr.s_addr = inet_addr( "127.0.0.1" );service.sin_port = htons( 3000 );if( SOCKET_ERROR == bind( hListener, (SOCKADDR*)&service, sizeof( service ) ) )goto App_Cleanup;// Start listeningif( SOCKET_ERROR == listen( hListener, 5 ) )goto App_Cleanup;//--> End socket code// Run the loopint i = 0;while( true ){// Accept the next socketSOCKET hAccept = accept( hListener, NULL, NULL );OutputDebugString( L"Connection accepted ...\n" );// Add the new socket to the completion portCreateIoCompletionPort( (HANDLE)hAccept, hCompletionPort, (ULONG_PTR)hAccept, 0 );// Begin receiving dataDWORD dwFlags = 0;IO_CONTEXT* pContextRecv = (IO_CONTEXT*)HeapAlloc( GetProcessHeap(), HEAP_ZERO_MEMORY, sizeof( IO_CONTEXT ) );pContextRecv->Overlapped.Internal = 0;pContextRecv->Overlapped.InternalHigh = 0;pContextRecv->Overlapped.Offset = 0;pContextRecv->Overlapped.OffsetHigh = 0;pContextRecv->Overlapped.hEvent = NULL;pContextRecv->Operation = 0;pContextRecv->WSABuffer.buf = pContextRecv->Buffer;pContextRecv->WSABuffer.len = sizeof( pContextRecv->Buffer );ZeroMemory( pContextRecv->WSABuffer.buf, pContextRecv->WSABuffer.len );WSARecv( hAccept, &pContextRecv->WSABuffer, 1, NULL, &dwFlags, &pContextRecv->Overlapped, NULL );// Begin sending datadwFlags = 0;IO_CONTEXT* pContextSend = (IO_CONTEXT*)HeapAlloc( GetProcessHeap(), HEAP_ZERO_MEMORY, sizeof( IO_CONTEXT ) );pContextSend->Overlapped.Internal = 0;pContextSend->Overlapped.InternalHigh = 0;pContextSend->Overlapped.Offset = 0;pContextSend->Overlapped.OffsetHigh = 0;pContextSend->Overlapped.hEvent = NULL;pContextSend->Operation = 1;pContextSend->WSABuffer.buf = pContextSend->Buffer;pContextSend->WSABuffer.len = sizeof( pContextSend->Buffer );ZeroMemory( pContextSend->WSABuffer.buf, pContextSend->WSABuffer.len );WSASend( hAccept, &pContextSend->WSABuffer, 1, NULL, dwFlags, &pContextSend->Overlapped, NULL );}App_Cleanup:// Tell the worker threads to exitfor( UINT32 iThread = 0; iThread < (UINT32)dwThreadCount; iThread++ )PostQueuedCompletionStatus( hCompletionPort, 0, 0, 0 );// Wait for the worker threads to exitWaitForMultipleObjects( dwThreadCount, hThread, TRUE, INFINITE );delete[] hThread;// Cleanup the listenerif( INVALID_SOCKET != hListener )closesocket( hListener );// Cleanup the completion portif( NULL != hCompletionPort )CloseHandle( hCompletionPort );// Cleanup WinSockWSACleanup();// Exit the applicationreturn 0;} [Edited by - Fallonsnest on May 22, 2009 3:11:56 PM];  There is quite a lot of topics regarding IOCP, some recent as well, such as these or this. They should have some answers to various implementation details.   