   IOCP+AcceptEx vs blocking accept()
  Hi all,I'm looking into IO Completion Ports for some test code I'm developing, and I have a quick question. I've been reading This Page (I know it's a bit dated), and it seems to imply that using AcceptEx is the correct way to accept connections when using IOCPs.However, I don't see any advantage to just using a blocking accept() in a thread without using IOCP at all.The only advantage I can see to using IOCPs for accepting connections is if the accept process takes a long time, so the server is spending a long time not waiting in an accept() call.So - am I missing something that makes IOCPs useful for accepting connections?Cheers,Steve   
  IOCP allows you to have outstanding operations. Simply have 50 accept calls pending on a socket. Then, if one of them takes long time, others will take over as long as there are workers that can process them.This is the core principle behind IOCP. There can be arbitrary number of pending operations waiting for network events (read/write/accept), and their handlers will be dispatched as needed.With synchronous model you always need to process one batch before starting with another. While simpler in implementation, misbehaving operations have more impact.   ;  I see. So does that mean that it's a good idea to create a "lot" of threads (I.e. 8 or 16) to process AcceptEx requests for when connection frequency spikes? In theory the only down side would be a few redundant (blocked) threads when usage is low.The code I'm writing is designed to be the network code for a large-ish multiplayer game or a web server - something designed to handle frequent connections and disconnections.Thanks for the reply,Steve   ;Quote:Original post by Evil SteveI see. So does that mean that it's a good idea to create a "lot" of threads (I.e. 8 or 16) to process AcceptEx requests for when connection frequency spikes? In theory the only down side would be a few redundant (blocked) threads when usage is low.IOCP has its own thread pool which takes care of that, you merely need to have enough outstanding requests.Application can use several threads if it can use them to do more work. But if each connection needs to lock the database, then having more than one application thread will not bring any benefit. IOCP will still be handling n accepts in different threads, but application would be unable to do more work.As an example:on_receive() {  query = parse_received_string();  database.lock();  result = database.query();  database.unlock();  client.send(result);}Running these handlers in multiple application threads would improve performance since despite database being shared, parsing and sending could be performed concurrently.But both receive and send would be handled concurrently by IOCP the way it deems best.Practical tests show that when there is no contention, the number of application threads that number of threads needed is same as number of cores +1. The +1 comes simply from various cases where a bit more work could be done. For two core, thread times on 2 core machine would be something like (97%, 97%, 4%), even if there are 10 threads in the pool. Overhead usually increases with more cores (consequently more threads).If blocking operations are involved, or expensive operations need to run inside locks, then it will be up to you to ensure they don't cause problems. Even if you put 100 threads in your pool, all of those could choose to block.Quote:The code I'm writing is designed to be the network code for a large-ish multiplayer game or a web server - something designed to handle frequent connections and disconnections.Web servers can usually be distributed quite effectively by keeping each request handler's logic as local as feasible. So your application would maintain a thread pool that would be handling connection callbacks from IOCP (threads calling GetQueuedCompletionStatus/Ex).Each logical connection (accept, multiple reads, logic, multiple writes, close) would then be handled sequentially. Once one of them completes, next step is performed in completion handler. This is where IOCP comes in - even though this is strictly sequential operation (and can be considered as single-threaded), it will multiplex hundreds of such connections doing the same in some predetermined number of threads. These will likely match the number of cores, since they will either be running at full load, or there will not be enough work to do. This is one part of equation.But since logic may involve long running operations, blocking calls or large operations on shared state those need to be addressed separately. If each of requests needs to modify some shared counter or container, IOCP cannot help with that, since it's up to you to lock or otherwise handle the scenario.   ;Quote:Original post by AntheusQuote:Original post by Evil SteveI see. So does that mean that it's a good idea to create a "lot" of threads (I.e. 8 or 16) to process AcceptEx requests for when connection frequency spikes? In theory the only down side would be a few redundant (blocked) threads when usage is low.IOCP has its own thread pool which takes care of that, you merely need to have enough outstanding requests.Do I not still have to have multiple threads each calling AcceptEx()?Quote:Original post by AntheusApplication can use several threads if it can use them to do more work. But if each connection needs to lock the database, then having more than one application thread will not bring any benefit. IOCP will still be handling n accepts in different threads, but application would be unable to do more work.[snip]Yeah, I'm aware of what happens when a worker thread enters a blocking operation, and I've been pouring over the MSDN for various IOCP-related information.Quote:Original post by AntheusQuote:The code I'm writing is designed to be the network code for a large-ish multiplayer game or a web server - something designed to handle frequent connections and disconnections.Web servers can usually be distributed quite effectively by keeping each request handler's logic as local as feasible. So your application would maintain a thread pool that would be handling connection callbacks from IOCP (threads calling GetQueuedCompletionStatus/Ex).I'd like to keep things as generic as possible here (I know that generic + high performance doesn't really go together) - for the multiplayer game aspect, I'd be using the accepting threads to accept the connection, create a new player information struct, and shove it in a queue of logging in or connected players, meaning the accepting threads wouldn't be doing a huge amount of activity, and in theory won't be blocking (Using lockless data structures and no heap access where possible).Cheers,Steve   ;  Advantage is very clear - thread context switching. Blocking accept will awake thread and cause performance degradation. Here is very useful ref about production server design:http://www.kegel.com/c10k.htmlM$'ve (with their iocp) just implemented one of the possible strategy...   ;  With IOCP, you issue requests from any thread, and they complete on one of the thread pool threads. Thus, you can issue a dozen AcceptEx requests all from the main thread, and then when an AcceptEx completes in the thread pool, you re-issue one AcceptEx as part of the completion handling.However, I don't see why you'd want more than one or perhaps two outstanding AcceptEx requests. If you're so overloaded on requests that calling AcceptEx takes a long time (looking up an IP address in a hash table of blocked addresses, perhaps?) then you wouldn't have the server resources to actually serve the users when they're accepted anyway.   ;Quote:Original post by hplus0603With IOCP, you issue requests from any thread, and they complete on one of the thread pool threads. Thus, you can issue a dozen AcceptEx requests all from the main thread, and then when an AcceptEx completes in the thread pool, you re-issue one AcceptEx as part of the completion handling.So is it usually preferable to have two IOCPs then, one for handling send/recv from clients and one for handling accept requests from new sockets?I was going to have two accepting worker threads, each one calling AcceptEx(), then GetQueuedCompletionStatus() and then processing the new connection in a loop. Should I instead make the AcceptEx() call from the main thread for whatever reason? Or does it really not matter either way?Quote:Original post by hplus0603However, I don't see why you'd want more than one or perhaps two outstanding AcceptEx requests. If you're so overloaded on requests that calling AcceptEx takes a long time (looking up an IP address in a hash table of blocked addresses, perhaps?) then you wouldn't have the server resources to actually serve the users when they're accepted anyway.That's a good point, yeah.Cheers,Steve   ;  You seem to misunderstand IOCP, have you used it before?Only ever have one completion port. It will handle everything, from WriteFile to WSASend and AcceptEx. The point is that the OS handles everything you post behind the scenes, in a way we are to assume optimal, and a completion notification will be returned and handled from GetQueuedCompletionStatus whenever an operation has completed or failed. Then there are some number of threads waiting on GetQueuedCompletionStatus, each thread exactly identical, and the OS will only ever allow exactly the same amount of threads to run at the same time as there are processors on the system. Theoretically this gives optimal performance, or at least that's the idea.I think the best about it is that once a software is designed properly to use it, it's a very beautiful and simple system, yet very powerful.   ;  The idea behind asynchronous connection-centric networking (such as HTTP) is that it creates a sort of cooperative multi-threading model. At conceptual level, the code looks like this (all function calls are non-blocking):on_start() {  accept(on_accept())}on_accept() {  start()   // start listening again  read(on_read)}on_read() {  if (read_enough) {    process(on_processed)  } else {    read(on_read)  }}on_processed() {  send(on_send)}on_send() {  if (done_sending)    close()  else send(on_send)}This is equivalent code to:accept()while (!read_enough) read()process()while (!done_sending) send()close()Since with this model each connection has only one outstanding request at any given time, it behaves as if everything were single-threaded, even without locking, even if there are many connections active at the same time.   