A word of sorry ahead. This is going to get lengthy. If you will, hear me out though as I am clueless as to how to progress.
I run https://schreibnacht.de. We host a monthly event where a renowned author joins the community for a Q&A for an hour. During this hour the forum slows down to being unusable with loading a page taking long than the timeout, some requests being completely cancelled and returning a 503 and some request going lightning fast, returning in less than a second.
Which requests return fast is random and does not have a pattern. 95% of all requests load within tens of seconds or run timeout/503.
The nodebb instance is a single one running on 6 vCores with 16 GB RAM with Redis as DB on Ubuntu 16.04. The system load is at 0.4/0.32/0.27 very constantly. RAM is 8.9 GB / 16 used, CPU usage is minimal outside of Redis' bg-save actions where one Core is used and the others keep idling. 336 MB of 1GB in swap. We use Apache as a proxy between the clients and nodebb.
Stopping the forum via ./nodebb stop, stopping redis (shutdown save) and then restarting redis then the forum does not change the situation which is one of the weirdest parts about it.
During our events we have around 50-150 concurrent users, depending on how renowned the special guest is. (So let's face it, it SHOULDN'T have any problems at all).
I desperately need some help identifying bottlenecks, hints for analysis where I can start looking.
I will go so far as to ping @julian directly, maybe you have some insight althought I know you are very busy.
I could bet my pants and shoes that it's some misconfiguration or some error happing somewhere between browser, Apache, Nodebb and Redis but I just don't know my way around them enough to find the hairpin in the haystack.
Thank you for all your help and time guys.
Yeah, definitely could be resource exhaustion. Check ss -s during peak times to see TIMEWAIT?
We prefer scaling out horizontally for this very reason.
Oh my...! Thank you for all the replies. And especially julian taking the time! Amazing, I check all of that right now and ss -s next time we have an event.
So...this took a long time to work on. I finally have some odd numbers and findings that i'd like to share.
One node process with high cpu load and 3 others (further down) with 0% usage.
During redis bgsaves the timewait of ss jumps from 18/19 or 24k to 31/35 or even 40k, they remain around 24k while no bgsave is going.
So ulimit shouldn't be the problem. Right?
1	GET	/user/janerow	u"ConnectionError(ProtocolError('Connection aborted.', error(54, 'Connection reset by peer')),)"
1	GET	/topic/11705/testthema2	u"ConnectionError(ProtocolError('Connection aborted.', error(54, 'Connection reset by peer')),)"
Redis bgsave basically disables communication with the forum, but I have not figured out why.
The next obvious thing is to get the Apache cluster working with correct load balancing between the 4 processes.
Next I need to understand why redis bgsave is killing the forum. It might just be that both run on the same core and redis uses this core 100%, so nothing else can use it in the meantime? Any other suggestions very welcome.
May I ask why you're using Apache as a load balancer and reverse proxy? It is a terrible idea to use Apache HTTPd for something like this. Apache is a general web server. Nginx was created as a reverse proxy and load balancer and is much faster and much more efficient at it. If you're running into performance issue with Apache, I would suggest to change over to Nginx.
In regards to the redis process: What are your specification for HDD/SSD? Could it be that the drive is too slow for the writes by Redis?
@Dravere thx for the reply. Sadly it couldn't be more true.
The reason why I chose this setup (for now, certainly won't stick around as I hate it) is because I have a Plesk installed to manage the standard websites on the server. Plesk comes with Apache and (at least in my plesk version) can put an nginx in front as a reverse proxy to all its features. Sadly the version that it brings with itself is an nginx < 1.3.13 as would be required for nodebb to be usable as a reverse proxy.
The Server is running on full SSDs. I would need to ask them which ones specifically are being used if it's of relevance. I figured "as long as it's SSD, it should be fine" up until now. Let me know if I should inquire and I will.
Thanks again for your time. Cheers!
There is the traffic management settings and the option to set the threshold for the event loop lag. Under the value you can set there is a short help text that confuses me a bit: Lowering this value decreases wait times for page loads, but will also show the "excessive load" message to more users. Why does lowering of this value decrease wait times for page loads? I went through the code and I didn't feel like this value would change anything in regards to the page load time. What am I missing?