October/November Development Objectives

Development, Research, and Testing Updates.

Re: October/November Development Objectives

Postby Eruptor » November 8th, 2015, 1:02 pm

Hi lei,
I think you are overthinking this :)

A timeout will trigger the thread even if the message queue is empty.
That's a waste of CPU-time.

Considering the opposite scenario, when you have many messages in the queue; why would you have the CPU spend time switching thread and then just wait, if there are more messages to process? That's also a waste of CPU-time.

The kind of micro-management you suggest to do here and there, like using secondary queues, burst limitations by timeout etc. will just complicate things.
In the end, the server must process (consume) data faster then the connected clients can produce data.

Keep it simple, you do have a OS, let the OS handle the thread load balance.
When you have your server running with live load, then you should analyze the data flow and identify issues like bottlenecks, starvation and subsystem responsiveness.
Eruptor
 
Posts: 15
Joined: January 24th, 2009, 3:07 pm

Re: October/November Development Objectives

Postby lei2 » November 8th, 2015, 5:08 pm

Eruptor wrote:Hi lei,
I think you are overthinking this :)

A timeout will trigger the thread even if the message queue is empty.
That's a waste of CPU-time.


ReHi Eru,

yes, thats what it does. If this is wasted time ... depends. I am not sure how often a notify might be missed, but a timeout always is a good idea. If long enough, the few cycles it needs are worth it.
In this case - I guess there is a misunderstanding with the not well named Message type.

Eruptor wrote:Considering the opposite scenario, when you have many messages in the queue; why would you have the CPU spend time switching thread and then just wait, if there are more messages to process? That's also a waste of CPU-time.


Its simple - we do not talk about daragrams or user messages here. An AO message is - in a separate thread - a call of the session->process method. This method checks all session queues if there is something to do. As this method has a housekeeping time, most of the AO message executions result into nothing to do.

Eruptor wrote:The kind of micro-management you suggest to do here and there, like using secondary queues, burst limitations by timeout etc. will just complicate things.
In the end, the server must process (consume) data faster then the connected clients can produce data.

Keep it simple, you do have a OS, let the OS handle the thread load balance.
When you have your server running with live load, then you should analyze the data flow and identify issues like bottlenecks, starvation and subsystem responsiveness.


You are right with simple things, I like them too. But things here are not as easy as they might seem.

Guess until there is a better idea, I'd say the secondary queue for idle sessions is good way to save cpu time within the existing cycling queue system. Its not perfect by any means, but it works as expected, resulting into 5% less cpu load instantly and still is self regulating. Anyway, currently its just a proof of concept and still needs improvement.

Greetz, lei
lei2
SWG:ANH Staff
 
Posts: 25
Joined: September 23rd, 2015, 1:11 pm
SWG Official Server: Farstar

Re: October/November Development Objectives

Postby Eruptor » November 9th, 2015, 2:48 am

Hi lei,

My comments were more from a point of "best practice", not a specific implementation.

I understand you may benefit from stuff like a secondary queue IF you still use the cyclic main loop design.
I got the impression that you were working on to get rid of that design, so my bad.

Speaking of implementations, I do have a similar design with a producer-designer pattern of "messages", but with the difference that all worker threads are using the same mutex-conditional variable. A "notify_one" signal will alert one of the waiting threads, if any. And "notify_all" will do just that.

This approach is one way to simplify scalability, you can easily adjust the number of threads used to service the messages either by configuration or during run time based on load, queue sizes etc. The part of your code that fires the notifications don't have to know about the number of threads, since they are using the same, (and only), condition variable for this type of events.

As of today, my "main scheduler" use static configuration for the number of threads used running the "session data", my network stack uses dynamic run time thread balancing.

For example, by increasing the number of threads in my "main scheduler" the server will handle more "player objects" simultaneously. Servers, especially zoneservers, with less load may use fewer threads. This helps a lot if you are using one or a limited number of PC's to run you entire server stack, the zoneservers with the most load will use more threads. It's a very basic form of load balancing (by resource utilization).

/Eru
Eruptor
 
Posts: 15
Joined: January 24th, 2009, 3:07 pm

Re: October/November Development Objectives

Postby lei2 » November 9th, 2015, 6:27 pm

Eruptor wrote:Hi lei,

My comments were more from a point of "best practice", not a specific implementation.

I understand you may benefit from stuff like a secondary queue IF you still use the cyclic main loop design.
I got the impression that you were working on to get rid of that design, so my bad.


Hi Eru,

yes, the session cycle will need to do for a bit still. And indeed network needs a redesign. Some small changes here and there wont do for that, it should be more a ... drastic ... change. What means its not done in a few. So for the moment its the small changes time.

Eruptor wrote:Speaking of implementations, I do have a similar design with a producer-designer pattern of "messages", but with the difference that all worker threads are using the same mutex-conditional variable. A "notify_one" signal will alert one of the waiting threads, if any. And "notify_all" will do just that.

This approach is one way to simplify scalability, you can easily adjust the number of threads used to service the messages either by configuration or during run time based on load, queue sizes etc. The part of your code that fires the notifications don't have to know about the number of threads, since they are using the same, (and only), condition variable for this type of events.

As of today, my "main scheduler" use static configuration for the number of threads used running the "session data", my network stack uses dynamic run time thread balancing.


That sounds interesting, but - with the notify on a common mutex - how do you determine the thread that got the session to wake it up?
Also - an issue with the current code - I find it being a bit too much attention to sessions. The zones and chat only run one of them, the conx has all client sessions and instead of swooping the packets from zone to clients it reassembles each packet to message and back to packet then.

Eruptor wrote:For example, by increasing the number of threads in my "main scheduler" the server will handle more "player objects" simultaneously. Servers, especially zoneservers, with less load may use fewer threads. This helps a lot if you are using one or a limited number of PC's to run you entire server stack, the zoneservers with the most load will use more threads. It's a very basic form of load balancing (by resource utilization).
/Eru


This sounds like a reasonable way to scale the threads. Once the idle cpu time waste is solved. Also it might be worth a thought if its not better to run more than one zone per zoneserver. Since all of them use a craving network engine it would ease things a lot, even more if space zones come in.

But thats still future music.

Greets, lei
lei2
SWG:ANH Staff
 
Posts: 25
Joined: September 23rd, 2015, 1:11 pm
SWG Official Server: Farstar

Re: October/November Development Objectives

Postby Obi-Two » November 10th, 2015, 2:19 am

I love this conversation, I don't understande any of it but I love that its happening :D

In an ideal world as I've mention before I'd have started this project from scratch, as both projects have there flaws, if we can attact half a doozen devs who knows I might surgest it again haha..

carry on thugh your doing great :)

oh Eru, have you ever though about coming back ;) I can sort you out a nice shade of green for your name haha
Obi-Two

Quality Assurance
Star Wars Galaxies:A new hope
there is another...

Server by : Hostwind
Project : SWGANHServices/SWGANHJava
Obi-Two
SWG:ANH Staff
 
Posts: 248
Joined: January 3rd, 2009, 9:37 am
Location: Manchester, England
SWG Official Server: Ahazi

Re: October/November Development Objectives

Postby lei2 » November 11th, 2015, 6:26 pm

Obi-Two wrote:I love this conversation, I don't understande any of it but I love that its happening :D


Sometimes indeed such things help. The threading is a good idea, I tried it yesterday and after some issues that popped on that way and a bit tuning it really helps to reduce load. What then again leads to other issues like a core class for messages that is thread unsafe. zomg. But well ... a good idea might work it off.

The critics with code are still valid, one main issue is missing documentation - the code speaks for itself. This is for a complex project a main fail. I guess working on this in parallel to code would help a lot since it with it you have not to riddle what might have been meant. If its documented you can evaluate the existing code to the documented intention. Not to be gotten wrong - Im not speaking about UML ... things like these tend to be more extensive than the coding itself - just a few lines in comment here and there already would do the job.



Greetz, lei
lei2
SWG:ANH Staff
 
Posts: 25
Joined: September 23rd, 2015, 1:11 pm
SWG Official Server: Farstar

Re: October/November Development Objectives

Postby lei2 » November 21st, 2015, 7:19 pm

November is nearly done so it might be time for something like a report.

When I started with the code, my intention was to give it a bit of a facelift to behave better in a linux environment and maybe to work on some game features. As it turned out in this process, every turned screw popped a couple of new issues. Threading was a good hint, it really looks good but soon reveals all the threading weaknesses in some of the core modules. Also boost was in way.

So in the last days I've been busy and what I have now is a boost free code. Needed a few utils but in the end wasnt as hard as I thought. Network core got some attention to message factory, heap, session and services, seems to work in threads by now. Not so the db core. The old async callback is not that handy, the new one might benefit from promise/future but that also would not make it easier to handle. I hoped I could keep it alas for the zone, but also here it seems to be better to use the sync replacement that login and chat already benefit from. Another core module that didnt survive is the event dispatcher. Maybe once there is time it might get updated to stl, or maybe a bit more simple task manager might replace it.

That is what was done in the last days, now I look ahead to rewrite zone to sync db. Not sure how much time I can spend on it through the next days, but its more a diligence job and hopefully straight forward.

So far for the moment,

Greetz, lei
lei2
SWG:ANH Staff
 
Posts: 25
Joined: September 23rd, 2015, 1:11 pm
SWG Official Server: Farstar

Re: October/November Development Objectives

Postby Obi-Two » November 22nd, 2015, 1:33 pm

This is a great update Lei2,

Code: Select all
obi cheers for Lei2


I've got my server sorted out, fingers crossed so will have to sort out a way to get your bin over to it and we can set soemthing up for the boys and girls to look at :D of sorts :S
Obi-Two

Quality Assurance
Star Wars Galaxies:A new hope
there is another...

Server by : Hostwind
Project : SWGANHServices/SWGANHJava
Obi-Two
SWG:ANH Staff
 
Posts: 248
Joined: January 3rd, 2009, 9:37 am
Location: Manchester, England
SWG Official Server: Ahazi

Previous

Return to Developer's Datapad

Who is online

Users browsing this forum: No registered users and 1 guest

cron