October/November Development Objectives

Development, Research, and Testing Updates.

October/November Development Objectives

Postby Obi-Two » October 17th, 2015, 3:18 am

Our objectives for September where few, but we have good new, we achoved one :D with massive and speacial thanks to our member lei2, the project now build on linuix devices.

I was hopeing to have a demo test centre up for this weekend however the server maxes out the box while doing nothing... so adding traffic to that might be a little unwise, so for the remainder of October and November the main project objectives are to make the code a little more stable and linux friendly, remove out dated deps while updating the code (though this is and will be an ongoing objective) and generally tidy things up.

I'm going to open up new forums for this project so once we a dedicated place to discuss and document whats we are doing.

We still need you, your our only hope......

Obi.

Edit: Yes I missed out Septembers development objectives post, if you read the August thread you'll see why :D
Obi-Two

Quality Assurance
Star Wars Galaxies:A new hope
there is another...

Server by : Hostwind
Project : SWGANHServices/SWGANHJava
Obi-Two
SWG:ANH Staff
 
Posts: 248
Joined: January 3rd, 2009, 9:37 am
Location: Manchester, England
SWG Official Server: Ahazi

Re: October/November Development Objectives

Postby lei2 » October 28th, 2015, 6:43 am

Currently I do still work on networking, meanwhile I got quite familiar with it. But it will take some time since there is a bit more to do than I expected.

In the meantime, a tiny little update might help to reduce the load issue. Because Halloween is near it might be called the sleeping Dragon Utility. Usually a Dragon sleeps on his heap of gold, opening an eye from time to time to see if something comes along that needs his special attention. If something comes along in his idle cycle, it will stumble over his well laid gold. So the noise wakes him up and now something gets his attention.

Code: Select all
/*
---------------------------------------------------------------------------------------
This source file is part of SWG:ANH (Star Wars Galaxies - A New Hope - Server Emulator)
Copyright (c) 2015 Unofficial Hope Edition, lei
Use of this source code is governed by the GPL v3 license, see COPYING in project root.
---------------------------------------------------------------------------------------
Pipe Utility

For polling queues the wait method can be used as replacement for the sleep. If there
is work to do, a send after something pushes into the queue will then wake the wait.

To reflect load situations, wait returns the number of items in the pipe. So if wait
returns higher values than 1, the pushers put more into the queue as the worker can pop
in its cycle. The worker might use the return then to bypass the wait for the next
cycle.

Note: Due to the nature of pipes and select, each wait needs its own pipe.

Note: If methods are realized in cpp, this class would share them over program borders
      which may lead to issues. Instead of changing the cmake ruleset, they are simply
      implemented here.
---------------------------------------------------------------------------------------
*/

#ifndef ANH_PIPE_H
#define ANH_PIPE_H

#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <sys/select.h>
#include <signal.h>
#include <errno.h>
#include <fcntl.h>
#include <pthread.h>
#include "logger.h"


namespace utils {

        using namespace std;

        class Pipe{
                public:
                        Pipe(){
                                if(pipe2(pipefd,O_NONBLOCK)<0){
                                        LOG(FATAL) << "Pipe::Pipe: failed to create pipe. Exit.";
                                               _ready = false;
                                        } else _ready = true;
                                }
                        ~Pipe(){
                                if(_ready) { close(pipefd[0]); close(pipefd[1]); }
                                _ready = false;
                                }

                int     wait(uint32 timeout_millis){
                                if(!_ready) return -1;

                                FD_ZERO(&fdset);
                                FD_SET(pipefd[0], &fdset);

                                timeout.tv_sec  = 0;
                                timeout.tv_nsec = timeout_millis * 1000;

                                sigemptyset(&sigmask);

                                int done=pselect(pipefd[0]+1, &fdset, NULL,NULL,&timeout,&sigmask);
                                if(done < 0){
                                        switch(errno){
                                                case  EINTR : LOG(NOTICE) << "Pipe::wait: pselect received signal.";
                                                        break;
                                                case  EBADF : LOG(FATAL) << "Pipe::wait: file descriptor invalid or not open,exit!";
                                                        break;
                                                case  EINVAL : LOG(FATAL) << "Pipe::wait: Parameter Error (fd negative or invalid timeout),exit.";
                                                        break;
                                                case  ENOMEM : LOG(FATAL) << "Pipe::wait: unable to allocate memory,exit.";
                                                        break;
                                                }
                                        //somebody else would assert here
                                        }

                                if (FD_ISSET(pipefd[0], &fdset)!=0) {
                                        char c;
                                        int ctr = 0;
                                        memset(buffer,0,sizeof(buffer));

                                        while(read(pipefd[0],&c,1)>0) {
                                                buffer[ctr++] = c;
                                                if(ctr>63) break;
                                                }
                                                buffer[ctr] = 0;
                                        return(ctr);
                                        }

                        return 0;
                        }

                void    send(char something){ if(_ready) write(pipefd[1],&something,1); }



                private:
                  Pipe(const Pipe&);
                  Pipe& operator=(const Pipe&);

                int             pipefd[2];
                fd_set          fdset;
                sigset_t        sigmask,oldmask;
                struct timespec timeout;
                char            buffer[64];
                bool            _ready;

        };

} // namespace

#endif







Put the above code as pipe.h into src/Utils directory. Edit src/NetworkManager/SocketWriteThread.h then:
- Add #include "Utils/pipe.h"
- above Class SocketWriteThread add using namespace utils;
- In bottom, under SessionQueueLight mAsyncSessionQueue add Pipe* sessionPipe;
Now edit src/NetworkManager/SocketWriteThread.cpp. In Constructor - line 50 or so - add
sessionPipe = new Pipe(); then near line 174 at bottom of SocketWriteThread::run() there is a directive boost::this_thread::sleep(boost::posix_time::milliseconds(1));
Replace it with sessionPipe->wait(2000);
In line 311, SocketWriteThread::NewSession(..) under mSessionQueue.push(session); add
sessionPipe->send('X'); and another in line 341 under mSessionQueue.push(session);

So far for the SocketWrite. In SocketReadThread.cpp line 194 the tv_usec can be set a bit higher too, 500 should work. In src/Utils/ActiveObject.cpp add and usleep(2000) between line 66/67. Same
for other spots where sleep is used:
ChatServer/ChatServer.cpp: boost::this_thread::sleep(boost::posix_time::milliseconds(1));
ConnectionServer/ConnectionServer.cpp: boost::this_thread::sleep(boost::posix_time::milliseconds(1));
LoginServer/LoginServer.cpp: boost::this_thread::sleep(boost::posix_time::milliseconds(10));
NetworkManager/Service.cpp: boost::this_thread::sleep(boost::posix_time::milliseconds(10));
NetworkManager/SocketReadThread.cpp: boost::this_thread::sleep(boost::posix_time::microseconds(10)); (<--- this doesnt make any sense here)
PingServer/PingServer.cpp: boost::this_thread::sleep(boost::posix_time::milliseconds(1));
Utils/Timer.cpp: boost::this_thread::sleep(boost::posix_time::milliseconds(10));
ZoneServer/ZoneServer.cpp: boost::this_thread::sleep(boost::posix_time::milliseconds(1));

These can be set up decently, minimum is 20ms.

Setting up the sleep times its just a quick hack for the moment. For some of the sleeps the sleeping Dragon technique might be an approach to get the load issue under control.

My personal trunk meanwhile differs a lot from the original, so I am not completely sure how much this will help. I am using signals too, some modules are rewritten. All in all I now have 3% cpu use for login and chat, 6% for connection and a bit over 10% for a Zone with a stable system load under 0.5 for an idle system. That still is far from ideal, but already a bit closer to a working solution.
lei2
SWG:ANH Staff
 
Posts: 25
Joined: September 23rd, 2015, 1:11 pm
SWG Official Server: Farstar

Re: October/November Development Objectives

Postby Obi-Two » October 28th, 2015, 1:50 pm

Thank you for the update, I will try and get them copied over as soon as I can,

I forsee cros platform problems, but at this time I'm only looking at liux building for our test centre but maybe soemthing to be mindful of :D other than that fantastic stuff.

lei2 wrote: In src/Utils/ActiveObject.cpp add and usleep(2000) between line 66/67. Same
for other spots where sleep is used:
ChatServer/ChatServer.cpp: boost::this_thread::sleep(boost::posix_time::milliseconds(1));


woudl this then become
Code: Select all
ChatServer/ChatServer.cpp: boost::this_thread::sleep(boost::posix_time::milliseconds(1));
usleep(2000);


just want to make sure I'm understanding this

Thanks again Obi
Obi-Two

Quality Assurance
Star Wars Galaxies:A new hope
there is another...

Server by : Hostwind
Project : SWGANHServices/SWGANHJava
Obi-Two
SWG:ANH Staff
 
Posts: 248
Joined: January 3rd, 2009, 9:37 am
Location: Manchester, England
SWG Official Server: Ahazi

Re: October/November Development Objectives

Postby lei2 » October 28th, 2015, 3:54 pm

Cross platform was a major concept error. Nothing else. If there is a future, its linux.

Obi-Two wrote:woudl this then become
Code: Select all
ChatServer/ChatServer.cpp: boost::this_thread::sleep(boost::posix_time::milliseconds(1));
usleep(2000);



Ah sorry, hell no : )

ActiveObjects seem to grab things, if it has one or more it will set up an event timer, delay them a bit and then execute them. Add the usleep here to make it wait a bit if there is nothing to do. It will be a 2ms delay.
If you want to you can use boost::this_thread::sleep(boost::posix_time::milliseconds(2) instead of the usleep. Also you can adjust the boost sleep in the while directive to sleep a bit longer, guess that will work as well.

For the other boost sleeps - nothing to add here, just adjust the sleep time a bit. It might need some experimenting to find the tolerable balance between answer time latency and system load for each server.
You see differences in timing for the servers, that already reflect their timing needs for a straight answer. So for instance start with factor 3 for each timer and then see how it works. Depends on your system and how many sessions you expect.

There is one boost sleep in the heightmap load, that should stay 1 as it is.

And the last one - the sleep in socket read thread. Have no idea why its in there and what its good for, just delete it. It does only limit the packet receive to 50 packets per second.


Greetz,

lei
lei2
SWG:ANH Staff
 
Posts: 25
Joined: September 23rd, 2015, 1:11 pm
SWG Official Server: Farstar

Re: October/November Development Objectives

Postby Obi-Two » October 29th, 2015, 5:34 am

I think i understand now,

we are replacing
Code: Select all
boost::this_thread::sleep(boost::posix_time::milliseconds(1));

with the
Code: Select all
usleep(2000)

code ?

also where does this go exactly,as line 50 seems to be in the middle of something lol, (I'm not much of a programmer sorry)
Code: Select all
 edit src/NetworkManager/SocketWriteThread.cpp. In Constructor - line 50 or so - add
sessionPipe = new Pipe();
,

as a further thought, in the interests of speed, would it be better for me to set up a folder via my own cloud server or drop box you can just update the src folder every now and again. would save all the typing haha

thank you again, i might have to paypal you some beer money one day
Obi
Obi-Two

Quality Assurance
Star Wars Galaxies:A new hope
there is another...

Server by : Hostwind
Project : SWGANHServices/SWGANHJava
Obi-Two
SWG:ANH Staff
 
Posts: 248
Joined: January 3rd, 2009, 9:37 am
Location: Manchester, England
SWG Official Server: Ahazi

Re: October/November Development Objectives

Postby lei2 » October 29th, 2015, 11:37 am

Obi-Two wrote:I think i understand now,

we are replacing
Code: Select all
boost::this_thread::sleep(boost::posix_time::milliseconds(1));

with the
Code: Select all
usleep(2000)

code ?


Yes, but you can as well say boost::this_thread::sleep(boost::posix_time::milliseconds(2));
if you will stay with boost.

Obi-Two wrote:also where does this go exactly,as line 50 seems to be in the middle of something lol, (I'm not much of a programmer sorry)
Code: Select all
 edit src/NetworkManager/SocketWriteThread.cpp. In Constructor - line 50 or so - add
sessionPipe = new Pipe();



I should have reminded that my code meanwhile differs too much for line numbers. So the reference should be https://github.com/obi-two/Unofficial_Hope/blob/development/src/NetworkManager/SocketWriteThread.cpp#L93 then. Sorry for that.

Obi-Two wrote:as a further thought, in the interests of speed, would it be better for me to set up a folder via my own cloud server or drop box you can just update the src folder every now and again. would save all the typing haha

thank you again, i might have to paypal you some beer money one day
Obi


Np, never mind. Ill set up a git acc if it once is too much to type. But else ... Im old-school, a bit typing helps in understanding and keeps fingers away from bad things ; )

And dont expect wonders from the timing things. They may help a bit, but wont make the polling issue go away.

Greetz, lei
lei2
SWG:ANH Staff
 
Posts: 25
Joined: September 23rd, 2015, 1:11 pm
SWG Official Server: Farstar

Re: October/November Development Objectives

Postby Obi-Two » October 29th, 2015, 12:41 pm

lei2 wrote:
Np, never mind. Ill set up a git acc if it once is too much to type. But else ... Im old-school, a bit typing helps in understanding and keeps fingers away from bad things ; )

And dont expect wonders from the timing things. They may help a bit, but wont make the polling issue go away.

Greetz, lei


it would be no problem I just like keep in the code moving forward, I'm not overly fuzzed about keeping boost, its wasa massive faff for me to build haha also I'm nto to bothered about it bebuild windows compatable, in this day and age its east to run a VM linux machine and push a server out, pluss as stated before who can afford a Windows Box to run threre server on unless they want a small private comunity runnig on a home laptop/pc but they I'd have to point them to the running of a VM server :D

I'll make the changes to the code and push it out, it might encourage others to help (might not lol)

Cheers
Obi :D

EDIT:
I think I missed something :S I get this on running the build server
Code: Select all
[ 10%] Building CXX object src/NetworkManager/CMakeFiles/NetworkManager.dir/Session.cpp.o
cd /unofficial_hope/build/src/NetworkManager && /usr/bin/c++   -DDYNLOAD_MYSQL_LIB=\"/unofficial_hope/deps/mysql-connector-c/lib/libmysqlclient_r.so\" -std=c++0x -Wno-invalid-offsetof -Wno-write-strings -I/unofficial_hope/deps/mysql-connector-c/include -I/unofficial_hope/src -I/unofficial_hope/deps/boost/include -I/unofficial_hope/deps/glm/include -I/unofficial_hope/deps/tbb/include    -fPIC -o CMakeFiles/NetworkManager.dir/Session.cpp.o -c /unofficial_hope/src/NetworkManager/Session.cpp
In file included from /unofficial_hope/deps/boost/include/boost/log/sources/basic_logger.hpp:31:0,
                 from /unofficial_hope/deps/boost/include/boost/log/sources/severity_logger.hpp:23,
                 from /unofficial_hope/src/Utils/logger.h:8,
                 from /unofficial_hope/src/NetworkManager/Session.cpp:60:
/unofficial_hope/src/Utils/pipe.h: In member function ‘int utils::Pipe::wait(uint32)’:
/unofficial_hope/src/Utils/logger.h:34:4: error: ‘::NOTICE’ has not been declared
   (::level))
    ^
/unofficial_hope/deps/boost/include/boost/preprocessor/seq/enum.hpp:31:33: note: in definition of macro ‘BOOST_PP_SEQ_ENUM_1’
 # define BOOST_PP_SEQ_ENUM_1(x) x
                                 ^
/unofficial_hope/deps/boost/include/boost/log/sources/record_ostream.hpp:306:63: note: in expansion of macro ‘BOOST_PP_SEQ_ENUM’
     for (::boost::log::record rec_var = (logger).open_record((BOOST_PP_SEQ_ENUM(params_seq))); !!rec_var;)\
                                                               ^
/unofficial_hope/deps/boost/include/boost/log/sources/record_ostream.hpp:317:5: note: in expansion of macro ‘BOOST_LOG_STREAM_WITH_PARAMS_INTERNAL’
     BOOST_LOG_STREAM_WITH_PARAMS_INTERNAL(logger, BOOST_LOG_UNIQUE_IDENTIFIER_NAME(_boost_log_record_), params_seq)
     ^
/unofficial_hope/deps/boost/include/boost/log/sources/severity_feature.hpp:299:5: note: in expansion of macro ‘BOOST_LOG_STREAM_WITH_PARAMS’
     BOOST_LOG_STREAM_WITH_PARAMS((logger), (::boost::log::keywords::severity = (lvl)))
     ^
/unofficial_hope/deps/boost/include/boost/log/sources/severity_feature.hpp:304:36: note: in expansion of macro ‘BOOST_LOG_STREAM_SEV’
 #define BOOST_LOG_SEV(logger, lvl) BOOST_LOG_STREAM_SEV(logger, lvl)
                                    ^
/unofficial_hope/src/Utils/logger.h:33:2: note: in expansion of macro ‘BOOST_LOG_SEV’
  BOOST_LOG_SEV(::logger::get(),\
  ^
/unofficial_hope/src/Utils/pipe.h:74:63: note: in expansion of macro ‘LOG’
                                                 case  EINTR : LOG(NOTICE) << "Pipe::wait: pselect received signal.";
                                                               ^
make[2]: *** [src/NetworkManager/CMakeFiles/NetworkManager.dir/Session.cpp.o] Error 1
make[2]: Leaving directory `/unofficial_hope/build'
make[1]: *** [src/NetworkManager/CMakeFiles/NetworkManager.dir/all] Error 2
make[1]: Leaving directory `/unofficial_hope/build'
make: *** [all] Error 2
Obi-Two

Quality Assurance
Star Wars Galaxies:A new hope
there is another...

Server by : Hostwind
Project : SWGANHServices/SWGANHJava
Obi-Two
SWG:ANH Staff
 
Posts: 248
Joined: January 3rd, 2009, 9:37 am
Location: Manchester, England
SWG Official Server: Ahazi

Re: October/November Development Objectives

Postby lei2 » October 29th, 2015, 5:41 pm

/unofficial_hope/src/Utils/pipe.h:74:63: note: in expansion of macro ‘LOG’
case EINTR : LOG(NOTICE) << "Pipe::wait: pselect received signal.";

This is the relevant line - the old logger doesnt understand the NOTICE severity. Replace it with ERR, that will work.


Obi-Two wrote:it would be no problem I just like keep in the code moving forward, I'm not overly fuzzed about keeping boost, its wasa massive faff for me to build haha also I'm nto to bothered about it bebuild windows compatable, in this day and age its east to run a VM linux machine and push a server out, pluss as stated before who can afford a Windows Box to run threre server on unless they want a small private comunity runnig on a home laptop/pc but they I'd have to point them to the running of a VM server :D

I'll make the changes to the code and push it out, it might encourage others to help (might not lol)


Thats exactly the point. Pull out a VM and run what is needed.

Dont know how some minor code changes might have a great influence but who knows. A little advance always is better as stagnation.

Greets ... lei
lei2
SWG:ANH Staff
 
Posts: 25
Joined: September 23rd, 2015, 1:11 pm
SWG Official Server: Farstar

Re: October/November Development Objectives

Postby Obi-Two » October 29th, 2015, 5:50 pm

Yay that seamms to have done the trick, I'll leave it to complie over night, (fingers crossed)

lei2 wrote:Dont know how some minor code changes might have a great influence but who knows. A little advance always is better as stagnation.

My thoughts exactly, I really appriciate what your doing for us :D

Obi
Obi-Two

Quality Assurance
Star Wars Galaxies:A new hope
there is another...

Server by : Hostwind
Project : SWGANHServices/SWGANHJava
Obi-Two
SWG:ANH Staff
 
Posts: 248
Joined: January 3rd, 2009, 9:37 am
Location: Manchester, England
SWG Official Server: Ahazi

Re: October/November Development Objectives

Postby lei2 » October 31st, 2015, 12:17 pm

Btw for the testcenter ... did you try apt-get install cpulimit? https://wiki.ubuntuusers.de/cpulimit
Might help to tame the beasts a bit, login and ping do not really need more than a few cpu%, chat and connection might work with 10% each, guess tat should run with 20% but tutorial and maybe corellia and naboo would be good with 10%.

Greetz, lei
lei2
SWG:ANH Staff
 
Posts: 25
Joined: September 23rd, 2015, 1:11 pm
SWG Official Server: Farstar

Next

Return to Developer's Datapad

Who is online

Users browsing this forum: No registered users and 1 guest

cron