We are developing a gamified environment which facilitates learning of 'boring' material. Recently, we have launched with the 1.0 release that comprises of the following modules:
Django based web portal for learning courses, viewing leaderboard (scoring), personal activity and achievements.
Tornado based real-time feeds and notification service. We are utilized sockjs-tornado developed by mrjoes. We took benefit from shared session: if user logged in to django site, he automatically opened a channel with our Comet backend.
Tornado based asynchronous multiplayer game server. Again, we have used sockjs-tornado for connecting and serving players (sockets), synchronizing game loop events across the network.
Now, I am working on scalability and high-availability features of the system. The main challenge is to distribute load across game cluster nodes properly and consistently (no data loss, fail-over in case of crash, re-balance if one of game servers (loop) suddenly crashes). The main goal is to make client interaction with game as seamless as possible: client shouldn't make additional actions: manually reconnecting if game crashed due to server fail. The figure depicts a possible architecture of distributed game server.
In case some actions performed on .../game on Django site (CRUD actions), tornado-based Game Router notified in real-time through a kind of oplog collection (capped collection in mongodb which is tailed by Game Router). If this is a new game, Game Router determines next host (game server) for the game by special algorithm (fastest heartbeat trip, or min number of live(active) games). Further, Game Router notifies appropriate Loop by publishing message to topic exchange.
If any game server crashes, Game Router will be immediately notified and re-balances load by taking idle games in memcached cluster.
In case of fail, how client can seamlessly reconnect to new game server ? I don't want to open redundant channel (socket connection).
Whether I am going right way or not ? Please, provide me with good approaches, critics, techniques. It would be nice if you share your minds by drawing diagrams.
Thank you in advance.
Related
I am developing a multiplayer game (client-server model) and I am stuck when it comes to scaling its servers.
I understand that most games never even reach 10 000+ players, and I don't think mine will either.
Though if I would be very lucky to get that I want to develop the servers so they cannot become a huge obsticle later.
I have searched a lot for a solution to my problem on the internet, checking GDC talks about it and checking other posts on this website, but none of them seems to solve my specific problem.
My current setup is below and all servers are written in C++ using ENet as my network library.
Game server
This server handles the actual gameplay of the game and requires quite a lot of CPU and packages being sent between the server and its connected clients.
But this dedicated server is hosted by the players themselves, so I don't have to think about scaling it at all.
Lobby server
This server handles the server list, containing all servers currently up.
All game servers are sending a UDP package to this server every 5 seconds to say they are still alive.
This is so the lobby server can keep an updated list of all servers currently online.
All clients are sending a UDP package to this server when they want to fetch all servers (which is only in the server
list screen), and the lobby server sends back a list of all servers.
This does not happen that often and the lobby server is limited to send 4 servers per second to a client (and not a huge package containing all servers).
Login server
This server handles creating accounts, lost password, logins, friends and their current game status,
private messages to other logged in players and player profiles that specifies what in-game items they have.
All clients are sending a UDP package to this server every 5 seconds to say they are still alive, while also
sending what game they are currently in. The server then sends back their friend lists online/offline/in-game statuses.
This is so their friends can keep an updated list of which friend is online/offline/in-game.
It sends messages only with player actions otherwise, like creating an account, logging in, changing/resetting password,
adding/removing/ignoring a friend, private messages to friends, etc.
My questions
What I am worried about is that my lobby and login server might not be scalable and that they would have too much traffic on them.
1. Could they in theory be hosted on just a single computer? Or would it be too much traffic for 10 000+ players?
2. If they can be hosted on a single computer, will the servers still not have issues for people that live far away?
Would it be better to have the lobby and login servers per region of the world in that case?
The bad thing about that is that the players would not be able to see servers in the US if they live in Europe, and that their account and items would not exist on the other servers.
3. Might be far-fetched, but if I would rewrite both servers to instead be on a website with a database and make the client/game server do
web requests instead (such as HTTPS or calling a php with specific headers),
would it help in solving my problems somehow?
All your problems and questions are solved by serverless cloud based solution AWS Lambda e.g. or similar. In this case the scalability is not your problem. Just develop the logic. This will save you much time.
If you would like to make servers as single app hosted by your own server. Consider using something like e.g. Go instead of C++. It's designed exactly for these purposes. I mean highly loaded web/network services.
Well, this is c++ and i code in java, but maybe the logic is useful for you any way so i will tell you how i end up implementing something similar but in a casino.
In my case I have 2 diferrent sockets in the same server program, one of the sockets is TCP and it handles all logins, registers and payments, while the second socket is UDP and it handle the actual game multiple payers are playing, then you could group internally all those UDP connection in groups (probably arrays of sockets) to generate those lobbies. Doing that all your server is just one class that could run in a single pc using 2 ports (one for each socket) However this do not solve the problem of the ping for people who live far away.
If ping is a problem (not my case in a casino) you could probably host your server region base but removing the login, registration and paymets of your server and replace it for a connection to a central server (this central server should be TCP and you could also implement a https socket to also allow your webpage to connect to this server and create accounts or pay you directly from the browser)
sorry to mess your life even more, but i hope it helps
I'm trying to modify a game engine so it records events (like key presses), and store these in a MySQL database on a remote server. The game engine is written in C++, and I currently have the following straightforward architecture, using mysql++ to directly INSERTrecords into appropriate databases:
Unfortunately, there's a very large overhead when connecting to the MySQL server, and the game stops for a significant amount of time. Pushing a batch of Xs worth of events to the server causes a significant delay in gameplay (60s worth of events can take 12s to synchronise). There are also apparently security concerns with leaving the MySQL port accessible publicly.
I was considering an alternative option, instead sending commands to the server, which can interact with the database in its own time:
Here the game would only send the necessary information (e.g. the table to update and the data to insert). I'm not sure whether the speed increase would be sufficient, or what system would be appropriate for managing the commands sent from the game.
Someone else suggested Log4j, but obviously I need a C++ solution. Is there an appropriate existing framework for accomplishing what I want?
Most applications gathering user-interface interaction data (in your case keystrokes) put it into a local file of some sort.
Then at an appropriate time (for example at the end of the game, or the beginning of another game), they POST that file, often in compressed form, to a publicly accessible web server. The software on the web server decompresses the data and loads it into the analytics system (the MySQL server in your case) for processing.
So, I suggest the following.
stop making your MySQL server's port available to people you don't know and trust.
get your game to gather keystrokes locally somehow.
get it to upload that data in big bunches when your game is not in realtime mode.
write a web service to receive and interpret these files.
That way you'll build a more secure analytics system and a more responsive game.
I want people to controll my arduino robot via the internet. It's important that the controlling reacts very fast. The user may send many requests per second.
Let me explain my architecture:
The user connects to a web-frontend. He can use a virtual joystick and buttons. The frontend will then send orders (like "motor1:255" "motor2:0"....) to an application server (Wildfly).
When a frontend-session starts, Wildfly will establish a connection to my computer or smartphone using a socket. The orders will be passed to the arduino using bluetooth. When a frontend-session is not longer active, the socket will be closed.
One Wildfly should be able to controll up to 10 robots. One robot can be controlled by exactly one user. Some developers use a mysql table and add a row for each incoming order. I don't think this would work in my case.
Is it okay to use REST to send the orders from the frontend to the application server? Is there any other fast and secure way to transport the user input from the frontend to the business logic?
Goot
REST, when properly understood and applied, is a solution for problems of long-term evolution and maintenance of your application. It doesn't seem to be your case.
What you mean is probably if an HTTP API is a good solution if you have lots of requests, and the answer is, it depends. I'd probably look into something like ZeroMQ for doing what you want.
Hello Stackoverflowers,
We're developing an online board-game (think online monopoly) site using Python for the backend.
We use Django for the non realtime stuff (authenticating, player profiles, ranking...). The chat server is implemented using socket.io and Tornodo. The game server part is what caused us problems.
We currently (that could change) also use Tornado and socket.io, each Tornado instance is located at a gameX.site.com address on a (maybe) different server and host several games simultaneously (much like a chat server in fact, except that messages would not go to all users but only to the ones involved in the same game).
What cause us trouble is how to we update the Django instance (game log, score, and so on) as games progress. Also we'd like to use Django for authentication as each player would ask the Django server to join the game and be given a disposable id/password couple just for it. Obviously we would have to communicate those to the game server in some way.
At first the chosen solution was to use something like Redis as a bidirectional message queue, Django would post is/password to Redis and the Tornado would then querying Redis on incoming connection. Also a Django cron would run every minute or so to deal with the waiting message. But we fear that frequently and possibly long running cron would impede the main site since the PostgreSQL database is hosted on the same server as Django (and some Game server may also run on the same machine).
We could alternatively wait for a player to request a ranking updated to process the past games results but we fear such an indefinite delay will skew the overall ranking (and experience) and would possibly cause data loss.
We could use Celery/RabbitMQ to update the Main database using Django ORM out of the Tornado processes, but would it be possible to use the same solution to communicate the temporary id/password to the game server ? It doesn't look like you can post a message to Celery and retrieve it on an other side.
Thank for your insight.
How are Massively Multiplayer Online RPG games built?
What server infrastructure are they built on? especially with so many clients connected and communicating in real time.
Do they manage with scripts that execute on page requests? or installed services that run in the background and manage communication with connected clients?
Do they use other protocols? because HTTP does not allow servers to push data to clients.
How do the "engines" work, to centrally process hundreds of conflicting gameplay events?
Thanks for your time.
Many roads lead to Rome, and many architectures lead to MMORPG's.
Here are some general thoughts to your bullet points:
The server infrastructure needs to support the ability to scale out... add additional servers as load increases. This is well-suited to Cloud Computing by the way. I'm currently running a large financial services app that needs to scale up and down depending on time of day and time of year. We use Amazon AWS to almost instantly add and remove virtual servers.
MMORPG's that I'm familiar with probably don't use web services for communication (since they are stateless) but rather a custom server-side program (e.g. a service that listens for TCP and/or UDP messages).
They probably use a custom TCP and/or UDP based protocol (look into socket communication)
Most games are segmented into "worlds", limiting the number of players that are in the same virtual universe to the number of game events that one server (probably with lots of CPU's and lots of memory) can reasonably process. The exact event processing mechanism depends on the requirements of the game designer, but generally I expect that incoming events go into a priority queue (prioritized by time received and/or time sent and probably other criteria along the lines of "how bad is it if we ignore this event?").
This is a very large subject overall. I would suggest you check over on Amazon.com for books covering this topic.
What server infrastructure are they built on? especially with so many clients connected and communicating in real time.
I'd guess the servers will be running on Linux, BSD or Solaris almost 99% of the time.
Do they manage with scripts that execute on page requests? or installed services that run in the background and manage communication with connected clients?
The server your client talks to will be a server running a daemons or service that sits idle listening for connections. For instances (dungeons), usually a new process is launched for each group, which would mean there is a dispatcher service somewhere mananging this (analogous to a threadpool)
Do they use other protocols? because HTTP does not allow servers to push data to clients.
UDP is the protocol used. It's fast as it makes no guarantees the packet will be received. You don't care if a bit of latency causes the client to lose their world position.
How do the "engines" work, to centrally process hundreds of conflicting gameplay events?
Most MMOs have zones which limit this to a certain amount of people. For those that do have 100s of people in one area, there is usually high latency. The server is having to deal with 100s of spells being sent its way, which it must calculate damage amounts for each one. For the big five MMOs I imagine there are teams of 10-20 very intelligent, mathematically gifted developers working on this daily and there isn't a MMO out there that has got it right yet, most break after 100 players.
--
Have a look for Wowemu (there's no official site and I don't want to link to a dodgy site). This is based on ApireCore which is an MMO simulator, or basically a reverse engineer of the WoW protocol. This is what the private WoW servers run off. From what I recall Wowemu is
mySQL
Python
However ApireCore is C++.
The backend for Wowemu is amazingly simple (I tried it in 2005 however) and probably a complete over simplification of the database schema. It does gives you a good idea of what's involved.
Because MMOs by and large require the resources of a business to develop and deploy, at which point they are valuable company IP, there isn't a ton of publicly available information about implementations.
One thing that is fairly certain is that since MMOs by and large use a custom client and 3D renderer they don't use HTTP because they aren't web browsers. Online games are going to have their own protocols built on top of TCP/IP or UDP.
The game simulations themselves will be built using the same techniques as any networked 3D game, so you can look towards resources for that problem domain to learn more.
For the big daddy, World of Warcraft, we can guess that their database is Oracle because Blizzard's job listings frequently cite Oracle experience as a requirement/plus. They use Lua for user interface scripting. C++ and OpenGL (for Mac) and Direct3D (for PC) can be assumed as the implementation languages for the game clients because that's what games are made with.
One company that is cool about discussing their implementation is CCP, creators of Eve online. They have published a number of presentations and articles about Eve's infrastructure, and it is a particularly interesting case because they use Stackless Python for a lot of Eve's implementation.
http://www.disinterest.org/resource/PyCon2006-StacklessInEve.wmv
http://us.pycon.org/2009/conference/schedule/event/91/
There was also a recent Game Developer Magazine article on Eve's architecture:
https://store.cmpgame.com/product/3359/Game-Developer-June%7B47%7DJuly-2009-Issue---Digital-Edition
The Software Engineering radio podcast had an episode with Jim Purbrick about Second Life which discusses servers, worlds, scaling and other MMORPG internals.
Traditionally MMOs have been based on C++ server applications running on Linux communicating with a database for back end storage and fat client applications using OpenGL or DirectX.
In many cases the client and server embed a scripting engine which allows behaviours to be defined in a higher level language. EVE is notable in that it is mostly implemented in Python and runs on top of Stackless rather than being mostly C++ with some high level scripts.
Generally the server sits in a loop reading requests from connected clients, processing them to enforce game mechanics and then sending out updates to the clients. UDP can be used to minimize latency and the retransmission of stale data, but as RPGs generally don't employ twitch gameplay TCP/IP is normally a better choice. Comet or BOSH can be used to allow bi-directional communications over HTTP for web based MMOs and web sockets will soon be a good option there.
If I were building a new MMO today I'd probably use XMPP, BOSH and build the client in JavaScript as that would allow it to work without a fat client download and interoperate with XMPP based IM and voice systems (like gchat). Once WebGL is widely supported this would even allow browser based 3D virtual worlds.
Because the environments are too large to simulate in a single process, they are normally split up geographically between processes each of which simulates a small area of the world. Often there is an optimal population for a world, so multiple copies (shards) are run which different sets of people use.
There's a good presentation about the Second Life architecture by Ian Wilkes who was the Director of Operations here: http://www.infoq.com/presentations/Second-Life-Ian-Wilkes
Most of my talks on Second Life technology are linked to from my blog at: http://jimpurbrick.com
Take a look at Erlang. It's a concurrent programming language and runtime system, and was designed to support distributed, fault-tolerant, soft-real-time, non-stop applications.