I have an application server. At a high level, this application server has users and groups. Users are part of one or more groups, and the server keeps all users aware of the state of their groups and other users in their groups. There are three major functions:
Updating and broadcasting meta-data relating to users and their groups; for example, a user logs in and the server updates this user's status and broadcasts it to all online users in this user's groups.
Acting as a proxy between two or more users; the client takes advantage of peer-to-peer transfer, but in the case that two users are unable to directly connect to each other, the server will act as a proxy between them.
Storing data for offline users; if a client needs to send some data to a user who isn't online, the server will store that data for a period of time and then send it when the user next comes online.
I'm trying to modify this application to allow it to be distributed across multiple servers, not necessarily all on the same local network. However, I have a requirement that backwards compatibility with old clients cannot be broken; essentially, the distribution needs to be transparent to the client.
The biggest problem I'm having is handling the case of a user connected to Server A making an update that needs to be broadcast to a user on Server B.
By extension, an even bigger problem is when a user on Server A needs the server to act as a proxy between them and a user on Server B.
My initial idea was to try to assign each user a preferred server, using some algorithm that takes which users they need to communicate with into account. This could reduce the number of users who may need to communicate with users on other servers.
However, this only minimizes how often users on different servers will need to communicate. I still have the problem of achieving the communication between users on different servers.
The only solution I could come up with for this is having the servers connect to each other, when they need to deal with a user connected to a different server.
For example, if I'm connected to Server A and I need a proxy with another user connected to Server B, I would ask Server A for a proxy connection to this user. Server A would see that the other user is connected to Server B, so it would make a 'relay' connection to Server B. This connection would just forward my requests to Server B and the responses to me.
The problem with this is that it would increase bandwidth usage, which is already extremely high. Unfortunately, I don't see any other solution.
Are there any well known or better solutions to this problem? It doesn't seem like it's very common for a distributed system to have the requirement of communication between users on different servers.
I don't know how much flexibility you have in modifying the existing server. The way I did this a long time ago was to have all the servers keep a TCP connection open to each other. I used a UDP broadcast which told the other servers about each other and allowed them to connect to new servers and remove servers that stopped sending the broadcast.
Then everytime a user connects to a server that server Unicasts a TCP message to all the servers it is connected to, and all the servers keeps a list of users and what server they are on.
Then as you suggest if you get a message from one user to another user on another server you have to relay that to the other server. The servers really need to be on the same LAN for this to work well.
You can run the server to server communications in a thread, and actually simulate the user being on the same server.
However maintaining the user lists and sending messages is prone to race conditions (like a user drops off while you are relaying the message from one server to another etc).
Maintaining the server code was a nightmare and this is really not the most efficient way to implement scalable servers. But if you have to use the legacy server code base then you really do not have too many options.
If you can look into using a language that supports remote processes and nodes like Erlang.
An alternative might be to use a message queue system like RabbitMQ or ActiveMQ, and have the servers talk to each other through that. Those system are designed to be scalable, and usually work off a Publish/Subscribe mechanism.
Related
I am developing a multiplayer game (client-server model) and I am stuck when it comes to scaling its servers.
I understand that most games never even reach 10 000+ players, and I don't think mine will either.
Though if I would be very lucky to get that I want to develop the servers so they cannot become a huge obsticle later.
I have searched a lot for a solution to my problem on the internet, checking GDC talks about it and checking other posts on this website, but none of them seems to solve my specific problem.
My current setup is below and all servers are written in C++ using ENet as my network library.
Game server
This server handles the actual gameplay of the game and requires quite a lot of CPU and packages being sent between the server and its connected clients.
But this dedicated server is hosted by the players themselves, so I don't have to think about scaling it at all.
Lobby server
This server handles the server list, containing all servers currently up.
All game servers are sending a UDP package to this server every 5 seconds to say they are still alive.
This is so the lobby server can keep an updated list of all servers currently online.
All clients are sending a UDP package to this server when they want to fetch all servers (which is only in the server
list screen), and the lobby server sends back a list of all servers.
This does not happen that often and the lobby server is limited to send 4 servers per second to a client (and not a huge package containing all servers).
Login server
This server handles creating accounts, lost password, logins, friends and their current game status,
private messages to other logged in players and player profiles that specifies what in-game items they have.
All clients are sending a UDP package to this server every 5 seconds to say they are still alive, while also
sending what game they are currently in. The server then sends back their friend lists online/offline/in-game statuses.
This is so their friends can keep an updated list of which friend is online/offline/in-game.
It sends messages only with player actions otherwise, like creating an account, logging in, changing/resetting password,
adding/removing/ignoring a friend, private messages to friends, etc.
My questions
What I am worried about is that my lobby and login server might not be scalable and that they would have too much traffic on them.
1. Could they in theory be hosted on just a single computer? Or would it be too much traffic for 10 000+ players?
2. If they can be hosted on a single computer, will the servers still not have issues for people that live far away?
Would it be better to have the lobby and login servers per region of the world in that case?
The bad thing about that is that the players would not be able to see servers in the US if they live in Europe, and that their account and items would not exist on the other servers.
3. Might be far-fetched, but if I would rewrite both servers to instead be on a website with a database and make the client/game server do
web requests instead (such as HTTPS or calling a php with specific headers),
would it help in solving my problems somehow?
All your problems and questions are solved by serverless cloud based solution AWS Lambda e.g. or similar. In this case the scalability is not your problem. Just develop the logic. This will save you much time.
If you would like to make servers as single app hosted by your own server. Consider using something like e.g. Go instead of C++. It's designed exactly for these purposes. I mean highly loaded web/network services.
Well, this is c++ and i code in java, but maybe the logic is useful for you any way so i will tell you how i end up implementing something similar but in a casino.
In my case I have 2 diferrent sockets in the same server program, one of the sockets is TCP and it handles all logins, registers and payments, while the second socket is UDP and it handle the actual game multiple payers are playing, then you could group internally all those UDP connection in groups (probably arrays of sockets) to generate those lobbies. Doing that all your server is just one class that could run in a single pc using 2 ports (one for each socket) However this do not solve the problem of the ping for people who live far away.
If ping is a problem (not my case in a casino) you could probably host your server region base but removing the login, registration and paymets of your server and replace it for a connection to a central server (this central server should be TCP and you could also implement a https socket to also allow your webpage to connect to this server and create accounts or pay you directly from the browser)
sorry to mess your life even more, but i hope it helps
I am running an Openfire server on a AWS EC2 instance and am able to connect to the server from my mobile devices and send messages back and forth. Of course, since XMPP is a client-server based protocol, I incur costs for running this traffic over the AWS server. However, for most use cases, this cost is not very high at all, as normal XMPP stanzas rarely seem to go above ca. 1 KB, so from this end all is ok.
I would now, however, like to include the ability to send images from one client to another. One way would be to use an HTTP server, to which user A uploads the picture and then sends the URL of the image to user B via XMPP, so that the user can now get the image via HTTP. There are also several other methods for sending images via XMPP. However, I am interested in doing this via Jingle.
As far as as I understand, Jingle is an out of band peer-to-peer extension to XMPP. My simple question is, since Jingle communicates peer-to-peer, i.e. without the use of a server, for the multimedia aspect of the session, will I even incur any data cost on AWS for transferring multimedia from one client to another using Jingle? Or put differently, if Jingle is peer-to-peer, does any data go via my AWS server using Jingle (except the session initiate, ack, session terminate stanzas)? If not, what kind of route does this data take, and how can anyone be billed for this traffic cost, if it is peer-to-peer?
Jingle is a negotiation mechanism, and there are a couple of different transports it could negotiate for file transfer. The most common transport is peer to peer bytestreams defined in http://xmpp.org/extensions/xep-0260.html - here the only traffic you'd see via the server would be the jingle negotiation, which is a similar sort of volume to other XMPP traffic). There is also an in-band bytestream transport defined in http://xmpp.org/extensions/xep-0261.html that some clients will use - typically for smaller transfers as it's inefficient, but has the advantage of working in hostile networks with NAT and firewalls. If you control the clients, simply not supporting IBB would be your best bet for ensuring the traffic doesn't travel via the server. If you don't, I'd suggest configuring your server to block IBB traffic.
I note as well that running a server-side proxy will drastically increase the odds of the out-of-band mechanism in 260 working in the face of hostile networks, at the cost of server bandwidth.
There is also the not-widely-deployed http://xmpp.org/extensions/xep-0343.html out of band transport.
Well...
I am working with an mobile application and a web server.
A characteristic of my web server is that it generates different set of data randomly. In other words, I cannot predict when the server will have ready data to send to the mobile app.
On other hand, the mobile app need to receive all data that the server generates. An approach could be request multiple times to get all these data. Indeed, It isn't a good approach, because I don't know when request the data.
If the mobile app could listen the server, after one start request or keep on the connection, for example, the server could sent any set of data in any time.
The question is: What is protocol suitable to this situation? How could I use that? Examples?
Thank you!
You could create a persistent TCP/IP connection to the server and permanently listen for incoming data (using a custom protocol or propably something websocket based). However such a permanent connection might seriously affect your battery life if it's for a mobile device. You will also lose the connection if the operating system automatically shuts down your application because it's out of memory.
The default approach to this problem are Push notification / Push services, where your server sends a notification about new data to a server of the phone provider (e.g. Microsoft or Apple push server), and this server sends the notification (as well as notificaiton from other online services) to your phone.
Some info for Windows Phone:
http://msdn.microsoft.com/en-us/library/hh221549.aspx
http://msdn.microsoft.com/en-us/library/windowsphone/develop/ff402558%28v=vs.105%29.aspx
Depending on how often you have new data both approaches can make sense.
WebSockets could be the answer: http://en.wikipedia.org/wiki/WebSocket
Specifically, for Windows Phone, there's a solution also: http://msdn.microsoft.com/en-us/library/windowsphone/develop/ff402558(v=vs.105).aspx
I would like to have some advices about my project. I am currently developing a musical application/game in C++/Qt, with a multiplayer mode, and I have the following requirements (ideally):
I want to be able to have friends contacts, and be able to chat and play with them
I need to send/receive realtime datas to these contacts (music notes)
I don't want to create a server application
What would you recommend to do this?
I was thinking of using xmpp protocol, so I can connect to google/jabber and retrieve contacts, chat with them. Actually this part works, but then I don't konw how to send/receive realtime datas. For the realtime part, I was thinking of using direct TCP communication, but I need to know external ip of my contacts, and I have no idea how to do it. I was thinking to automatically send my external ip and tcp port to all my contacts every time I connect, but I didn't find solution to retrieve external ip from code. So I'm a bit stuck. Any advice?
Is there alternative solutions? alternative protocols?
Thanks you,
Laurent
You're going to have a really hard time avoiding writing a server, for realistic, practical, and performance reasons:
Many residential internet connections are behind firewalls (at the ISP, local router, or OS level) that limit accepting connections from outside the network. NAT further complicates accepting connections from the internet on a LAN.
There are precious few methods of internet communication that are serverless, and those that are are subject to using local peer discovery to find peers. Most LPD traffic will not make it off your lan, the ISP will filter it (otherwise you'd be able to "locally" discover peers on the entire internet).
Bandwidth can be a concern for games. Not everyone has a high speed internet connection yet (though market penetration of fiber optics and fast DSL is pretty high at this point), and you'd end up with problems connecting slower hosts to a large swarm.
Servers facilitate star-like networks, which are very efficient. Other network topologies exist, but many suffer from drawbacks that severely inhibit their ability to scale.
Star networks, for example, for clients, require O(connections) =
O(1), O(bandwidth) = O(1), and O(latency) = O(1).
Fully connected networks require every client to be connected to
every other client, so O(connections) = O(bandwidth) = O(n), and
O(latency) = O(1).
In ring networks, each client connects to 2 neighbors and messages of
distant clients are forwarded. Therefore, they have O(connections) =
O(1), but O(bandwidth) = O(latency) = O(n).
If all you need is a chat system, or want badly enough not to write your own server that you're willing to piggyback the entire online experience over a chat server, you could probably rely on something like an XMPP server.
If you choose to go that route, make sure that proper authentication and encryption is used wherever necessary to protect user's private data (passwords, etc). I recommend using a cryptographic authentication scheme that allows clients to authenticate other clients (such as a challenge/response scheme, or something else). Or, you could mediate all authentication with a central service.
Keep in mind that many chat services will not want to provide your project with free bandwidth. Even if you do decide to use XMPP as the heart of your multiplayer protocol, expect to be running your own server.
The idea i'm having is that clients can connect to a chat room on a server and communicate with each other. In the chat room you should also be able to target another user and they should be able to talk with each other.
Now to the problem. I'm not sure which way is the easiest/best way to implement this. For the chat room I thought of when a user writes something, the message is sent to the server and the server then echoes that message to the other clients. Not sure what other options I have.
What i'm most confused about is how I can make only 2 clients talk to each other. Either the server act as a proxy and just forward the messages to the other client, this seem inefficient though. The only alternative to this I can think of is that the 2 clients establish a connection between each other. Which implementation is most common in order to achieve this?
I am using unix sockets with C++.
There are a few options for many-to-many chat and one-to-one chat. However, the only reasonably sane option for many-to-many chat is to do as you've said: send a message to the centralized server, the server relays the message to all other connected clients (or, in the same "room" / "channel").
For one-to-one chat, I recommend that you follow the same exact model: it's just a special case of the many-many chat relay in which the message is sent by the server, as a proxy, to only one other connected client. This is simple, and it hides every client's IP address from every other.
However, if the one-to-one communication were to become more voluminous than chat (e.g., a file transfer), direct one-to-one communication may be appropriate. In this case, the server should relay the initiation of a direct, peer-to-peer communication channel to the remote user, probably exchanging IP addresses upon setup, and the clients would then connect directly to each other for their special-purpose direct communication (while usually, though optionally, remaining connected to the server).
Thus, one-to-one communication is normally proxied by the server as in the general case of many-to-many, and the degree to which that practice is inefficient is superficial. Special purpose one-to-one communication (file transfers, VoIP, etc.) are done with direct client-to-client connections usually orchestrated at first by the server (i.e., to prepare each side for direct communication).
Implementation hint: the server is all TCP. Read about non-blocking sockets, the POSIX system call poll, and let the idea of message framing over TCP roll around in your head. You can then skip multithreading [and scalability] issues in the server code. The clients, besides speaking the same custom TCP protocol as your server, are up to you.
Well you can implement a multi-threaded client/server. A single "server" relaying the messages is the right way to go(to preserve global ordering of messages). Also think about leader election algorithms(http://en.wikipedia.org/wiki/Bully_algorithm) for example in case your "server" goes down. Check out
Another way to do this would be to use signals and event-driven programming.