Currently I am using enet to stream info, updated multiple times per second, to a list of
provided ip addresses / ports. I am creating a client connection for each of these
addresses (and there are server enet host connections running on those computers).
I am wondering if there is a more efficient way to send these messages(or simply a better solution to the problem)? Some way that I do not have to create
a enet host for each connection? I do need reliable delivery.
I am only using enet as it is convenient and already available in the system I am using.
A great solution for this problem is ZeroMQ.
It is a networking library with C++ bindings that specializes in the reliable delivery of messages in a variety of configurations, like the pub/sub model that looks like what you are trying to do.
ZeroMQ would effectively allow you to not even need to list the IPs of the subscribers, since they would indeed subscribe by connecting to your publisher. The publisher then would send data regardless of the number of subscribers (see http://zguide.zeromq.org/page:all#Getting-the-Message-Out).
Related
I am a newbie to AWS and cloud computing in general, so I apologize if this question is foolish.
I am currently working on developing an app for Amazon Echo that would allow it to remotely control a PC (i.e. change volume, pause a movie, etc.). My problem is that I do not know how to communicate between my Amazon Lambda service and my Windows Application.
Any ideas?
There are potentially some problems with the way you have posed the question -- how to communicate between a Lambda Function and a Windows machine could involve a number of different solutions, but what you are looking for (as far as I can tell) is a more specific -- yet simultaneously more generalizable -- solution.
Are you trying to actually make an Alexa skill that users could use, or just something for yourself? It makes a big difference, because for just yourself there are a number of hacky solutions you could implement, like port forwarding and dynamic DNS, which fail dramatically if you try to do them in the real world. You need another component -- some kind of real-time push messaging -- that bridges between an "agent" in your Windows app and requests emitted by your Lambda code.
Your actual problem to solve is not so much how to communicate between AWS Lambda and a Windows Application, but rather one of a need for understanding how a platform like Alexa needs to communicate with a "smart home" device, specifically an entertainment device.
It is a relatively complicated undertaking, because -- fundamentally -- there is no way of communicating directly between Lambda and an arbitrary device out on the Internet. Dynamic IP addresses, network address translation (NAT), firewalls, security considerations, and other factors make it impossible to reliably initiate a connection from a Lambda function (or indeed from any Internet connected device) to any other arbitrary destination device. Most devices (my phone, my Alexa-controlled light switch, my Windows laptop) are running behind a boundary that assumes requests are initiated behind the boundary. When I open web sites, stream video, etc., I initiate the request and the response returns on the channel (often a TCP connection) that I have created, from behind my boundary (e.g. the router in my cable modem) that doesn't allow external initiation of TCP connections. They are bidirectional once established, but must be initiated from inside.
Of course, you can statically "poke a hole" in your router configuration by forwarding a specific TCP port to a specific internal (usually private) IP address, which works as long as your Internet provider doesn't change your IP address, and your internal device doesn't get a new IP address... and there'a UPnP NAT Traversal, which seems like a good solution until you realize that it is also terrible (though for a "hobbyist" application, it could work).
While this is a long and complex topic, the short answer is that Alexa, via Lambda code, is only capable of initiating connections, and your device, wherever it may be, is only capable of initiating connections -- not receiving them... and thus you need some kind of "meet in the middle" solution: something that allows the device to maintain its "connection" to a central "service" that can coordinate the interactions on demand.
For example:
AWS IoT Core is a managed cloud platform that lets connected devices easily and securely interact with cloud applications and other devices. AWS IoT Core can support billions of devices and trillions of messages, and can process and route those messages to AWS endpoints and to other devices reliably and securely. With AWS IoT Core, your applications can keep track of and communicate with all your devices, all the time, even when they aren’t connected.
https://aws.amazon.com/iot-core/
The client initiates the connection (e.g. via a web socket) to the IoT platform, and maintains it, so that when a message arrives at IoT, the service knows how to deliver that message to the client when it's received. ("even when they aren't online" refers to the "device shadow" capability, which allows you to programmatically interact with a proxy for the device, e.g. knowing the last temperature setting of a thermostat, and asking the thermostat to change its set point when the connection is re-established at some future point).
Or, potentially something like this:
Firebase Cloud Messaging (FCM) is a cross-platform messaging solution that lets you reliably deliver messages at no cost.
Using FCM, you can notify a client app that new email or other data is available to sync.
https://firebase.google.com/docs/cloud-messaging/
Both of these potential solutions solve the problem by "knowing how to contact" arbitrary devices, wherever they may be... and I would suggest that this is the core of your actual need.
There are a lot of alternatives for such a "service," including roll-your-own websocket or HTML EventSource implementations with servers... the purpose of this is not product recommendations but rather to give you an idea of what you would need for such a scenario -- an intermediate platform that can be interacted with by the Lambda code, which also knows how to communicate with "agent" code running on the device... because both Lambda and the agent need to initiate the communication channels and thus additional components are required to bridge them together.
I am trying to write a simple client-server application where a client can send or broadcast a message to one or all clients in the network. The server stores all IP addresses that are connected to it, and broadcasts a new IP if a new client connects itself.
I'm not quite sure how to implement the sending of a single message to another client. Would I just have to send a TCP message to the server and put the desired recipient as data in the TCP layer which is then extracted by the server so it knows where to send it?
I also want to add encryption to the messages which would then no longer allow the server to read the data, so I'm not sure how to solve that!?
I am using c++ and Qt5 for the implementation
I'm not quite sure how to implement the sending of a single message to
another client. Would I just have to send a TCP message to the server
and put the desired recipient as data in the TCP layer which is then
extracted by the server so it knows where to send it?
In an ideal world, the clients could talk to each other directly, since they could find out the IP addresses of the other clients from the server (either via its broadcast or by requesting a list of IP addresses from the server). If all of your clients are running on the same LAN, that can work well.
Assuming you want your system to run on the general Internet, however, that won't work so well, since many/most clients will be behind various firewalls and so they won't accept incoming TCP connections. (There are some ways around that, but they require a very advanced understanding of how TCP works, and even then they only work in certain situations, so I don't recommend attempting them in a first project)
Therefore, for a reliable client->client messaging mechanism, your best bet is indeed to have the sending client send the message to the server, along with some short header that tells the server which other client(s) the message ought to be forwarded to. My own client/server messaging system works along these lines, and I've found it to work well.
I also want to add encryption to the messages which would then no
longer allow the server to read the data, so I'm not sure how to solve
that!?
Don't worry about adding encryption until you've got the basic non-encrypted functionality working first, since encryption will make things much more difficult to debug. That said, it's perfectly possible to pass encrypted/opaque data to the server, as long as the aforementioned header data (which tells the server where to forward the message to) is not encrypted (since the server will need to be able to read the header to know what to do with the encrypted data). The trickier part will be when the receiving client gets the forwarded data from the server -- how will the receiving client know how to decrypt it? You'll need some external mechanism for clients to share keys (either symmetric keys or public/private keypairs), since if you sent the encryption keys themselves through the server, there wouldn't be much point in encrypting anything (since the server could retain a copy of any keys it forwarded, and use them to decrypt, if it wanted to)
First, I want to give thanks for that amazing lib! I love it. A client is connecting himself to a server. The server should save the IP and do stuff with it later on (I really need the IP). I found that answer: http://lists.zeromq.org/pipermail/zeromq-dev/2010-September/006381.html but I don't understand how I get the IP out of the message (a XREP)... I think I am only able to read the ID, but the IP is managed internally by 0MQ. His second solution suggests to send the IP as part of the message, but I don't understand how to get the "public"-IP. I found that post: Get TCP address information in ZeroMQ
is pass bind a service to an ephemeral port, get a full connection endpoint ("tcp://ipaddress:port")
I don't get how this works. Does he mean something like a web-service?
In my opinion, it would be best to get the IP out of 0MQ (it has the IP already). I would even adjust 0MQ for that, if somebody could point to the place where the IP is saved, couldn't find it. The socket types are not that important, at the moment. I would prefer smth REQ-REP like. Thank you!
Summary:
TL;DR answer to your question is: you can't get IP address of the peer that sent a message, using ZeroMQ API.
Explanation:
ZeroMQ does not expose peer IP address because it is irrelevant for the message based communication that ZeroMQ is designed for. When it is possible for ZeroMQ to get IP address of client that is connecting to server (in example using method described here), it is useless. For a longer explanation here is how it works inside ZeroMQ and any other server implementation.
Server side of the connection does not handle connected clients by the means of the hashtable that maps IP to client, but by keeping track of connected "sockets" (socket descriptors) - when a server accepts (using accept()) a connection, it receives from operating system socket descriptor to use to communicate with connected peer. All server has to do is keep that descriptor around to read() from and write() to that client. Another client that connects to server receives another socket descriptor.
To summarize: even if ZeroMQ would be able to provide you with IP of connected peer, you should not depend on it. ZeroMQ hides from you connection management so you can focus on messaging. Connection management includes reconnections, which may result in a change of IP without changing the actual ZeroMQ socket connected on the other side.
So here's an example of why you might want to get the ip address a message was delivered from: we have a server whose job it is to synchronize updates onto occasionally-connected clients (think mobile devices here, though this is an extreme example of a mobile deivce.)
When the mobile unit comes onto the network, it sends a list of it's firmware files to the server via a dealer-router connection. The server has a list of all applicable firmware files; if the client needs an update it will initiate an update via a separate mechanism.
Since the IPs for the devices can (and do) change, we need to know the IP address associated with the mobile device FOR THIS CONNECTION, i.e. right now.
Yes, we absolutely can have the client send it's IP address in the message, but that's a waste of another n bytes of valuable satellite air time, and while not pure evil, is sure annoying. Zmq already has this information, if it didn't have it, it wouldn't be able to generate replies. The address is in the socket data, there's no reason the message couldn't (optionally, for all you guys who use wired networks and think disconnects are the exception) include a reference to the socket structure so you can get the address out of it. Other than pedantic religiosity, which is far too common in zmq.
The way ZeroMQ is designed there's no information provided on the remote IP. As far as I know you have to manage this through your application by sending that information as a message of some sort.
The messages themselves use an IP-agnostic ID which has more to do with the instance of ZeroMQ running than any particular interface. This is because there may be more than one transport method and interface connecting the two instances.
I have a certain application running on my computer. The same application can run on many computers on a LAN or different places in the world. I want to communicate between them. So I basically want a p2p system. But I will always know which computers(specific IP address) will be peers. I just want peers to have join and leave functionality. The single most important aim will be communication speed and time required. I assume simple UDP multicast (if anything like that exists) between peers will be fastest possible solution. I dont want to retransmit messages even if lost. Should I use an existing p2p library e.g. libjingle,etc. or just create some basic framework from scratch as my needs are pretty basic?
I think you're missing the point of UDP. It's not saving any time in a sense that a message gets faster to the destination, it's just you're posting the message and don't care if it arrives safely to the other side. On WAN - it will probably not arrive on the other side. UDP accross networks is problematic, as it can be thrown out by any router on the way which is tight on bandwidth - there's no guarantee of delivery for it.
I wouldn't suggest using UDP out of the topology under your control.
As to P2P vs directed sockets - the question is what it is that you need to move around. Do you need bi/multidirectional communication between all the peers, or you're talking to a single server from all the nodes?
You mentioned multicast - that would mean that you have some centralized source of data that transmits information and all the rest listen - in this case there's no benefit for P2P, and multicast, as a UDP protocol, may not work well accross multiple networks. But you can use TCP connections to each of the nodes, and "multicast" on your own, and not through IGMP. You can (and should) use threading and non-blocking sockets if you're concerned about sending blocking you, and of course you can use the QoS settings to "ask" routers to rush your sockets through.
You can use zeromq for support all network communication:
zeromq is a simple library encapsulate TCP and UDP for high level communication.
For P2P you can use the different mode of 0mq :
mode PGM/EPGM for discover member of P2P on your LAN (it use multicast)
mode REQ/REP for ask a question to one member
mode PULL/PUSH for duplicate one resource on the net
mode Publish/subscribe for transmission a file to all requester
Warning, zeromq is hard to install on windows...
And for HMI, use green-shoes ?
i think you should succeed using multicast,
unfortunately i do not know any library,
but still in case you have to do it from scratch
take a look at this:
http://www.tldp.org/HOWTO/Multicast-HOWTO.html
good luck :-)
I would like to create a connection between two applications. Should I be using Client-Server or is there another way of efficiently communicating between one another? Is there any premade C++ networking client server libraries which are easy to use/reuse and implement?
Application #1 <---> (Client) <---> (Server) <---> Application #2
Thanks!
Client / server is a generic architecture pattern (much like factory, delegation, inheritance, bridge are design patterns). What you probably want is a library to eliminate the tedium of packing and unpacking your data in a format that can be sent over the wire. I strongly recommend you take a look at the protocol buffers library, which is used extensively at Google and released as open source. It will automatically encode / decode data, and it makes it possible for programs written in different languages to send and receive messages of the same type with all the dirty work done for you automatically. Protobuf only deals with encoding, not actually sending and receiving. For that, you can use primitive sockets (strongly recommend against that) or the Boost.Asio asynchronous I/O library.
I should add that you seem to be confused about the meaning of client and server, since in your diagram you have the application talking to a client which talks to a server which talks to another application. This is wrong. Your application is the client (or the server). Client / server is simply a role that your application takes on during the communication. An application is considered to be a client when it initiates a connection or a request, while an application is considered to be a server when it waits for and processes incoming requests. Client / server are simply terms to describe application behavior.
If you know the applications will be running on the same machine, you can use sockets, message queues, pipes, or shared memory. Which option you choose depends on a lot of factors.
There is a ton of example code for any of these strategies as well as libraries that will abstract away a lot of the details.
If they are running on different machines, you will want to communicate through sockets.
There's a tutorial here, with decent code samples.