I a django app running on Elasticbeanstalk in AWS. Within my Django application I'd like to gather IoT data coming in via TCP/IP. Currently, I open the socket and switch it to listening through a View function. This leads to the problem that the socket closes, or stops. Furthermore, the socket needs to listen on the port steadily although data is not coming in continiously.
What is a more elegant way to solve this problem? Is there any Django extension to get the socket and listening from a view to a background task? E.g. listen every 60 seconds on the ports and create an object when data comes in?
Related
Given an AWS Elastic-Beanstalk Worker box, is it possible to use Flask/port:80 to serve the messages coming in from the associated SQS queue?
I have seen conflicting information about what is going on, inside an ELB-worker. The ELB Worker Environment page says:
Elastic Beanstalk simplifies this process by managing the Amazon SQS queue and running a daemon process on each instance that reads from the queue for you. When the daemon pulls an item from the queue, it sends an HTTP POST request locally to http://localhost/ on port 80 with the contents of the queue message in the body. All that your application needs to do is perform the long-running task in response to the POST.
This SO question Differences in Web-server versus Worker says:
The most important difference in my opinion is that worker tier instances do not run web server processes (apache, nginx, etc).
Based on this, I would have expected that I could just run a Flask-server on port 80, and it would handle the SQS messages. However, the post appears incorrect. Even the ELB-worker boxes have Apache running on them, apparently for doing health-checks (when I stopped it, my server turned red). And of course it's using port 80...
I already have Flask/Gunicorn on an EC2 server that I was trying to move to ELB, and I would like to keep using that - is it possible? (Note: the queue-daemon only posts messages to port 80, that can't be changed...)
The docs aren't clear, but it sounds like they expect you to modify Apache to proxy to Flask, maybe? I hope that's not the only way.
Or, what is the "correct" way of setting up an ELB-worker to process the SQS messages? How are you supposed to "perform the long-running task"?
Note: now that I've used ELB more, and have a fairly good understanding of it - let me make clear that this it not the use-case that Amazon designed the ELB-workers for, and it has some glitches (which will be noted). The standard use-case, basically, is that you create a simple Flask app, and hook it into an ELB-EC2 server, that is configured to make it simple to run that Flask app.
My use-case was, I already had an EC2 server with a large Flask app, running under gunicorn, as well as various other things going on. I wanted to use that server (as an image) to build the ELB server, and have it respond to SQS-queue messages. It's possible there are better solutions, like just writing a queue-polling daemon, and that no-one else will ever take this option, but there it is...
The ELB worker is connected to an SQS queue, by a daemon that listens to that queue, and (internally) posts any messages to http://localhost:80. Apache is listening on port 80. This is to handle health-checks, which are done by the ELB manager (or something in the eco-system). Apache passes non-health-check-requests, using mod_wsgi, to the Flask app that was uploaded, which is at:
/opt/python/current/app/application.py
I suspect it would be possible but difficult to remove Apache and handle the health-checks some other way (flask), thus freeing up port 80. But that's enough of a change that I decided it wasn't worth it.
So the solution I found, is to change which port the local daemon posts to - by reconfiguring it via a YAML config-file, it will post to port 5001, where my Flask app was running. This mean Apache can continue to handle the health-checks on port 80, and Flask can handle the SQS messages from the daemon.
You configure the daemon, and stop/start it (as root):
/etc/aws-sqsd.d/default.yaml
/opt/elasticbeanstalk/addons/sqsd/hooks/stop-sqsd.sh
/opt/elasticbeanstalk/addons/sqsd/hooks/start-sqsd.sh
/opt/elasticbeanstalk/addons/sqsd/hooks/restart-sqsd.sh
Actual daemon:
/opt/elasticbeanstalk/lib/ruby/bin/aws-sqsd
/opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/aws-sqsd-2.3/bin/aws-sqsd
Glitches:
If you ever use the ELB GUI to configure daemon options, it will over-write the config-file, and you will have to re-edit the Port (and re-start the daemon).
Note: All of the HTTP traffic is internal, either to the ELB eco-system or the worker - so it is possible to close off all external ports (I keep 22 open), such as Port 80. Otherwise your Worker has Apache responding to http://:80 posts, meaning it's open to the world. I assume the server is configured fairly securely, but Port 80 doesn't need to be open at all, for healthchecks or anything else.
I've got server running in background and a program which should display data from server. I want to somehow launch method in my program from server. So server should be a sender, but how to do it ?
There is no reason why a server can't also be a client, just implement the interfaces from both sides and you're good.
The main thing to worry about is deadlocking: if you have a single threaded program which is waiting for the reply of the server, then it will not handle the request that the server sends, so the server is stuck and will not send a reply to the program.
This can be solved by starting the server implementations on different threads and letting them not block on the client thread.
Even better is to avoid having a server send back requests before sending replies, but cascading requests (forward requests to more specialized servers) should be no problem.
I'm struggling with trying to figure out how to implement a TCP server in python. I currently have a TCP server that can connect to one client at a time. The Client can successfully communicate with the server, and upload a file using a single socket. Currently, it is single threaded and when one Client connects it blocks the other clients from connecting until it is done.
I'm struggling with the design portion on making this multi-client friendly. If I'm uploading files concurrently should there be multiple sockets? If I go with one socket, how do I differentiate data from different clients?
Can anyone give some advice on this?
I need to do some validation testing of a new feed handler I have made. I have some pcap data that I captured from the production network and I would like to have my development feed handler connect to the "replay" of this data and compare the results.
My pcap:
I have a prod application that connects to a data feed, a TCP connection to an external server lets call assume this is 123.456.789.1:1234. This external sever then sends data to my application there is almost no client to server communication the server just sends the client data until the client drops. I have a pcap of all the packets sent to and from port 1234. I got this pcap by mirroring the production port (SPAN) on the switch and attaching tcpdump to an interface plugged in to the mirrored network port. When I look at the PCAP in wireshark it has all the data I would expect.
My problem:
I am in no way a network engineer and I am unsure how I can use this pcap to test my application. What I would like to do is "replay" this stream form the pcap and connect to it with my development application to validate that the data is being handled the same was it was on the prod connection.
I would like to some how "replay" the data sent from 123.456.789.1:12344 on 127.0.0.1:1234 and then connect to 127.0.0.1:1234 with my dev application. I looked at tcpreplay but from the documents I can not seem to figure out if it can do this, I get the feeling that they do not handle the tcp session data, and I could do this if it was a UDP stream, but tcpreplay can not act as the external server. Did I read this wrong or is there another tool that will let me do this?
thanks!
You may want to use netcat if you just want to throw some data back at your tool, and you don't care about what the tool sends.
You would do this by extracting the raw data sent by your tool from the pcap file (this tool may be helpful) and then piping that into netcat.
Well...
I am working with an mobile application and a web server.
A characteristic of my web server is that it generates different set of data randomly. In other words, I cannot predict when the server will have ready data to send to the mobile app.
On other hand, the mobile app need to receive all data that the server generates. An approach could be request multiple times to get all these data. Indeed, It isn't a good approach, because I don't know when request the data.
If the mobile app could listen the server, after one start request or keep on the connection, for example, the server could sent any set of data in any time.
The question is: What is protocol suitable to this situation? How could I use that? Examples?
Thank you!
You could create a persistent TCP/IP connection to the server and permanently listen for incoming data (using a custom protocol or propably something websocket based). However such a permanent connection might seriously affect your battery life if it's for a mobile device. You will also lose the connection if the operating system automatically shuts down your application because it's out of memory.
The default approach to this problem are Push notification / Push services, where your server sends a notification about new data to a server of the phone provider (e.g. Microsoft or Apple push server), and this server sends the notification (as well as notificaiton from other online services) to your phone.
Some info for Windows Phone:
http://msdn.microsoft.com/en-us/library/hh221549.aspx
http://msdn.microsoft.com/en-us/library/windowsphone/develop/ff402558%28v=vs.105%29.aspx
Depending on how often you have new data both approaches can make sense.
WebSockets could be the answer: http://en.wikipedia.org/wiki/WebSocket
Specifically, for Windows Phone, there's a solution also: http://msdn.microsoft.com/en-us/library/windowsphone/develop/ff402558(v=vs.105).aspx