I met a problem when create stream video from Django server.
generator_list = []
#gzip.gzip_page
def get_stream_video(request, stream_link):
if len(generator_list) == 2:
generator_list[1].close()
del(generator_list[1])
try:
generator = detect_stream(stream_link)
generator_list.append(generator)
stream = StreamingHttpResponse(generator, content_type="multipart/x-mixed-replace;boundary=frame")
return stream
except:
pass
Everytime I call this controller from client, it will create new instance of connection. Then I found that if those streaming response not closed, CPU will load more and more. I try to save stream generator and close but it did not work.
Related
REST API service has a limit of requests (say a maximum of 100 requests per minute). In Django, I am trying to allow USERs to access such API and retrieve data in real-time to update SQL tables. Therefore there is a problem that if multiple users are trying to access the API, the limit of requests is likely to be exceeded.
Here is a code snippet as an example of how I currently perform requests - each user will add a list of objects he wants to request and run request_engine().start(object_list) to access the API. I use multithreading to speed up requests. I also allow retrying failed API requests via setting a limit on the number of requests for each request object upper_limit.
As I understand there should be some queue for API requests. I anticipate there must be a more elegant solution for this, however, I could not find any similar examples. How can one implement/rewrite this for multiUSER usage with Django?
import requests
from multiprocessing.dummy import Pool as ThreadPool
N=50 # number of threads
upper_limit=1 # limit on the number of requests for a single object
class request_engine():
def __init__(self):
pass
def start(self,objs):
self.objs={obj:{'status':0,'data':None} for obj in objs}
done=False
while not done:
self.parallel_requests()
done=all(_['status']>upper_limit or _['status']==-1 for obj,_ in self.objs.items())
return dict(self.objs)
def single_request(self,request_obj):
URL = f"https://reqres.in/api/users?page={request_obj}"
r = requests.get(url = URL)
if r.ok:
res = r.json()
self.objs[request_obj]['status']=-1
self.objs[request_obj]['data']=res
else:
self.objs[request_obj]['status']+=1
def parallel_requests(self):
objs=[obj for obj,_ in self.objs.items() if _['status']!=-1 and _['status']<=upper_limit]
pool = ThreadPool(N)
pool.map(self.single_request, objs)
pool.close()
pool.join()
objs=[1,2,3,4,5,6,7,7,8,234,124,24,535,6,234,24,4,1,3,4,5,4,3,5,3,1,5,2,3,5,3]
result=request_engine().start(objs)
print([_['status'] for obj,_ in result.items()])
# status corresponds to the number of unsuccessful requests
# status=-1 implies success of the request
Thanks in advance.
I have two containers (docker) running the application and I'm trying to redirect the request from one of the container to another. The container where I'm redirecting from has this code and the 172.17.0.3 is the IP of the second container. I have seen that it can be pinged. In the other container I don't have the else part and no if condition check. When I run a curl request to this container using another client container in the same network curl http://172.17.0.2:3333?count=100, it should ideally redirect but I get Internal Server Error as the response. However, when I login to the container 2 and run curl, I get redirected to ... response.
from flask_restful import Resource, Api
app = Flask(__name__)
api = Api(app)
class Greeting (Resource):
def get(self):
offload = True
if offload == False:
count = request.args.get('count')
count = int(count)
for i in range(count):
continue
return count
else:
count = request.args.get('count')
redirect_str = "http://172.17.0.3:3333?count=" + count
return redirect(redirect_str, code=302)
api.add_resource(Greeting, '/') # Route_1
if __name__ == '__main__':
app.run('0.0.0.0', '3333')
I want to be able to wait for the response back from the server 172.17.0.3 and once I receive the message, send the response back to the client. Can anyone tell me how it can be done?
You need to send a request to the other container instead of redirect to it if you want to wait for its response. Using e.g. the requests library this would look something like
import requests
resp = requests.get('http://172.17.0.3:3333?count=' + count)
return resp.text
Check request's Quickstart guide for more info: https://requests.readthedocs.io/en/master/user/quickstart.
I am implementing server sent events using flask. If I use time.sleep inside my function, the sse doesn't return anything and the request stays as pending in the browser. If I don't use sleep, there would be overload of responses in the browser, so I need to use some delay. Why is time.sleep blocking the request? Is there another way I can add time delay here?
def get_message():
time.sleep(1.0)
s="xyz" #some function here for our business logic
return s
#app.route('/stream')
def stream():
def eventStream():
while True:
yield 'data: {}\n\n'.format(get_message())
return Response(eventStream(), mimetype="text/event-stream")
I have to fetch the retweets for the tweets and create the JSON file with retweets,user id etc using the python script. Kindly help me to sort it our this issues.
Thanks in advance!!
This task require some fields of knowledge, and since you ask in a general way, I reckon you need a script to run immediately, but setting up this process requires sometime
This part to get connect to twitter API
from twython import Twython, TwythonError
APP_KEY = 'YOUR_APP_KEY'
APP_SECRET = 'YOUR_APP_SECRET'
twitter = Twython(APP_KEY, APP_SECRET)
Use Twitter API call from Twython,
you can find a list here https://twython.readthedocs.io/en/latest/api.html, the param is the same as twitter API
response = twitter.get_retweets(id, 100)
Pagnation
each call to API have limit of returns, in example for engine.get_friends_ids was limited to 5000 (https://dev.twitter.com/rest/reference/get/friends/ids), if you want to get more than 5000, you have to use the cursor in the returned result (if cur = 0 in json returned means no more results), following is example of how to handling cursor
#Set a temp to loop
cur = -1
#Stop when no more result
while cur !=0:
response = twitter.get_friends_ids(user_id=user_id, cursor=cur)
#Some code to handle the response
cur = response["next_cursor"]
API key
Key expires after some calls (https://dev.twitter.com/rest/public/rate-limits), so you need to set some code to auto change your key, or wait for some period (key reached limit return error code 429)
Response
The response from API was in JSON format, which was easy to use, you can access data by selecting base on response[key], in example
reponse["ids"] or response["next_cursor"]
I'll first explain the architecture of my system and then move to the question:
I have a REST API which is used as my API gateway. This server is build using Flask. I also have RabbitMQ cluster, and a client I wrote that listens to a specific queue and executes the tasks its getting.
Until now, all of my requests were asynchronous, so once a request has reached to the API gateway, a callback_uri field with URL to POST the results to provided as part of the request, and the API gateway was just responsible for sending the task to RabbitMQ and the worker processed the task, and at the end POSTed the results back to the callback URL.
My question is:
I want a new endpoint to be synchronous in the sense of, that the processing will be done still by the same worker as before, but I want to get the results back to the API gateway to return to the user and release the connection.
My current solution:
I'm sending a unique callback_uri as part of the request to the worker as before, but now the specific endpoint is implemented by my API gateway and allow both POST and GET methods, so the worker can POST the results once it finished, and my API gateway keeps polling the callback URL until a result is available and then return the result to the client.
Is there any other preferred option other than having a busy-waiting HTTP worker polling its own endpoint to get the results? but still be synchronous so the connection released only when the results become available?
Code for illustration only:
#app.route('/long_task', methods=['POST'])
#sync_request
def long_task():
try:
if request.get_json() is None:
return ERROR_MSG_NO_JSON, 400
create_and_send_request_to_rabbitmq()
return '', 200
except Exception as ex:
return ERROR_MSG_NO_DATA, 400
def sync_request(func):
def call(*args, **kwargs):
create_callback_uri()
result = func(*args, **kwargs)
status_code = result[1]
if status_code == 200:
result = get_callback_result()
return result
return call
def get_callback_result():
callback_uri = request.get_json()['callback_uri']
has_answer = False
headers = {'content-type': 'application/json'}
empty_response = {}
content = json.dumps(empty_response)
try:
with Timeout(seconds=SYNC_REQUEST_TIMEOUT_SECONDS):
while not has_answer:
response = requests.get(callback_uri, headers=headers)
if response.status_code == 200:
has_answer = True
content = response.content
else:
time.sleep(0.2)
except TimeoutException:
log.debug('Timed out on sync request for request %s ' % request)
return content, 200
So if I understand you correctly you want your backend to wait for the response from some worker (via RabbitMQ). You can achieve that by implementing rpc over rabbitmq. The key idea is to use the correlation id.
But definitely the most efficient way would be to run the client over websockets (or raw tcp socket if it is not a browser) and notify him directly when the job is done. That way you don't lock resources (client connection, rabbitmq queues) and you avoid performance hit (rpc).