Python timed scheduler - scheduling

Okay, so I'm working on a scheduler and I was thinking of something like, timeOut(3,print,'hello') and it would print hello every three seconds, I have tried some methods but all failed. Also Using time.sleep for this wouldn't quite work because I need to run other tasks as well besides just one
Edit:
I found out how to do what I needed, sorry for being confusing but this did the trick for what I needed, thanks for answering everyone.
class test:
def __init__(self):
self.objectives = set()
class Objective:
pass
def interval(self,timeout,function,*data):
newObjective = self.Objective()
newObjective.Class = self
newObjective.timeout = time.time()+timeout
newObjective.timer = timeout
newObjective.function = function
newObjective.repeate = True
newObjective.data = data
self.objectives.add(newObjective)
return True
def runObjectives(self):
timeNow = time.time()
for objective in self.objectives:
timeout = objective.timer
if objective.timeout <= timeNow:
objective.function(*objective.data)
if objective.repeate:
objective.timeout = timeNow + timeout
self.main()
else:
self.objectives.remove(objective)
print('removed')
def main(self):
while True:
self.runObjectives()

The standard library includes a module called sched for scheduling. It can be adapted to work in a variety of environments using the delayfunc constructor parameter. Using it your question would likely read:
def event():
scheduler.enter(3, 0, event, ()) # reschedule
print('hello')
Now it depends on how you run the other tasks. Are you using an event loop? It probably has a similar scheduling mechanism (at least twisted has callLater and GObject has timeout_add). If all else fails, you can spawn a new thread and execute a sched.scheduler with time.sleep there.

Related

How to schedule a celery task without blocking Django

I have a Django service that register lot of clients and render a payload containing a timer (lets say 800s) after which the client should be suspended by the service (Change status REGISTERED to SUSPENDED in MongoDB)
I'm running celery with rabbitmq as broker as follows:
celery/tasks.py
#app.task(bind=True, name='suspend_nf')
def suspend_nf(pk):
collection.update_one({'instanceId': str(pk)},
{'$set': {'nfStatus': 'SUSPENDED'}})
and calling the task inside Django view like:
api/views.py
def put(self, request, pk):
now = datetime.datetime.now(tz=pytz.timezone(TIME_ZONE))
timer = now + datetime.timedelta(seconds=response_data["heartBeatTimer"])
suspend_nf.apply_async(eta=timer)
response = Response(data=response_data, status=status.HTTP_202_ACCEPTED)
response['Location'] = str(request.build_absolute_uri())
What am I missing here?
Are you asking that your view blocks totally or view is waiting the "ETA" to complete the execution?
Did you receive any error?
Try using countdown parameter instead of eta.
In your case it's better because you don't need to manipulate dates.
Like this: suspend_nf.apply_async(countdown=response_data["heartBeatTimer"])
Let's see if your view will have some different behavior.
I have finally find a work around, since working on a small project, I don't really need Celery + rabbitmq a simple Threading does the job.
Task look like this :
def suspend_nf(pk, timer):
time.sleep(timer)
collection.update_one({'instanceId': str(pk)},
{'$set': {'nfStatus': 'SUSPENDED'}})
And calling inside the view like :
timer = int(response_data["heartBeatTimer"])
thread = threading.Thread(target=suspend_nf, args=(pk, timer), kwargs={})
thread.setDaemon(True)
thread.start()

Python: multiprocessing - improving function time 6x didn't impact the performance of the loop

someone?
I have a function:
def _process_sensor(self, last_value, raw_data, raw_string, tariff_data, counter, electricity):
raw_data = OrderedDict(sorted(raw_data.items(), key=lambda t: iso8601.parse_date(t[0])))
interval_data = self._transform_cumulative_to_interval(last_value, raw_data, electricity)
interval_data = self._calculate_sensor_commodities(interval_data, tariff_data, tariff_data['ID'], electricity)
self._persist_results(raw_string, interval_data, tariff_data['ID'], electricity)
And yesterday each function execution took 2s, today I improved it to take around 0.25s and I am very happy about that but when I call my function:
from multiprocessing.pool import ThreadPool as Pool
pool = Pool(processes=8)
for sensor_id in today_values:
try:
pool.apply(self._process_sensor, (yesterday_values[sensor_id], today_values[sensor_id],
raw_string[sensor_id], self.electricity_tariff_data[sensor_id],
processed, True, ))
except Exception as e:
self.logger.warning('Some data error: {}'.format(e))
processed += 1
Looping through 100 elements takes the same amount of time: about 24 seconds. What can I be doing wrong? Parameters passed are parts of dictionaries. self._persist_results calls AWS from s3transfer.manager import TransferManager.
Edit: I know I have an 8 core box I'm running the code on. And did a pool.apply_async with same results.
If a single run takes 0.25s and 100 take 24s, it sounds like the function is running into contention, so that most of its run time isn't executing in parallel. This is quite common when there's contention around I/O resources, so my guess would be that _persist_results call is the culprit.
Looking into TransferManager, it seems there's a max_concurrency setting on the [TransferConfig][1] - looks like that should default to 10, any chance you've reduced that? I'd suggest checking if that's being set, and if not, see if setting it explicitly helps.
Fixed it:
from multiprocessing.pool import Pool
def _looop(dict_obj):
_process_sensor(**dict_obj)
to_process = []
for sensor_id in today_values:
to_process.append(copy.deepcopy({'fields': True,
'for': True,
'_process_sensor': True}))
pool.map_async(_looop, to_process)
Didn't have time to check if copy.deepcopy is necessary (and to be honest I don't need to put it in a separate loop - I will refactor that soon).

Twisted Python for responding multiple clients at a time

I have a problem. I'm having a echoserver which will accept clients and process his requirement and it returns the result to client.
Suppose I have two clients and 1 client requirement processing time would be 10 sec and 2 client requirement processing time would be 1 sec.
So when both clients connected to server at a time. how to run both the clients tasks at a time parallely and return the response to specific client which ever finishes first.
I have read that we can achieve this problem using python twisted. I have tried my luck, but Im unable to do it.
Please help me out of this Issue
Your code (https://trinket.io/python/87fd18ca9e) has many mistakes in terms of async design patterns, but I will only address the most blatant mistake. There are a few calls to time.sleep(), this is blocking code and is causing your code to stop until the sleep function is done running. The number 1 rule it async programming is do not use blocking functions! Don't worry, this is a very common mistake and the Twisted and Python async communities are there to help you :) I'll give you a naive solution for your server:
from twisted.internet.protocol import Factory
from twisted.internet import reactor, protocol, defer, task
def sleep(n):
return task.deferLater(reactor, n, lambda: None)
class QuoteProtocol(protocol.Protocol):
def __init__(self, factory):
self.factory = factory
def connectionMade(self):
self.factory.numConnections += 1
#defer.inlineCallbacks
def recur_factorial(self,n):
fact=1
print(n)
for i in range(1,int(n)+1):
fact=fact*i
yield sleep(5) # async sleep
defer.returnValue(str(fact))
def dataReceived(self, data):
try:
number = int(data) # validate data is an int
except ValueError:
self.transport.write('Invalid input!')
return # "exit" otherwise
# use Deferreds to write to client after calculation is finished
deferred_factorial = self.recur_factorial(number)
deferred_factorial.addCallback(self.transport.write)
def connectionLost(self, reason):
self.factory.numConnections -= 1
class QuoteFactory(Factory):
numConnections = 0
def buildProtocol(self, addr):
return QuoteProtocol(self)
reactor.listenTCP(8000, QuoteFactory())
reactor.run()
The main differences are in recur_factorial() and dataReceived(). The recur_factorial() is now utilizing Deferred (search how inlineCallbacks or coroutine's works) which allows for functions to execute after the result is available. So when the data in received, the factorial is calculated, then written to the end user. Finally there's the new sleep() function which allows for an async sleep function. I hope this helps. Keep reading the Krondo blog.

how to make flask pass a generator to task such as celery

I have a bunch of code that I have working in flask correctly, but these requests can take over 30 minutes to finish. I am using chained generators to use my existing code with yields to return to the browser.
Since these tasks take 30 minutes or more to complete, I want to offload these tasks but at am a loss. I have not succesfully gotten celery/rabbitmq/redis or any other combination to work correctly and am looking for how I can accomplish this so my page returns right away and I can check if the task is complete in the background.
Here is example code that works for now but takes 4 seconds of processing for the page to return.
I am looking for advice on how to get around this problem, can celery/redis or rabbitmq deal with generators like this? should I be looking at a different solution?
Thanks!
import time
import flask
from itertools import chain
class TestClass(object):
def __init__(self):
self.a=4
def first_generator(self):
b = self.a + 2
yield str(self.a) + '\n'
time.sleep(1)
yield str(b) + '\n'
def second_generator(self):
time.sleep(1)
yield '5\n'
def third_generator(self):
time.sleep(1)
yield '6\n'
def application(self):
return chain(tc.first_generator(),
tc.second_generator(),
tc.third_generator())
tc = TestClass()
app = flask.Flask(__name__)
#app.route('/')
def process():
return flask.Response(tc.application(), mimetype='text/plain')
if __name__ == "__main__":
app.run(host="0.0.0.0", port=5000, debug=True)
Firstly, it's not clear what it would even mean to "pass a generator to Celery". The whole point of Celery is that is not directly linked to your app: it's a completely separate thing, maybe even running on a separate machine, to which you would pass some fixed data. You can of course pass the initial parameters and get Celery itself to call the functions that create the generators for processing, but you can't drip-feed data to Celery.
Secondly, this is not at all an appropriate use for Celery in any case. Celery is for offline processing. You can't get it to return stuff to a waiting request. The only thing you could do would be to get it to save the results somewhere accessible by Flask, and then get your template to fire an Ajax request to get those results when they are available.

How to execute tasks one per second?

I have this:
eta_date = date.today()
eta = datetime.combine(eta_date, time.max)
scheduled_task.apply_async(eta=eta)
scheduled_tasks:
#task
def scheduled_task():
for obj in ModelData.objects.all():
send_data(obj)
send_data function sends object to other server as JSON. I use Celery. I want to start task on end of the day but in such a way that one of the objects is sent once per second. How to do it?
allcaps already told you the answer in the comment section, but it's what I would have answered anyway. Just add a sleep after send_data to wait X seconds.
import time
#task
def scheduled_task():
for obj in ModelData.objects.all():
send_data(obj)
time.sleep(1) # You can also use a float here if 1 second is too long
Another option could be to spawn a task per obj in ModelData and set a limit on it to 1s.
#task
def scheduled_task():
for obj in ModelData.objects.all():
send_data_task.delay(obj)
#task(rate_limit='1/s')
def send_data_task(obj):
send_data(obj)