This may be a strange request. I have an infinite While loop and each loop lasts ~7 minutes, then the program sleeps for a couple minutes to let the computer cool down, and then starts over.
This is how it looks:
import time as t
t_cooling = 120
while True:
try:
#7 minutes of uninterrupted calculations here
t.sleep(t_cooling)
except KeyboardInterrupt:
break
Right now if I want to interrupt the process, I have to wait until the program sleeps for 2 minutes, otherwise all the calculations done in the running cycle are wasted. Moreover the calculations involve writing on files and working with multiprocessing, so interrupting during the calculation phase is not only a waste, but can potentially damage the output on the files.
I'd like to know if there is a way to signal to the program that the current cycle is the last one it has to execute, so that there is no risk of interrupting at the wrong moment. To add one more limitation, it has to be a solution that works via command line. It's not possible to add a window with a stop button on the computer the program is running on. The machine has a basic Linux installation, with no graphical environment. The computer is not particularly powerful or new and I need to use the most CPU and RAM possible.
Hope everything is clear enough.
Not so elegant, but it works
#!/usr/bin/env python
import signal
import time as t
stop = False
def signal_handler(signal, frame):
print('You pressed Ctrl+C!')
global stop
stop = True
signal.signal(signal.SIGINT, signal_handler)
print('Press Ctrl+C')
t_cooling = 1
while not stop:
t.sleep(t_cooling)
print('Looping')
You can use a separate Thread and an Event to signal the exit request to the main thread:
import time
import threading
evt = threading.Event()
def input_thread():
while True:
if input("") == "quit":
evt.set()
print("Exit requested")
break
threading.Thread(target=input_thread).start()
t_cooling = 5
while True:
#7 minutes of uninterrupted calculations here
print("starting calculation")
time.sleep(5)
if evt.is_set():
print("exiting")
break
print("cooldown...")
time.sleep(t_cooling)
Just for completeness, I post here my solution. It's very raw, but it works.
import time as t
t_cooling = 120
while True:
#7 minutes of uninterrupted calculations here
f = open('stop', 'r')
stop = f.readline().strip()
f.close()
if stop == '0':
t.sleep(t_cooling)
else:
break
I just have to create a file named stop and write a 0 in it. When that 0 is changed to something else, the program stops at the end of the cycle.
Related
I have sript for 3 lcd display on my rassbery pi, and have problem with
time sleep function.
I would like to have a pause only for screen 3 ( after 10s show different data ), and the other LCD should run normaly, without time sleep - pause
example :
# LCD1
mylcd1.lcd_display_string("TEMP1:",1,0)
mylcd1.lcd_display_string(str(temp1),1,5)
mylcd1.lcd_display_string ("%s" %time.strftime("%H:%M:%S"),3, 0)
# LCD2
mylcd2.lcd_display_string("TEMP2:",1,0)
mylcd2.lcd_display_string(str(temp2),1,5)
# LCD3
mylcd3.lcd_clear()
mylcd3.lcd_display_string("TEMP3:",1,0)
mylcd3.lcd_display_string(str(temp3),1,5)
time.sleep(10)
mylcd3.lcd_clear()
mylcd3.lcd_display_string("DIM:",1,0)
mylcd3.lcd_display_string(str(dp1),1,4)
In this example is problem time in LCD1, It does not run smoothly, it has to wait 10s, and temperature data on LCD 1 and LCD 2 it must be refreshed in real time without delay....
Thank you for help!
I would say an easy way to go about this is have your LCD's that need to be updated running in a loop, and have a variable that is keeping track of the time.
Then within your loop have an if statement checking if the time % 10 seconds == 0 then run the refresh of the Screen that needs the delay.
I forgot to write .... the compete code is already in loop...
I want add time.sleep only for part for LCD3 ...
LCD3 change data every 10 sec.
I am using multiprocessing.Pool.imap to run many independent jobs in parallel using Python 2.7 on Windows 7. With the default settings, my total CPU usage is pegged at 100%, as measured by Windows Task Manager. This makes it impossible to do any other work while my code runs in the background.
I've tried limiting the number of processes to be the number of CPUs minus 1, as described in How to limit the number of processors that Python uses:
pool = Pool(processes=max(multiprocessing.cpu_count()-1, 1)
for p in pool.imap(func, iterable):
...
This does reduce the total number of running processes. However, each process just takes up more cycles to make up for it. So my total CPU usage is still pegged at 100%.
Is there a way to directly limit the total CPU usage - NOT just the number of processes - or failing that, is there any workaround?
The solution depends on what you want to do. Here are a few options:
Lower priorities of processes
You can nice the subprocesses. This way, though they will still eat 100% of the CPU, when you start other applications, the OS gives preference to the other applications. If you want to leave a work intensive computation run on the background of your laptop and don't care about the CPU fan running all the time, then setting the nice value with psutils is your solution. This script is a test script which runs on all cores for enough time so you can see how it behaves.
from multiprocessing import Pool, cpu_count
import math
import psutil
import os
def f(i):
return math.sqrt(i)
def limit_cpu():
"is called at every process start"
p = psutil.Process(os.getpid())
# set to lowest priority, this is windows only, on Unix use ps.nice(19)
p.nice(psutil.BELOW_NORMAL_PRIORITY_CLASS)
if __name__ == '__main__':
# start "number of cores" processes
pool = Pool(None, limit_cpu)
for p in pool.imap(f, range(10**8)):
pass
The trick is that limit_cpu is run at the beginning of every process (see initializer argment in the doc). Whereas Unix has levels -19 (highest prio) to 19 (lowest prio), Windows has a few distinct levels for giving priority. BELOW_NORMAL_PRIORITY_CLASS probably fits your requirements best, there is also IDLE_PRIORITY_CLASS which says Windows to run your process only when the system is idle.
You can view the priority if you switch to detail mode in Task Manager and right click on the process:
Lower number of processes
Although you have rejected this option it still might be a good option: Say you limit the number of subprocesses to half the cpu cores using pool = Pool(max(cpu_count()//2, 1)) then the OS initially runs those processes on half the cpu cores, while the others stay idle or just run the other applications currently running. After a short time, the OS reschedules the processes and might move them to other cpu cores etc. Both Windows as Unix based systems behave this way.
Windows: Running 2 processes on 4 cores:
OSX: Running 4 processes on 8 cores:
You see that both OS balance the process between the cores, although not evenly so you still see a few cores with higher percentages than others.
Sleep
If you absolutely want to go sure, that your processes never eat 100% of a certain core (e.g. if you want to prevent that the cpu fan goes up), then you can run sleep in your processing function:
from time import sleep
def f(i):
sleep(0.01)
return math.sqrt(i)
This makes the OS "schedule out" your process for 0.01 seconds for each computation and makes room for other applications. If there are no other applications, then the cpu core is idle, thus it will never go to 100%. You'll need to play around with different sleep durations, it will also vary from computer to computer you run it on. If you want to make it very sophisticated you could adapt the sleep depending on what cpu_times() reports.
On the OS level
you can use nice to set a priority to a single command. You could also start a python script with nice. (Below from: http://blog.scoutapp.com/articles/2014/11/04/restricting-process-cpu-usage-using-nice-cpulimit-and-cgroups)
nice
The nice command tweaks the priority level of a process so that it runs less frequently. This is useful when you need to run a
CPU intensive task as a background or batch job. The niceness level
ranges from -20 (most favorable scheduling) to 19 (least favorable).
Processes on Linux are started with a niceness of 0 by default. The
nice command (without any additional parameters) will start a process
with a niceness of 10. At that level the scheduler will see it as a
lower priority task and give it less CPU resources.Start two
matho-primes tasks, one with nice and one without:
nice matho-primes 0 9999999999 > /dev/null &matho-primes 0 9999999999 > /dev/null &
matho-primes 0 9999999999 > /dev/null &
Now run top.
As a function in Python
Another approach is to use psutils to check your CPU load average for the past minute and then have your threads check the CPU load average and spool up another thread if you are below the specified CPU load target, and sleep or kill the thread if you are above the CPU load target. This will get out of your way when you are using your computer, but will maintain a constant CPU load.
# Import Python modules
import time
import os
import multiprocessing
import psutil
import math
from random import randint
# Main task function
def main_process(item_queue, args_array):
# Go through each link in the array passed in.
while not item_queue.empty():
# Get the next item in the queue
item = item_queue.get()
# Create a random number to simulate threads that
# are not all going to be the same
randomizer = randint(100, 100000)
for i in range(randomizer):
algo_seed = math.sqrt(math.sqrt(i * randomizer) % randomizer)
# Check if the thread should continue based on current load balance
if spool_down_load_balance():
print "Process " + str(os.getpid()) + " saying goodnight..."
break
# This function will build a queue and
def start_thread_process(queue_pile, args_array):
# Create a Queue to hold link pile and share between threads
item_queue = multiprocessing.Queue()
# Put all the initial items into the queue
for item in queue_pile:
item_queue.put(item)
# Append the load balancer thread to the loop
load_balance_process = multiprocessing.Process(target=spool_up_load_balance, args=(item_queue, args_array))
# Loop through and start all processes
load_balance_process.start()
# This .join() function prevents the script from progressing further.
load_balance_process.join()
# Spool down the thread balance when load is too high
def spool_down_load_balance():
# Get the count of CPU cores
core_count = psutil.cpu_count()
# Calulate the short term load average of past minute
one_minute_load_average = os.getloadavg()[0] / core_count
# If load balance above the max return True to kill the process
if one_minute_load_average > args_array['cpu_target']:
print "-Unacceptable load balance detected. Killing process " + str(os.getpid()) + "..."
return True
# Load balancer thread function
def spool_up_load_balance(item_queue, args_array):
print "[Starting load balancer...]"
# Get the count of CPU cores
core_count = psutil.cpu_count()
# While there is still links in queue
while not item_queue.empty():
print "[Calculating load balance...]"
# Check the 1 minute average CPU load balance
# returns 1,5,15 minute load averages
one_minute_load_average = os.getloadavg()[0] / core_count
# If the load average much less than target, start a group of new threads
if one_minute_load_average < args_array['cpu_target'] / 2:
# Print message and log that load balancer is starting another thread
print "Starting another thread group due to low CPU load balance of: " + str(one_minute_load_average * 100) + "%"
time.sleep(5)
# Start another group of threads
for i in range(3):
start_new_thread = multiprocessing.Process(target=main_process,args=(item_queue, args_array))
start_new_thread.start()
# Allow the added threads to have an impact on the CPU balance
# before checking the one minute average again
time.sleep(20)
# If load average less than target start single thread
elif one_minute_load_average < args_array['cpu_target']:
# Print message and log that load balancer is starting another thread
print "Starting another single thread due to low CPU load balance of: " + str(one_minute_load_average * 100) + "%"
# Start another thread
start_new_thread = multiprocessing.Process(target=main_process,args=(item_queue, args_array))
start_new_thread.start()
# Allow the added threads to have an impact on the CPU balance
# before checking the one minute average again
time.sleep(20)
else:
# Print CPU load balance
print "Reporting stable CPU load balance: " + str(one_minute_load_average * 100) + "%"
# Sleep for another minute while
time.sleep(20)
if __name__=="__main__":
# Set the queue size
queue_size = 10000
# Define an arguments array to pass around all the values
args_array = {
# Set some initial CPU load values as a CPU usage goal
"cpu_target" : 0.60,
# When CPU load is significantly low, start this number
# of threads
"thread_group_size" : 3
}
# Create an array of fixed length to act as queue
queue_pile = list(range(queue_size))
# Set main process start time
start_time = time.time()
# Start the main process
start_thread_process(queue_pile, args_array)
print '[Finished processing the entire queue! Time consuming:{0} Time Finished: {1}]'.format(time.time() - start_time, time.strftime("%c"))
In Linux:
Use nice() with a numerical value:
#on Unix use ps.nice(10) for very low priority
p.nice(10)
https://en.wikipedia.org/wiki/Nice_(Unix)#:~:text=nice%20is%20a%20program%20found,CPU%20time%20than%20other%20processes.
I'm having a tough time trying to develop a threaded app wherein the threads are each doing REST calls over the network.
I need to kill active threads after a certain timeout. I've tried every python 2.7.x approach I've seen on here and can't get this working.
I'm using python 2.7.6 on OEL linux (3.8.13-44.1.1.el6uek.x86_64).
Here is a simplified snippet of code:
class cthread(threading.Thread):
def __init__(self, cfg, host):
self.cfg = cfg
self.host = host
self.runf = False
self.stop = threading.Event()
threading.Thread.__init__(self. target=self.collect)
def terminate(self):
self.stop.set()
def collect(self):
try:
self.runf = True
while (not self.stop.wait(1)):
# Here I do urllib2 GET request to a REST service which could hang
<rest call>
finally:
self.runf = False
timer_t1 = 0
newthr = cthread(cfg, host)
newthr.start()
while True:
if timer_t1 > 600:
newthr.terminate()
break
time.sleep(30)
timer_t1 += 30
Basically after my timeout period I need to kill all remaining threads, either gracefully or not.
Haven't a heck of a time getting this to work.
Am I going about this the correct way?
There's no official API to kill a thread in python.
In your code relying on urllib2 you might periodically pass timeout left for your threads to run from the main loop and use urllib2 with the timeout option. Or even track the timers in threads exploiting the same approach with urllib2.
I'm using web2py for a project and found that gevent.sleep seems to hang in unexpected disconnects. I'm guessing this is due to improperly handled exception. I can not find it properly written into the documentation, how do I catch, link, or monitor exceptions from gevent.sleep()?
Thank you in advance.
Strange guess, it might be wrong. sleep() suspends current Greenlet and resumes next, pending, Greenlet. Most likely it is next Geenlet that runs after sleep() that blocks execution.
If you don't see traceback printed out it is not coming from sleep().
Source code of sleep function:
def sleep(seconds=0):
"""Put the current greenlet to sleep for at least *seconds*.
*seconds* may be specified as an integer, or a float if fractional seconds
are desired.
If *seconds* is equal to or less than zero, yield control the other coroutines
without actually putting the process to sleep. The :class:`core.idle` watcher
with the highest priority is used to achieve that.
"""
hub = get_hub()
loop = hub.loop
if seconds <= 0:
watcher = loop.idle()
watcher.priority = loop.MAXPRI
else:
watcher = loop.timer(seconds)
hub.wait(watcher)
I'm looking for a way to set a time limit, a countdown. I will use this function to control the amount of time the user can run a while loop. Here's an example of how it's doing to run.
def run(strings, timeout=30):
while timeout > 0:
input = raw_input("prompts that will continue prompting as long as not timed out")
if input != strings:
#do something here, maybe be able to deduct off timer.
else:
score += 1
print "%d /r" % timeout #print timeout
return score
My example may not be very neat, but it's the general idea. The timeout will control the loop.
Can this be done for the timer library? Do I need a separate thread to handle the time while my function runs (prompting user)?