I am a novice user of a cluster running in RedHat Enterprise Linux. I run python script (version 2.6.5) by using bsub command. Somehow this python program just stops during the multiprocessing. The program goes like:
from multiprocessing import Pool
import multiprocessing
def pop_genomics(chrom):
os.system('run analysis on DNA')
os.system('run analysis on DNA')
os.system('run analysis on DNA')
os.system('run analysis on DNA')
print 'Finished!'
return 'Done'
pool = multiprocessing.Pool(multiprocessing.cpu_count())
finalfiledirs=pool.map(pop_genomics, chroms)
pool.close()
pool.join()
I get 'Finished!' message from all workers, but this program does not proceed beyond that 'finalfiledirs=pool.map(pop_genomics, chroms)' line. Can you suggest why this is happening?
You should be getting an error on that line because on the
pool.map(pop_genomics,chroms)
you never pass any parameters to pop_genomics, so you need to add some so it would be:
pool.map(pop_genomics(parameters),chroms)
Related
I have written a simple c program with a bufferoveflow. It is basically a game to guess 4 digits number but starts by asking players to enter their name and this is where buffer overflow happens...I have written an exploit to basically inject shellcode when the "Please enter your name" When I run it without program attached to the immunity debugger it works fine but when I attach the exe file to the immunity debugger python script does noting as it is not something that is running on the debugger.....so basically nothing happens when I execute the code. Python code is below:
import sys, struct, os
import subprocess
import time
from subprocess import Popen, PIPE
location ='C:\Users\ZEIT8042\Desktop\Guessthenumber\guess.exe
p= Popen([location],stdin=PIPE,stdout=PIPE,stderr=PIPE)
time.sleep(15) #tried this to make the program stall for 15 seconds so that it can be attached to immunity debugger.
junk='A'*40
o,e= p.communicate(input=junk)
print(o)
What I am trying to do is check if the program is running...if it is running then inject the shellcode when the exe asks for the name.....any help would be appreciated...
elif is used in multiple conditions that is seen wrong.is this wrong meaning
I have multiple python scripts, each with print statements and prompts for input. I run these scripts from a single python script as below.
os.system('python script1.py ' + sys.argv[1])
os.system('python script2.py ' + sys.argv[1]).....
The run is completed successfully, however, when I run all the scripts from a single file, I no longer see any print statements or prompts for input on the run console. Have researched and attempted many different ways to get this to work w/o success. Help would be much appreciated. Thanks.
If I understand correctly you want to run multiple python scripts synchronously, i.e. one after another.
You could use a bash script instead of python, but to answer your question of starting them from python...
Checkout out the subprocess module: https://docs.python.org/3.4/library/subprocess.html
In particular the call method, it accepts a stdin and stdout which you can pass sys.stdin and sys.stdout to.
import sys
import subprocess
subprocess.call(['python', 'script1.py', sys.argv[1]], stdin=sys.stdin, stdout=sys.stdout)
subprocess.call(['python', 'script2.py', sys.argv[1]], stdin=sys.stdin, stdout=sys.stdout)
^
This will work in python 2.7 and 3, another way of doing this is by importing your file (module) and calling the methods in it. The difference here is that you're no longer running the code in a separate process.
subroutine.py
def run_subroutine():
name = input('Enter a name: ')
print(name)
master.py
import subroutine
subroutine.run_subroutine()
Trying to get a project I'm working on to wait on the results of the Scrapy crawls. Pretty new to Python but I'm learning quickly and I have liked it thus far. Here's my remedial function to refresh my crawls;
def refreshCrawls():
os.system('rm JSON/*.json)
os.system('scrapy crawl TeamGameResults -o JSON/TeamGameResults.json --nolog')
#I do this same call for 4 other crawls also
This function gets called in a for loop in my 'main function' while I'm parsing args:
for i in xrange(1,len(sys.argv)):
arg = sys.argv[i]
if arg == '-r':
pprint('Refreshing Data...')
refreshCrawls()
This all works and does update the JSON files, however the rest of my application does not wait on this as I foolishly expected it to. Didn't really have a problem with this until I moved the app over to a Pi and now the poor little guy can't refresh soon enough, Any suggestions on how to resolve this?
My quick dirty answer says split it into a different automated script and just run it an hour or so before I run my automated 'main function,' or use a sleep timer but I'd rather go about this properly if there's some low hanging fruit that can solve this for me. I do like being able to enter the refresh arg in my command line.
Instead of using os use subprocess:
from subprocess import Popen
import shlex
def refreshCrawls():
os.system('rm JSON/*.json')
cmd = shlex.split('scrapy crawl TeamGameResults -o JSON/TeamGameResults.json --nolog')
p = Popen(cmd)
#I do this same call for 4 other crawls also
p.wait()
for i in xrange(1,len(sys.argv)):
arg = sys.argv[i]
if arg == '-r':
pprint('Refreshing Data...')
refreshCrawls()
OK so I am trying to run one python script (test1.py) and print something into a webpage at the end. I also want a subprocess (test2.py) to begin during this script. However, as it happens, test2.py is going to take longer to execute than test1. The problem that I am having is that test1.py is being held up until test2 completes. I have seen numerous people post similar questions but none of the solutions I've seen have fixed my issue.
Here's a heavily simplified version of my code that demonstrates the issue. I am running it on a local server, and displaying it in firefox.
test1.py:
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
import cgi
import cgitb; cgitb.enable()
import subprocess
form = cgi.FieldStorage()
print("Content-Type: text/html") # HTTP header to say HTML is following
print # blank line, end of headers
p = subprocess.Popen(['python','test2.py'])
print "script1 ended"
test2.py:
from time import sleep
print "script2 sleeping...."
sleep(60)
print "script2 ended"
Essentially I want to execute test1.py and have it say "script1 ended" in firefox, without having to wait 60 seconds for the subprocess to exit. I don't want anything from the test2 subprocess to show up.
EDIT: For anyone interested.. I solved the problem in windows by using os.startfile(), and as it turns out subprocess.Popen([...], stderr=subprocess.STDOUT, stdout=subprocess.PIPE) works in Linux. The scripts run in the background without delaying anything in the parent script when I do a wget on it.
I have been having problems trying to run an external program from a python program that was generated from a trigger in a postgres 9.2 database. The trigger works. It writes to a file. I had tried just running the external program but the permissions would not allow it to run. I was able to create a folder (using os.system(“mkdir”) ). The owner of the folder is NETWORK SERVICE.
I need to run a program called sdktest. When I try to run it no response happens so I think that means that the python program does not have enough permissions (with an owner of NETWORK SERVICE) to run it.
I have been having my program copy files that it needs into that directory so they would have the correct permissions and that has worked to some degree but the program that I need to run is the last one and it is not running because it does not have enough permissions.
My python program runs a C++ program called PG_QB_Connector which calls sdktest.
Is there any way I can change the owner of the process to be a “normal” owner? Is there a better way to do this? Basically I just need to have this C++ program have eniough perms to run correctly.
BTW, when I run the C++ program by hand, the line that runs the sdktest program runs correctly, however, when I run it from the postgres/python it does not do anything...
I have Windows 7, python 3.2. The other 2 questions that I asked about this are located here and here
The python program:
CREATE or replace FUNCTION scalesmyone (thename text)
RETURNS int
AS $$
a=5
f = open('C:\\JUNK\\frompython.txt','w')
f.write(thename)
f.close()
import os
os.system('"mkdir C:\\TEMPWITHOWNER"')
os.system('"mkdir C:\\TEMPWITHOWNER\\addcustomer"')
os.system('"copy C:\\JUNK\\junk.txt C:\\TEMPWITHOWNER\\addcustomer"')
os.system('"copy C:\\BATfiles\\junk6.txt C:\\TEMPWITHOWNER\\addcustomer"')
os.system('"copy C:\\BATfiles\\run_addcust.bat C:\\TEMPWITHOWNER\\addcustomer"')
os.system('"copy C:\\Workfiles\\PG_QB_Connector.exe C:\\TEMPWITHOWNER\\addcustomer"')
os.system('"copy C:\\Workfiles\\sdktest.exe C:\\TEMPWITHOWNER\\addcustomer"')
import subprocess
return_code = subprocess.call(["C:\\TEMPWITHOWNER\\addcustomer\\PG_QB_Connector.exe", '"hello"'])
$$ LANGUAGE plpython3u;
The C++ program that is called from the python program and calls sdktest.exe is below
command = "copy C:\\Workfiles\\AddCustomerFROMWEB.xml C:\\TEMPWITHOWNER\\addcustomer\\AddCustomerFROMWEB.xml";
system(command.c_str());
//everything except for the qb file is in my local folder
command = "C:\\TEMPWITHOWNER\\addcustomer\\sdktest.exe \"C:\\Users\\Public\\Documents\\Intuit\\QuickBooks\\Company Files\\Shain Software.qbw\" C:\\TEMPWITHOWNER\\addcustomer\\AddCustomerFROMWEB.xml C:\\TEMPWITHOWNER\\addcustomer\\outputfromsdktestofaddcust.xml";
system(command.c_str());
It sounds like you want to invoke a command-line program from within a PostgreSQL trigger or function.
A usually-better alternative is to have the trigger send a NOTIFY and have a process with a PostgreSQL connection LISTENing for notifications. When a notification comes in, the process can start your program. This is the approach I would recommend; it's a lot cleaner and it means your program doesn't have to run under PostgreSQL's user ID. See NOTIFY and LISTEN.
If you really need to run commands from inside Pg:
You can use PL/Pythonu with os.system or subprocess.check_call; PL/Perlu with system(); etc. All these can run commands from inside Pg if you need to. You can't invoke programs directly from PostgreSQL, you need to use one of the 'untrusted' (meaning fully privileged, not sandboxed) procedural languages to invoke external executables. PL/TCL can probably do it too.
Update:
Your Python code as shown above has several problems:
Using os.system in Python to copy files is just wrong. Use the shutil library: http://docs.python.org/3/library/shutil.html to copy files, and the simple os.mkdir command to create directories.
The double-layered quoting looks wrong; didn't you mean to quote only each argument not the whole command? You should be using subprocess.call instead of os.system anyway.
Your final subprocess.call invocation appears OK, but fails to check the error code so you'll never know if it went wrong; you should use subprocess.check_call instead.
The C++ code also appears to fail to check for errors from the system() invocations so you'll never know if the command it runs fails.
Like the Python code, copying files in C++ by using the copy shell command is generally wrong. Microsoft Windows provides the CopyFile function for this; equivalents or alternatives exist on other platforms and you can use portable-but-less-efficient stream copying too.