web.py running main twice, ignoring changes - python-2.7

I have a simple web.py app that reads a config file and serves to URL paths. However I get two strange behaviors. One, changes made to data in the Main are not reflected in the results of GET. Two, Main appears to run twice.
Desired behavior is modifying data in Main will cause methods to see modified data, and not having main re-run.
Questions:
What is really happening here, that mydict is not modified in either
GET.
Why am I getting some code running twice.
Simplest path to desired behavior (most important)
Pythonic path to desired behavior (least important)
From pbuck (Accepted Answer): Answer for 3.) is replace
app = web.application(urls, globals())
with:
app = web.application(urls, globals(), autoreload=False)
Same behavior on pythons Linux (CentOS 6 python 2.6.6) and MacBook (brew python 2.7.12)
When started I get:
$ python ./foo.py 8080
Initializing mydict
Modifying mydict
http://0.0.0.0:8080/
When queried with:
wget http://localhost:8080/node/first/foo
wget http://localhost:8080/node/second/bar
Which results in (notice a second "Initializing mydict"):
Initializing mydict
firstClass.GET called with clobber foo
firstClass.GET somevalue is something static
127.0.0.1:52480 - - [17/Feb/2017 17:30:42] "HTTP/1.1 GET /node/first/foo" - 200 OK
secondClass.GET called with clobber bar
secondClass.GET somevalue is something static
127.0.0.1:52486 - - [17/Feb/2017 17:30:47] "HTTP/1.1 GET /node/second/bar" - 200 OK
Code:
#!/usr/bin/python
import web
urls = (
'/node/first/(.*)', 'firstClass',
'/node/second/(.*)', 'secondClass'
)
# Initialize web server, start it later at "app . run ()"
#app = web.application(urls, globals())
# Running web.application in Main or above does not change behavior
# Static Initialize mydict
print "Initializing mydict"
mydict = {}
mydict['somevalue'] = "something static"
class firstClass:
def GET(self, globarg):
print "firstClass.GET called with clobber %s" % globarg
print "firstClass.GET somevalue is %s" % mydict['somevalue']
return mydict['somevalue']
class secondClass:
def GET(self, globarg):
print "secondClass.GET called with clobber %s" % globarg
print "secondClass.GET somevalue is %s" % mydict['somevalue']
return mydict['somevalue']
if __name__ == '__main__':
app = web.application(urls, globals())
# read configuration files for initializations here
print "Modifying mydict"
mydict['somevalue'] = "something dynamic"
app.run()

Short answer, avoid using globals as they don't do what you think they do. Especially when you eventually deploy this under nginx / apache where there will (likely) be multiple processes running.
Longer answer
Why am I getting some code running twice?
Code, global to app.py, is running twice because it runs once, as it normally does. The second time is within the web.application(urls, globals()) call. Really, that call to globals() sets up module loading / re-loading. Part of that is re-loading all modules (including app.py). If you set autoreload=False in the web.applications() call, it won't do that.
What is really happening here, that mydict is not modified in either GET?
mydict is getting set to 'something dynamic', but then being re-set to 'something static' on second load. Again, set autoreload=False and it will work as you expect.
Shortest path?
autoreload=False
Pythonic path?
.... well, I wonder why you have mydict['somevalue'] = 'something static' and mydict['somevalue'] = 'something dynamic' in your module this way: why not just set it once under '__main__'?

Related

Flask Frozen - 404 or Could not build url for endpoint error

I'm trying out the Flask Frozen library to generate static pages, but am either getting 404 errors or "Could not build url for endpoint..." in my URL generator. I am using Flask 2.0.3 and Frozen-Flask 0.18
app = Flask(__name__)
freezer = Freezer(app)
# Then I initialise a SQL Alchemy DB connection - engine & metadata
# Then register my URL generator; I also tried giving the generator the same name as my function
#freezer.register_generator
def something():
query, limit, offset = q_companies(engine, metadata, request.args)
result = engine.execute(query)
for row in result:
# when I print this I can see the first value, but then the app crashes
foo = row[0]
#this throws a 404 error
# yield '/graphics/' + foo
# this throws a Could not build url for endpoint 'graphics' with values ['symbol']. Did you mean 'static' instead?
# yield url_for('graphics', symbol=foo)
# this throws a Could not build url for endpoint 'graphics' with values ['symbol']. Did you mean 'static' instead?
# yield {'symbol': foo}
# this throws a 404 error
# yield '/graphics/', {'symbol': foo}
My function, which is declared below the if __name__ == 'main': freezer.freeze() initialiser looks like this:
#app.route('/graphics/<symbol>')
def graphics(symbol):
company, stats = get_overview_data(symbol) #SQLAlchemy query
name = company['name']
return render_template('combo.html', company=company, stats=stats, name=symbol)
I did look at this example but couldn't figure out what I may be doing wrong. Problems with URL generator in Frozen-flask
As an aside, is Frozen-Flask still the defacto lib for this use case, or are there others out there?
PS: I should add that on first initialization Frozen did copy all my static assets to build as expected
I needed to call freeze at the very bottom of my main file
if __name__ == 'main':
freezer.freeze()

Load topic file to NAO robot 2.1

Hello I want to know how to load a Dialog Topic file using python.
I made sure that the file path is right, but it keeps saying that it isn't. I have also used the tutorials in NAO 2.1's documentation ALDialog and ALModule
Please send me a code that works or tell me the error. I tried using the following code:
NAO_IP = "nao.local"
dialog_p = None
ModuleInstance = None
class NaoFalanteModule(ALModule):
def __init__(self, name):
ALModule.__init__(self, name)
self.tts = ALProxy("ALTextToSpeech")
self.tts.setLanguage("Brazilian")
global dialog_p
try:
dialog_p = ALProxy("ALDialog")
except Exception, e:
print "Error dialog"
print str(e)
exit(1)
dialog_p.setLanguage("Brazilian")
self.naoAlc()
def naoAlc(self):
topf_path = "/simpleTestes/diaSimples/testeSimples_ptb.top"
topf_path = topf_path.decode("utf-8")
topic = dialog_p.loadTopic(topf_path.encode("utf-8"))
# Start dialog
dialog_p.subscribe("NaoFalanteModule")
dialog_p.activateTopic(topic)
raw_input(u"Press 'Enter' to exit.")
dialog_p.unload(topic)
dialog_p.unsubscribe
def main():
parser = OptionParser()
parser.add_option("--pip",
help="Parent broker port. The IP address or your robot",
dest="pip")
parser.add_option("--pport",
help="Parent broker port. The port NAOqi is listening to",
dest="pport",
type="int")
parser.set_defaults(
pip=NAO_IP,
pport=9559)
(opts, args_) = parser.parse_args()
pip = opts.pip
pport = opts.pport
myBroker = ALBroker("myBroker",
"0.0.0.0",
0,
pip,
pport)
global ModuleInstance
ModuleInstance = NaoFalanteModule("ModuleInstance")
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
printI tried using the following code:
print "Interrupted by user, shutting down"
myBroker.shutdown()
sys.exit(0)
if __name__ == "__main__":
main()
The path to the topic needs to be the absolute path to that file, whereas you're passing a relative path compared to your current execution directory. The reason is that ALDialog is a separate service running in it's own process and knows nothing about the execution context of whoever is calling it.
And the program .top file must be uploaded to the robot using Choregraphe.
So, your absolute path in this case might be something like
topf_path = "/home/nao/simpleTestes/diaSimples/testeSimples_ptb.top"
... or if you want to be a bit cleaner, if you know your script is being executed at the root of your application package, use os.path:
topf_path = os.path.abspath("diaSimples/testeSimples_ptb.top")

Element not found in cache - Selenium (Python)

I just wrote a simple webscraping script to give me all the episode links on a particular site's page. The script was working fine, but, now it's broke. I didn't change anything.
Try this URL (For scraping ) :- http://www.crunchyroll.com/tabi-machi-late-show
Now, the script works mid-way and gives me an error stating, ' Element not found in the cache - perhaps the page has changed since it was looked up'
I looked it up on internet and people said about using the 'implicit wait' command at certain places. I did that, still no luck.
UPDATE : I tried this script in a demote desktop and it's working there without any problems.
Here's my script :-
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import os
import time
from subprocess import Popen
#------------------------------------------------
try:
Link = raw_input("Please enter your Link : ")
if not Link:
raise ValueError('Please Enter A Link To The Anime Page. This Application Will now Exit in 5 Seconds.')
except ValueError as e:
print(e)
time.sleep(5)
exit()
print 'Analyzing the Page. Hold on a minute.'
driver = webdriver.Firefox()
driver.get(Link)
assert "Crunchyroll" in driver.title
driver.implicitly_wait(5) # <-- I tried removing this lines as well. No luck.
elem = driver.find_elements_by_xpath("//*[#href]")
driver.implicitly_wait(10) # <-- I tried removing this lines as well. No luck.
text_file = open("BatchLink.txt", "w")
print 'Fetching The Links, please wait.'
for elem in elem:
x = elem.get_attribute("href")
#print x
text_file.write(x+'\n')
print 'Links have been fetched. Just doing the final cleaning now.'
text_file.close()
CleanFile = open("queue.txt", "w")
with open('BatchLink.txt') as f:
mylist = f.read().splitlines()
#print mylist
with open('BatchLink.txt', 'r') as inF:
for line in inF:
if 'episode' in line:
CleanFile.write(line)
print 'Please Check the file named queue.txt'
CleanFile.close()
os.remove('BatchLink.txt')
driver.close()
Here's a screenshot of the error (might be of some help) :
http://i.imgur.com/SaANlsg.png
Ok i didn't work with python but know the problem
you have variable that you init -> elem = driver.find_elements_by_xpath("//*[#href]")
after that you doing some things with it in loop
before you finishing the loop try to init this variable again
elem = driver.find_elements_by_xpath("//*[#href]")
The thing is that the DOM is changes and you loosing the element collection.

django generator function not working with real server

I have some code written in django/python. The principal is that the HTTP Response is a generator function. It spits the output of a subprocess on the browser window line by line. This works really well when I am using the django test server. When I use the real server it fails / basically it just beachballs when you press submit on the page before.
#condition(etag_func=None)
def pushviablah(request):
if 'hostname' in request.POST and request.POST['hostname']:
hostname = request.POST['hostname']
command = "blah.pl --host " + host + " --noturn"
return HttpResponse( stream_response_generator( hostname, command ), mimetype='text/html')
def stream_response_generator( hostname, command ):
proc = subprocess.Popen(command.split(), 0, None, subprocess.PIPE, subprocess.PIPE, subprocess.PIPE )
yield "<pre>"
var = 1
while (var == 1):
for line in proc.stdout.readline():
yield line
Anyone have any suggestions on how to get this working with on the real server? Or even how to debug why it is not working?
I discovered that the generator function is actually running but it has to complete before the httpresponse throws up a page onscreen. I don't want to have to wait for it to complete before the user sees output. I would like the user to see output as the subprocess progresses.
I'm wondering if this issue could be related to something in apache2 rather than django.
#evolution did you use gunicorn to deploy your app. If yes then you have created a service. I am having a similar kind of issue but with libreoffice. As much as I have researched I have found that PATH is overriding the command path present on your subprocess. I did not have a solution till now. If you bind your app with gunicorn in terminal then your code will also work.

Apache Django Mod_Wsgi - auto reload

I am trying to auto reload my django app which uses apache + mod_wsgi on my local windows machine.
I'd like to know where do I add this code that's referenced in the following article:
http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode
def _restart(path):
_queue.put(True)
prefix = 'monitor (pid=%d):' % os.getpid()
print >> sys.stderr, '%s Change detected to \'%s\'.' % (prefix, path)
print >> sys.stderr, '%s Triggering Apache restart.' % prefix
import ctypes
ctypes.windll.libhttpd.ap_signal_parent(1)
Read:
http://blog.dscpl.com.au/2008/12/using-modwsgi-when-developing-django.html
It tells you exactly where to place the file when using Django. You just need to make the code change that everyone is pointing out to you in the source code reloading documentation section related to Windows. Also read:
http://blog.dscpl.com.au/2009/02/source-code-reloading-with-modwsgi-on.html
which explains the variations on the first related to Windows.
You replace the restart function that is mentioned in the block of code above in the same article.
I use this code on my server
touch site.wsgi
and it work. After reload page in browser I get page with changes.
May be it ugly - but simple and no necessary restart apache.
In your Virtual Host config file add this:
WSGIScriptReloading On
And reload Apache
systemctl reload apache2
Enjoy!
Reference https://flask.palletsprojects.com/en/1.1.x/deploying/mod_wsgi/
You replace the restart function in the following block of code you find on the page:
Monitoring For Code Changes
The use of signals to restart a daemon process could also be employed in a mechanism which automatically detects changes to any Python modules or dependent files. This could be achieved by creating a thread at startup which periodically looks to see if file timestamps have changed and trigger a restart if they have.
Example code for such an automatic restart mechanism which is compatible with how mod_wsgi works is shown below.
import os
import sys
import time
import signal
import threading
import atexit
import Queue
_interval = 1.0
_times = {}
_files = []
_running = False
_queue = Queue.Queue()
_lock = threading.Lock()
def _restart(path):
_queue.put(True)
prefix = 'monitor (pid=%d):' % os.getpid()
print >> sys.stderr, '%s Change detected to \'%s\'.' % (prefix, path)
print >> sys.stderr, '%s Triggering process restart.' % prefix
os.kill(os.getpid(), signal.SIGINT)
I test this with Bitnami DjangoStack http://bitnami.org/stack/djangostack and Windows XP installed on D:\BitNami DjangoStack and C:\Documents and Settings\tsurahman\BitNami DjangoStack projects\myproject as project directory (default install)
as in http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode#Restarting_Apache_Processes, I added
MaxRequestsPerChild 1
in file D:\BitNami DjangoStack\apps\django\conf\django.conf
see comment by Graham Dumpleton
then I created a file monitor.py in my project directory with content as in http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode#Monitoring_For_Code_Changes and replace the _restart method with http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode#Restarting_Windows_Apache, here is the part of the script
....
_running = False
_queue = Queue.Queue()
_lock = threading.Lock()
def _restart(path):
_queue.put(True)
prefix = 'monitor (pid=%d):' % os.getpid()
print >> sys.stderr, '%s Change detected to \'%s\'.' % (prefix, path)
print >> sys.stderr, '%s Triggering Apache restart.' % prefix
import ctypes
ctypes.windll.libhttpd.ap_signal_parent(1)
def _modified(path):
try:
....
and in file D:\BitNami DjangoStack\apps\django\scripts\django.wsgi,
....
import django.core.handlers.wsgi
import monitor
monitor.start(interval=1.0)
monitor.track(os.path.join(os.path.dirname(__file__), 'site.cf'))
application = django.core.handlers.wsgi.WSGIHandler()
and then restart the Apache server