I have some code written in django/python. The principal is that the HTTP Response is a generator function. It spits the output of a subprocess on the browser window line by line. This works really well when I am using the django test server. When I use the real server it fails / basically it just beachballs when you press submit on the page before.
#condition(etag_func=None)
def pushviablah(request):
if 'hostname' in request.POST and request.POST['hostname']:
hostname = request.POST['hostname']
command = "blah.pl --host " + host + " --noturn"
return HttpResponse( stream_response_generator( hostname, command ), mimetype='text/html')
def stream_response_generator( hostname, command ):
proc = subprocess.Popen(command.split(), 0, None, subprocess.PIPE, subprocess.PIPE, subprocess.PIPE )
yield "<pre>"
var = 1
while (var == 1):
for line in proc.stdout.readline():
yield line
Anyone have any suggestions on how to get this working with on the real server? Or even how to debug why it is not working?
I discovered that the generator function is actually running but it has to complete before the httpresponse throws up a page onscreen. I don't want to have to wait for it to complete before the user sees output. I would like the user to see output as the subprocess progresses.
I'm wondering if this issue could be related to something in apache2 rather than django.
#evolution did you use gunicorn to deploy your app. If yes then you have created a service. I am having a similar kind of issue but with libreoffice. As much as I have researched I have found that PATH is overriding the command path present on your subprocess. I did not have a solution till now. If you bind your app with gunicorn in terminal then your code will also work.
Related
New to using Pytest on APIs. From my understanding, testing creates another instance of Flask. Additionally, from the tutorials I have seen, they also suggest to create a separate DB table instance to add, fetch and remove data for test purposes. However, I simply plan to use the remote api URL as host to simply make the call.
Now, I set my conftest like this, where the flag --testenv would indicate to make the get/post call on the host listed below:
import pytest
import subprocess
def pytest_addoption(parser):
"""Add option to pass --testenv=api_server to pytest cli command"""
parser.addoption(
"--testenv", action="store", default="exodemo", help="my option: type1 or type2"
)
#pytest.fixture(scope="module")
def testenv(request):
return request.config.getoption("--testenv")
#pytest.fixture(scope="module")
def testurl(testenv):
if testenv == 'api_server':
return 'http://api_url:5000/'
else:
return 'http://locahost:5000'
And my test file is written like this:
import pytest
from app import app
from flask import request
def test_nodes(app):
t_client = app.test_client()
truth = [
{
*body*
}
]
res = t_client.get('/topology/nodes')
print (res)
assert res.status_code == 200
assert truth == json.loads(res.get_data)
I run the code using this:
python3 -m pytest --testenv api_server
The thing I expect is that the test file would simply make a call to the remote api with the creds, fetch the data regardless of how it gets pulled in the remote code, and bring it here for assertion. However, I am getting the 400 BAD REQUEST error, with the error being like this:
assert 400 == 200
E + where 400 = <WrapperTestResponse streamed [400 BAD REQUEST]>.status_code
single_test.py:97: AssertionError
--------------------- Captured stdout call ----------------------
{"timestamp": "2022-07-28 22:11:14,032", "level": "ERROR", "func": "connect_to_mysql_db", "line": 23, "message": "Error connecting to the mysql database (2003, \"Can't connect to MySQL server on 'mysql' ([Errno -3] Temporary failure in name resolution)\")"}
<WrapperTestResponse streamed [400 BAD REQUEST]>
Does this mean that the test file is still trying to lookup the database locally for fetching? I am unable to figure out on which host are they sending the test url as well, so I am kind of stuck here. Looking to get some help around here.
Thanks.
I have a Django web app served from Apache2 with mod_wsgi in docker containers running on a Kubernetes cluster in Google Cloud Platform, protected by Identity-Aware Proxy. Everything is working great, but I want to send GCP Stackdriver traces for all requests without writing one for each view in my project. I found middleware to handle this, using Opencensus. I went through this documentation, and was able to manually generate traces that exported to Stackdriver Trace in my project by specifying the StackdriverExporter and passing the project_id parameter as the Google Cloud Platform Project Number for my project.
Now to make this automatic for ALL requests, I followed the instructions to set up the middleware. In settings.py, I added the module to INSTALLED_APPS, MIDDLEWARE, and set up the OPENCENSUS_TRACE dictionary of options. I also added the OPENCENSUS_TRACE_PARAMS. This works great with the default exporter 'opencensus.trace.exporters.print_exporter.PrintExporter', as I can see the Trace and Span information, including Trace ID and all details in my Apache2 web server logs. However, I want to send these to my Stackdriver Trace processor for analysis.
I tried setting the EXPORTER parameter to opencensus.trace.exporters.stackdriver_exporter.StackdriverExporter, which works when run manually from the shell, as long as you supply the project number.
When it is set up to use StackdriverExporter, the web page will not respond load, the health check starts to fail, and ultimately the web page comes back with a 502 error, stating I should try again in 30 seconds (I believe the Identity-Aware Proxy is generating this error, once it detects the failed health check), but the server generates no errors, and there are no logs in access or errors for Apache2.
There is another dictionary in settings.py named OPENCENSUS_TRACE_PARAMS, which I presume is needed to determine which project number the exporter should be using. The example has GCP_EXPORTER_PROJECT set as None, and SERVICE_NAME set as 'my_service'.
What options do I need to set to get the exporter to send back to Stackdriver instead of printing to logs? Do you have any idea about how I can set this up?
settings.py
MIDDLEWARE = (
...
'opencensus.trace.ext.django.middleware.OpencensusMiddleware',
)
INSTALLED_APPS = (
...
'opencensus.trace.ext.django',
)
OPENCENSUS_TRACE = {
'SAMPLER': 'opencensus.trace.samplers.probability.ProbabilitySampler',
'EXPORTER': 'opencensus.trace.exporters.stackdriver_exporter.StackdriverExporter', # This one just makes the server hang with no response or error and kills the health check.
'PROPAGATOR': 'opencensus.trace.propagation.google_cloud_format.GoogleCloudFormatPropagator',
# 'EXPORTER': 'opencensus.trace.exporters.print_exporter.PrintExporter', # This one works to print the Trace and Span with IDs and details in the logs.
}
OPENCENSUS_TRACE_PARAMS = {
'BLACKLIST_PATHS': ['/health'],
'GCP_EXPORTER_PROJECT': 'my_project_number', # Should this be None like the example, or Project ID, or Project Number?
'SAMPLING_RATE': 0.5,
'SERVICE_NAME': 'my_service', # Not sure if this is my app name or some other service name.
'ZIPKIN_EXPORTER_HOST_NAME': 'localhost', # Are the following even necessary, or are they causing a failure that is not detected by Apache2?
'ZIPKIN_EXPORTER_PORT': 9411,
'ZIPKIN_EXPORTER_PROTOCOL': 'http',
'JAEGER_EXPORTER_HOST_NAME': None,
'JAEGER_EXPORTER_PORT': None,
'JAEGER_EXPORTER_AGENT_HOST_NAME': 'localhost',
'JAEGER_EXPORTER_AGENT_PORT': 6831
}
Here's an example (I prettified the format for readability) of the Apache2 log when it is set to use the PrintExporter:
[Fri Feb 08 09:00:32.427575 2019]
[wsgi:error]
[pid 1097:tid 139801302882048]
[client 10.48.0.1:43988]
[SpanData(
name='services.views.my_view',
context=SpanContext(
trace_id=e882f23e49e34fc09df621867d753532,
span_id=None,
trace_options=TraceOptions(enabled=True),
tracestate=None
),
span_id='bcbe7b96906a482a',
parent_span_id=None,
attributes={
'http.status_code': '200',
'http.method': 'GET',
'http.url': '/',
'django.user.name': ''
},
start_time='2019-02-08T17:00:29.845733Z',
end_time='2019-02-08T17:00:32.427455Z',
child_span_count=0,
stack_trace=None,
time_events=[],
links=[],
status=None,
same_process_as_parent_span=None,
span_kind=1
)]
Thanks in advance for any tips, assistance, or troubleshooting advice!
Edit 2019-02-08 6:56 PM UTC:
I found this in the middleware:
# Initialize the exporter
transport = convert_to_import(settings.params.get(TRANSPORT))
if self._exporter.__name__ == 'GoogleCloudExporter':
_project_id = settings.params.get(GCP_EXPORTER_PROJECT, None)
self.exporter = self._exporter(
project_id=_project_id,
transport=transport)
elif self._exporter.__name__ == 'ZipkinExporter':
_service_name = self._get_service_name(settings.params)
_zipkin_host_name = settings.params.get(
ZIPKIN_EXPORTER_HOST_NAME, 'localhost')
_zipkin_port = settings.params.get(
ZIPKIN_EXPORTER_PORT, 9411)
_zipkin_protocol = settings.params.get(
ZIPKIN_EXPORTER_PROTOCOL, 'http')
self.exporter = self._exporter(
service_name=_service_name,
host_name=_zipkin_host_name,
port=_zipkin_port,
protocol=_zipkin_protocol,
transport=transport)
elif self._exporter.__name__ == 'TraceExporter':
_service_name = self._get_service_name(settings.params)
_endpoint = settings.params.get(
OCAGENT_TRACE_EXPORTER_ENDPOINT, None)
self.exporter = self._exporter(
service_name=_service_name,
endpoint=_endpoint,
transport=transport)
elif self._exporter.__name__ == 'JaegerExporter':
_service_name = self._get_service_name(settings.params)
self.exporter = self._exporter(
service_name=_service_name,
transport=transport)
else:
self.exporter = self._exporter(transport=transport)
The exporter is now named StackdriverExporter, instead of GoogleCloudExporter. I set up a class in my app named GoogleCloudExporter that inherits StackdriverExporter, and updated my settings.py to use GoogleCloudExporter, but it didn't seem to work, I wonder if there is other code referencing these old naming schemes, possibly for the transport. I'm searching the source code for clues... This at least tells me I can get rid of the ZIPKIN and JAEGER param options, as this is determined on the EXPORTER param.
Edit 2019-02-08 11:58 PM UTC:
I scrapped Apache2 to isolate the problem and just set my docker image to use Django's built in webserver CMD ["python", "/path/to/manage.py", "runserver", "0.0.0.0:80"] and it works! When I go to the site, it writes traces to Stackdriver Trace for each request, the Span name is the module and method being executed.
Somehow Apache2 is not being allowed to send these, but I can do so from the shell when running as root. I'm adding Apache2 and mod-wsgi tags to the question, because I have a funny feeling this has to do with forking child processes in Apache2 and mod-WSGI. Would it be the child process being unable to be created as apache2's child process is sandboxed, or could this be a permissions thing? It seems strange, because it is just calling python modules, no external system OS binaries, that I am aware of. Any other ideas would be greatly appreciated!
I had this problem while using gunicorn with gevent as the worker class. To resolve and get cloud traces working the solution was to monkey patch grpc like so
from gevent import monkey
monkey.patch_all()
import grpc.experimental.gevent as grpc_gevent
grpc_gevent.init_gevent()
See https://github.com/grpc/grpc/issues/4629#issuecomment-376962677
I have a process that scans a tape library and looks for media that has expired, so they can be removed and reused before sending the tapes to an offsite vault. (We have some 7 day policies that never make it offsite.) This process takes around 20 minutes to run, so I didn't want it to run on-demand when loading/refreshing the page. Rather, I set up a django-cron job (I know I could have done this in Linux cron, but wanted the project to be as self-contained as possible) to run the scan, and creates a file in /tmp. I've verified that this works -- the file exists in /tmp from this morning's execution. The problem I'm having is that now I want to display a list of those expired (scratch) media on my web page, but the script is saying that it can't find the file. When the file was created, I use the absolute filename "/tmp/scratch.2015-11-13.out" (for example), but here's the error I get in the browser:
IOError at /
[Errno 2] No such file or directory: '/tmp/corpscratch.2015-11-13.out'
My assumption is that this is a "web root" issue, but I just can't figure it out. I tried copying the file to the /static/ and /media/ directories configured in django, and even in the django root directory, and the project root directory, but nothing seems to work. When it says it cant' find /tmp/file, where is it really looking?
def sample():
""" Just testing """
today = datetime.date.today() #format 2015-11-31
inputfile = "/tmp/corpscratch.%s.out" % str(today)
with open(inputfile) as fh: # This is the line reporting the error
lines = [line.strip('\n') for line in fh]
print(lines)
The print statement was used for testing in the shell (which works, I might add), but the browser gives an error.
And the file does exist:
$ ls /tmp/corpscratch.2015-11-13.out
/tmp/corpscratch.2015-11-13.out
Thanks.
Edit: was mistaken, doesn't work in python shell either. Was thinking of a previous issue.
Use this instead:
today = datetime.datetime.today().date()
inputfile = "/tmp/corpscratch.%s.out" % str(today)
Or:
today = datetime.datetime.today().strftime('%Y-%m-%d')
inputfile = "/tmp/corpscratch.%s.out" % today # No need to use str()
See the difference:
>>> str(datetime.datetime.today().date())
'2015-11-13'
>>> str(datetime.datetime.today())
'2015-11-13 15:56:19.578569'
I ended up finding this elsewhere:
today = datetime.date.today() #format 2015-11-31
inputfilename = "tmp/corpscratch.%s.out" % str(today)
inputfile = os.path.join(settings.PROJECT_ROOT, inputfilename)
With settings.py containing the following:
PROJECT_ROOT = os.path.abspath(os.path.dirname(__file__))
Completely resolved my issues.
I am trying out spotipy with python 2.7.10 preinstalled on my mac 10.10, specifically [add_a_saved_track.py][1] Here is the code as copied from github:
# Add tracks to 'Your Collection' of saved tracks
import pprint
import sys
import spotipy
import spotipy.util as util
scope = 'user-library-modify'
if len(sys.argv) > 2:
username = sys.argv[1]
tids = sys.argv[2:]
else:
print("Usage: %s username track-id ..." % (sys.argv[0],))
sys.exit()
token = util.prompt_for_user_token(username, scope)
if token:
sp = spotipy.Spotify(auth=token)
sp.trace = False
results = sp.current_user_saved_tracks_add(tracks=tids)
pprint.pprint(results)
else:
print("Can't get token for", username)
I registered the application with developer.spotify.com/my-applications and received client_id and client_secret. I am a bit unclear about selection of redirect_uri so I set that to 'https://play.spotify.com/collection/songs'
Running this from terminal I get an error that says:
You need to set your Spotify API credentials. You can do this by
setting environment variables like so:
export SPOTIPY_CLIENT_ID='your-spotify-client-id'
export SPOTIPY_CLIENT_SECRET='your-spotify-client-secret'
export SPOTIPY_REDIRECT_URI='your-app-redirect-url'
I put that into my code with the id, secret, and url as strings, just following the imports but above the util.prompt_for_user_token method.
That caused a traceback:
File "add-track.py", line 8
export SPOTIPY_CLIENT_ID='4f...6'
^
SyntaxError: invalid syntax
I noticed that Text Wrangler does not recognize 'export' as a special word. And I searched docs.python.org for 'export' and came up with nothing helpful. What is export? How am I using it incorrectly?
I next tried passing the client_id, client_secret, and redirect_uri as arguments in the util.prompt_for_user_token method like so:
util.prompt_for_user_token(username,scope,client_id='4f...6',client_secret='xxx...123',redirect_uri='https://play.spotify.com/collection/songs')
When I tried that, this is what happens in terminal:
User authentication requires interaction with your
web browser. Once you enter your credentials and
give authorization, you will be redirected to
a url. Paste that url you were directed to to
complete the authorization.
Opening https://accounts.spotify.com/authorize?scope=user-library-modify&redirect_uri=https%3A%2F%2Fplay.spotify.com%2Fcollection%2Fsongs&response_type=code&client_id=4f...6 in your browser
Enter the URL you were redirected to:
I entered https://play.spotify.com/collection/songs and then got this traceback:
Traceback (most recent call last):
File "add-track.py", line 21, in <module>
token = util.prompt_for_user_token(username, scope, client_id='4f...6', client_secret='xxx...123', redirect_uri='https://play.spotify.com/collection/songs')
File "/Library/Python/2.7/site-packages/spotipy/util.py", line 86, in prompt_for_user_token
token_info = sp_oauth.get_access_token(code)
File "/Library/Python/2.7/site-packages/spotipy/oauth2.py", line 210, in get_access_token
raise SpotifyOauthError(response.reason)
spotipy.oauth2.SpotifyOauthError: Bad Request
It seems like I am missing something, perhaps another part of Spotipy needs to be imported, or some other python module. It seems I am missing the piece that sets client credentials. How do I do that? I am fairly new at this (if that wasn't obvious). Please help.
UPDATE: I changed redirect_uri to localhost:8888/callback. That causes a Firefox tab to open with an error -- "unable to connect to server." (Since I do not have a server running. I thought about installing node.js as in the Spotify Web API tutorial, but I have not yet). The python script then asks me to copy and paste the URL I was redirected to. Even though FF could not open a page, I got this to work by copying the entire URL including the "code=BG..." that follows localhost:8888/callback? I am not sure this is an ideal setup, but at least it works.
Does it matter if I set up node.js or not?
The process you've followed (including your update) is exactly as the example intends and you are not missing anything! Obviously, it is a fairly simple tutorial, but it sets you up with a token and you should be able to get the information you need.
For the credentials, you can set these directly in your Terminal by running each of the export commands. Read more about EXPORT here: https://www.cyberciti.biz/faq/linux-unix-shell-export-command/
Hi am absolutely new to django,Now I am tring for builtin reset password view..
I follow the link link
But I got the error when I click on reset password button at /password_reset/ :
error at /accounts/password_reset/
[Errno 10061] No connection could be made because the target machine actively refused it.
Exception Location: C:\Python27\lib\socket.py in create_connection, line 571
'urls.py'
(r'^accounts/password_reset$','django.contrib.auth.views.password_reset','template_name':'user/password_reset_form.html','email_template_name':'user/password_reset_email.html'}),
(r'^accounts/password/reset/confirm/(?P[0-9A-Za-z]{1,13})-(?P[0-9Aa-z]{1,13}-[0-9A-Za-z]{1,20})/$', 'django.contrib.auth.views.password_reset_confirm', {'template_name' : 'user/password_reset.html', 'post_reset_redirect': '/logout/' }),
(r'^accounts/password_reset/done/$',<b>'django.contrib.auth.views.password_reset_done'</b>,{'template_name':'user/password_reset_done.html'}),
(r'^accounts/change_password/$',<b> 'password_change'</b>, {'post_change_redirect' : '/accounts/change_password/done/'}),
(r'^accounts/change_password/done/$',<b> 'password_change_done'</b>,{'template_name':'user/password_change_done.html'}),
<b>password_reset_email.html</b>
{% extends 'base.html'%}
{% block content%}
{% url django.contrib.auth.views.password_reset_confirm uidb36=uid, token=token %}
{% endblock %}
I add necessary templates in the folder 'user'.Please help me,Thanks in advance.
As rdegges said, it's a connection error. Check which port the request is trying to access, and make sure windows firewall is set up to receive on that port. You can check the port by looking through django's traceback page, and looking at the "local vars".
From the looks of it, it's an email port. Take another look at the traceback and look out for django trying to send a request to port 25. If it does, make sure your port 25 is configured to receive.
Also, you'll need a makeshift SMTP server for testing purposes because you probably wouldn't want to be using a real one. Just have this running in a separate command prompt window while you're running django, and any emails that django tries to send through your port 25 will be saved in an "emails" folder in the working directory.
#/usr/bin/env python
from datetime import datetime
import asyncore
from smtpd import SMTPServer
class EmlServer(SMTPServer):
no = 0
def process_message(self, peer, mailfrom, rcpttos, data):
filename = '%s-%d.eml' % (datetime.now().strftime('%Y%m%d%H%M%S'), self.no)
f = open(filename, 'w')
f.write(data)
f.close()
print '%s saved.' % filename
self.no += 1
def run():
foo = EmlServer(('localhost', 25), None)
try:
asyncore.loop()
except KeyboardInterrupt:
pass
if __name__ == '__main__':
from os.path import exists, join
from os import mkdir, chdir, getcwd
target_directory = join( getcwd(), "emails" )
if exists(target_directory):
chdir(target_directory)
else:
try:
mkdir(target_directory)
except OSError:
from sys import exit
exit("The containing folder couldn't be created, check your permissions.")
chdir(target_directory)
print "Using directory %s" % target_directory
run()
The error you're getting is a connection error--that means that the server you're using to run your Django site is likely not working correctly. Here are some things you can try on your Django server:
If you're running your Django site via python manage.py runserver, you can try to simply re-run that command.
If you're running your site via a webserver like apache, try restarting apache.
If neither of those work, post a comment and let me know what operating system you're running your site on, and how you're running it, and we can do further debugging.