Flask throwing 'Working outside of request context.' - flask

I am trying to use celery for my app which is made in flask but I get the following error "Working outside of request context". It sounds like I am trying to access a request object before the front end makes a request, but I cannot figure out what is wrong. I appreciate if you can let me know what is the problem.
[2017-04-26 13:33:04,940: INFO/MainProcess] Received task: app.result[139a2679-e9df-49b9-ab42-1f53a09c01fd]
[2017-04-26 13:33:06,168: ERROR/PoolWorker-2] Task app.result[139a2679-e9df-49b9-ab42-1f53a09c01fd] raised unexpected: RuntimeError('Working outside of request context.\n\nThis typically means that you attempted to use functionality that needed\nan active HTTP request. Consult the documentation on testing for\ninformation about how to avoid this problem.',)
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/celery/app/trace.py", line 367, in trace_task
R = retval = fun(*args, **kwargs)
File "/Users/Pooneh/projects/applications/ray_tracer_app_flask/flask_celery.py", line 14, in __call__
return TaskBase.__call__(self, *args, **kwargs)
File "/Library/Python/2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
return self.run(*args, **kwargs)
File "/Users/Pooneh/projects/applications/ray_tracer_app_flask/app.py", line 33, in final_result
light_position = request.args.get("light_position", "(0, 0, 0)", type=str)
File "/Library/Python/2.7/site-packages/werkzeug/local.py", line 343, in __getattr__
return getattr(self._get_current_object(), name)
File "/Library/Python/2.7/site-packages/werkzeug/local.py", line 302, in _get_current_object
return self.__local()
File "/Library/Python/2.7/site-packages/flask/globals.py", line 37, in _lookup_req_object
raise RuntimeError(_request_ctx_err_msg)
RuntimeError: Working outside of request context.
This typically means that you attempted to use functionality that needed
an active HTTP request. Consult the documentation on testing for
information about how to avoid this problem.
app.py
app = Flask(__name__)
app.config.update(CELERY_BROKER_URL = 'amqp://localhost//',
CELERY_RESULT_BACKEND='amqp://localhost//')
celery = make_celery(app)
#app.route('/')
def my_form():
return render_template("form.html")
#app.route('/result')
def result():
final_result.delay()
return "celery!"
#celery.task(name='app.result')
def final_result():
light_position = request.args.get("light_position", "(0, 0, 0)", type=str)
light_position_coor = re.findall("[-+]?\d*\.\d+|[-+]?\d+", light_position)
x = float(light_position_coor[0])
y = float(light_position_coor[1])
z = float(light_position_coor[2])
encoded = base64.b64encode(open("/Users/payande/projects/applications/app_flask/static/pic.png", "rb").read())
return jsonify(data=encoded)

Celery tasks are run by a background worker asynchronously outside of the HTTP request (which is one of they main benefits of using them), so you cannot access the request object within the task.
You could pass the data to the task as arguments instead:
final_result.delay(request.args.get("light_position"))
#celery.task(name='app.result')
def final_result(light_position):
...
Of course this also means that the return value of the task cannot be used in a HTTP response (since the task can complete after the response has been already sent).

Related

How to list Google Scheduler Jobs from all locations within a project with Python

I want to list all the Google Cloud Schedule jobs within a project, but to use the ListJobsRequest() class the parent parameter is required: projects/PROJECT_ID/locations/LOCATION_ID. Since I have jobs in different locations I would like to list all jobs, is it possible to do?
I already tried projects/PROJECT_ID/locations/* and projects/PROJECT_ID/locations.
In both cases I got the following error:
Traceback (most recent call last):
File "C:\Users\user\Desktop\TI\GCP\bq-to-scheduler\main.py", line 97, in <module>
list_scheduler_jobs()
File "C:\Users\user\Desktop\TI\GCP\bq-to-scheduler\main.py", line 51, in list_scheduler_jobs
page_result = client.list_jobs(request=request)
File "C:\Users\user\Desktop\TI\GCP\bq-to-scheduler\venv\lib\site-packages\google\cloud\scheduler_v1\services\cloud_scheduler\client.py", line 548, in list_jobs
response = rpc(
File "C:\Users\user\Desktop\TI\GCP\bq-to-scheduler\venv\lib\site-packages\google\api_core\gapic_v1\method.py", line 154, in __call__
return wrapped_func(*args, **kwargs)
File "C:\Users\user\Desktop\TI\GCP\bq-to-scheduler\venv\lib\site-packages\google\api_core\retry.py", line 283, in retry_wrapped_func
return retry_target(
File "C:\Users\user\Desktop\TI\GCP\bq-to-scheduler\venv\lib\site-packages\google\api_core\retry.py", line 190, in retry_target
return target()
File "C:\Users\user\Desktop\TI\GCP\bq-to-scheduler\venv\lib\site-packages\google\api_core\grpc_helpers.py", line 52, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.PermissionDenied: 403 The principal (user or service account) lacks IAM permission "cloudscheduler.jobs.list" for the resource "projects/projeto1-358102/locations/*" (or the resource may not exist).
Help?
You can list all your available locations in your project using the code below. I also included calling list_jobs() to send a request to list the available jobs on the location.
I got the code on listing location in this document but I edited the authentication to use google.auth library instead of oauth2client.client since this is already deprecated.
from googleapiclient import discovery
import google.auth
from google.cloud import scheduler_v1
def get_project_locations():
credentials, project = google.auth.default(scopes=['https://www.googleapis.com/auth/cloud-platform'])
service = discovery.build('cloudscheduler', 'v1', credentials=credentials)
name = f'projects/{project}'
loc_arr = []
request = service.projects().locations().list(name=name)
while True:
response = request.execute()
for location in response.get('locations', []):
loc_arr.append(location['labels']['cloud.googleapis.com/region'])
request = service.projects().locations().list_next(previous_request=request, previous_response=response)
if request is None:
break
return project,loc_arr
def list_jobs(project_id,location_list):
client = scheduler_v1.CloudSchedulerClient()
for location in location_list:
request = scheduler_v1.ListJobsRequest(parent = f"projects/{project_id}/locations/{location}")
page_result = client.list_jobs(request=request)
for response in page_result:
print(response.name)
print(response.http_target)
print(response.schedule)
project_id,location_list = get_project_locations()
list_jobs(project_id,location_list)
Output:
From GCP scheduler:

Api with flask-jwt-extended with authentication problems?

I have built an api with flask-restful and flask-jwt-extended and have correctly configured the validation passages for token expiration and invalidation. However, even though it has built the token expiration and invalid validation callbacks, api does not process correctly and reports the error: Signature has expired
On the server in the cloud, we have a Centos 7 x64 of 16gb ram, running the application using gunicorn in version 19.9.0. Using the miniconda to create the applications' python environments.
In tests in the production environment, the application complains of the expired token. However in a test environment, using Ubuntu 18.04.2, x64 with 16 gb ram, using the same settings with miniconda and gunicorn, the application has no problems executing it, returning the correct message when the token expires.
My jwt.py
from flask import Blueprint, Response, json, request
from flask_jwt_extended import (JWTManager, create_access_token,
create_refresh_token, get_jwt_identity,
jwt_required)
from app.models.core import User
from .schemas import UserSchema
from .utils import send_reponse, user_roles
def configure_jwt(app):
JWT = JWTManager(app)
#JWT.expired_token_loader
def my_expired_token_callback(expired_token):
return Response(
response=json.dumps({
"message": "Expired token"
}),
status=401,
mimetype='application/json'
)
#JWT.invalid_token_loader
def my_invalid_token_callback(invalid_token):
return Response(
response=json.dumps({
"message": "Invalid token"
}),
status=422,
mimetype='application/json'
)
Error log:
[2019-05-23 15:42:02 -0300] [3745] [ERROR] Exception on /api/company [POST]
Traceback (most recent call last):
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask_restful/__init__.py", line 458, in wrapper
resp = resource(*args, **kwargs)
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask/views.py", line 88, in view
return self.dispatch_request(*args, **kwargs)
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask_restful/__init__.py", line 573, in dispatch_request
resp = meth(*args, **kwargs)
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask_jwt_extended/view_decorators.py", line 102, in wrapper
verify_jwt_in_request()
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask_jwt_extended/view_decorators.py", line 31, in verify_jwt_in_request
jwt_data = _decode_jwt_from_request(request_type='access')
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask_jwt_extended/view_decorators.py", line 266, in _decode_jwt_from_request
decoded_token = decode_token(encoded_token, csrf_token)
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask_jwt_extended/utils.py", line 107, in decode_token
allow_expired=allow_expired
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/flask_jwt_extended/tokens.py", line 138, in decode_jwt
leeway=leeway, options=options)
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/jwt/api_jwt.py", line 104, in decode
self._validate_claims(payload, merged_options, **kwargs)
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/jwt/api_jwt.py", line 134, in _validate_claims
self._validate_exp(payload, now, leeway)
File "/home/company/miniconda3/envs/api_ms/lib/python3.6/site-packages/jwt/api_jwt.py", line 175, in _validate_exp
raise ExpiredSignatureError('Signature has expired')
jwt.exceptions.ExpiredSignatureError: Signature has expired
I'm trying to understand why the application is able to correctly return the token expiration message in the test environment, where in the production environment it returns the error code 500 Internal Server Error. In addition to fixing this problem in our application.
Based on this link found inside the project repository, I discovered that the problem is related to the flask configuration option called PROPAGATE_EXCEPTIONS, which must be True.
The issue in the flask-jwt-extended repository that helped me find the answer.
This comment states that Flask Restful needs to ignore JWT and JWT Extended Exceptions and provides a simple snippet that solves the issue.
Copying the code from above link,
from flask_jwt_extended.exceptions import JWTExtendedException
from jwt.exceptions import PyJWTError
class FixedApi(Api):
def error_router(self, original_handler, e):
if not isinstance(e, PyJWTError) and not isinstance(e, JWTExtendedException) and self._has_fr_route():
try:
return self.handle_error(e)
except Exception:
pass # Fall through to original handler
return original_handler(e)

Flask SQLAlchemy Transaction error when using server_default in model in models_committed signal handler

My model (before) contains client-side defaults:
created_ts = db.Column(db.DateTime(timezone=True), default=dt.datetime.now)
My model (after) now contains server-side defaults:
created_ts = db.Column(db.DateTime(timezone=True), server_default=text('NOW()'))
However, I now start seeing the error:
InvalidRequestError: This session is in 'committed' state; no further SQL can be emitted within this transaction.
In my models_committed hook:
#models_committed.connect_via(app)
def handle(sender, changes):
for model, operation in changes:
model.to_dict() # error here
I stole to_dict from flask_sandboy:
def to_dict(self):
"""Return dict representation of class by iterating over database
columns."""
value = {}
for column in self.__table__.columns:
attribute = getattr(self, column.name) # error here
if isinstance(attribute, datetime.datetime):
attribute = str(attribute)
value[column.name] = attribute
return value
So, getattr(self, column.name) seems to trigger the server-side default somehow (presumably, since that's the change I introduced).
From this line in my own code, I provide the rest of the stack trace:
File "/code/models/session.py", line 20, in to_dict
attribute = getattr(self, column.name)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 239, in __get__
return self.impl.get(instance_state(instance), dict_)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 589, in get
value = callable_(state, passive)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/state.py", line 433, in __call__
self.manager.deferred_scalar_loader(self, toload)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 613, in load_scalar_attributes
only_load_props=attribute_names)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 235, in load_on_ident
return q.one()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2398, in one
ret = list(self)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2441, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2454, in _execute_and_instances
close_with_result=True)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2445, in _connection_from_session
**kw)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 880, in connection
execution_options=execution_options)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 885, in _connection_for_bind
engine, execution_options)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 305, in _connection_for_bind
self._assert_active()
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 196, in _assert_active
"This session is in 'committed' state; no further "
InvalidRequestError: This session is in 'committed' state; no further SQL can be emitted within this transaction.
How do I get around this problem?
pip freeze:
aniso8601==0.92
blinker==1.3
boto==2.36.0
Flask==0.10.1
Flask-Cors==1.10.3
Flask-HTTPAuth==2.4.0
Flask-SQLAlchemy==2.0
gunicorn==19.3.0
itsdangerous==0.24
psycopg2==2.6
pytz==2014.10
six==1.9.0
SQLAlchemy==0.9.9
Werkzeug==0.10.1
TL;DR solution #4 below works
The created_ts field is not known in that request thread, nor in the subsequent signal handler, since the value is chosen server-side by PostgresQL.
getattr(self, column.name) then tries to hydratepopulate that value by going back to the DB server, but unfortunately is now outside of a transaction.
A few options here:
Start a new session in the signal handler so as to retrieve the value from the DB (this is an untested suggestion)
Use the built-in _dict to get the model's state, but it won't have the created_ts field, because it is not known at this point:
Code
modeldict = dict(model.__dict__)
modeldict.pop('_sa_instance_state', None)
Stick with default=dt.datetime.now, but non-Python apps which write to the database need to provide created_ts.
So, the trade-off is whether the consumer of the model (as alluded to with the pika comment) needs created_ts or not.
(tested to work) Thanks to a combination of advice from agronholm on the #sqlalchemy IRC room and kbussel on this ticket, I tried to serialize the data while the session is still open, and when I'm sure the data is committed, I re-use the serialized data and by-pass going back to the DB:
from sqlalchemy.event import listens_for
from flask.ext.sqlalchemy import SignallingSession
#listens_for(SignallingSession, 'after_flush')
def after_flush_handler(session, tx):
try:
d = session._model_changes
except AttributeError:
return
if d:
changes = []
for model, operation in list(d.values()):
changes.append((model.to_dict(), operation))
session.info['my_changes'] = changes
d.clear()
#listens_for(SignallingSession, 'after_commit')
def after_commit_handler(session):
if 'my_changes' in session.info:
changes = session.info['my_changes']
for model, operation in changes:
# use model here, with all data populated

Haystack: KeyError in the elasticsearch_backend module

I'm using Django + Haystack + Elasticsearch.
When I send a request to this view
from haystack.views import FacetedSearchView
from .models import Object
class ObjectView(FacetedSearchView):
def extra_context(self):
extra = super(ObjectView, self).extra_context()
if not self.results:
extra['objects'] = Object.objects.all()
else:
searchqueryset = self.form.search()
results = [ result.pk for result in searchqueryset ]
extra['facets'] = self.results.facet_counts()
extra['objects'] = Object.objects.filter(pk__in=results)
extra['results'] = self.results
return extra
this error is raised:
File "/home/deploy/.virtualenvs/deploy/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 115, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/home/deploy/.virtualenvs/deploy/local/lib/python2.7/site-packages/haystack/views.py", line 49, in __call__
return self.create_response()
File "/home/deploy/.virtualenvs/deploy/local/lib/python2.7/site-packages/haystack/views.py", line 129, in create_response
(paginator, page) = self.build_page()
File "/home/deploy/.virtualenvs/deploy/local/lib/python2.7/site-packages/haystack/views.py", line 106, in build_page
self.results[start_offset:start_offset + self.results_per_page]
File "/home/deploy/.virtualenvs/deploy/local/lib/python2.7/site-packages/haystack/query.py", line 266, in __getitem__
self._fill_cache(start, bound)
File "/home/deploy/.virtualenvs/deploy/local/lib/python2.7/site-packages/haystack/query.py", line 164, in _fill_cache
results = self.query.get_results(**kwargs)
File "/home/deploy/.virtualenvs/deploy/local/lib/python2.7/site-packages/haystack/backends/__init__.py", line 485, in get_results
self.run(**kwargs)
File "/home/deploy/.virtualenvs/deploy/local/lib/python2.7/site-packages/haystack/backends/elasticsearch_backend.py", line 942, in run
results = self.backend.search(final_query, **search_kwargs)
File "/home/deploy/.virtualenvs/deploy/local/lib/python2.7/site-packages/haystack/backends/__init__.py", line 26, in wrapper
return func(obj, query_string, *args, **kwargs)
File "/home/deploy/.virtualenvs/deploy/local/lib/python2.7/site-packages/haystack/backends/elasticsearch_backend.py", line 521, in search
distance_point=kwargs.get('distance_point'), geo_sort=geo_sort)
File "/home/deploy/.virtualenvs/deploy/local/lib/python2.7/site-packages/haystack/backends/elasticsearch_backend.py", line 571, in _process_results
raw_suggest = raw_results['suggest']['suggest']
KeyError: 'suggest'
A curious fact: the problem occurs only when the project is under production settings, even when I haven't changed a single thing involving Haystack or Elasticsearch in the settings_production module(except for the URL key).
project/settings_production.py
'URL': 'http://0.0.0.0:9200/'
In production, I'm using nothing more than a simple FastCGI.
And here's what really bothers me: sometimes I get no errors on this view, and everything works just fine...
Please, someone has an idea of what's going on?
Thanks a lot!
UPDATE:
SO, I setup my whole project in another computer. After some tests I verified:
this problem is not related to my production settings like I
described above;
the error is not raised when the elasticsearch service is stopped;
if the service is running:
when the method Object.objects.all() returns some QuerySet results, I got no errors;
when the method Object.objects.all() returns an empty QuerySet, the problem persists;
I guess this is some kind of bug in the Haystack's elasticsearch_backend module.
Still, i'm not sure.
Yup, it's a bug in haystack. I've put in a pull request, but in the meantime, options to get running are:
Set INCLUDE_SPELLING in your haystack settings to False, or
Use our fork: https://github.com/greenkahuna/django-haystack

django web app testing

I have the following in tests.py.
def setUp(self):
self.client = Client()
self.client.get('/homepage',{'join':'NPO2','siteid':1450})
self.client.session.save()
self.oraganisation_list = ['NPO1','NPO2','NPO3']
self.pay_recursion_list = ['annual','monthly','bi-annual','quarter']
def test_paytermpage(self):
for org in self.organisation_list:
response = self.client.get('',{'join':org,'siteid':1450})
self.failUnlessEqual(response.status_code,200)
self.assertTemplateUsed(response,'some.html')
def test_infopage(self):
for term in self.pay_recurstion_list:
response = self.client.post('',{'pay-term':term,'submit':'payterm'})
self.failUnlessEqual(response.status_code,200)
test_infopage() is failing and here is the traceback.
Traceback (most recent call last):
File "/var/lib/django/bsdata/shoppingcart/tests.py", line 50, in test_infopage
response = self.client.post('',{'pay-term':term,'submit':'payterm'})
File "/usr/lib/pymodules/python2.6/django/test/client.py", line 313, in post
response = self.request(**r)
File "/usr/lib/pymodules/python2.6/django/core/handlers/base.py", line 92, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/var/lib/django/.../views.py", line 22, in start
term,costdict,webobj = costInfo(request)
File "/var/lib/django/...views.py", line 238, in getCostInfo
cost_dict = Site.objects.getDict(request.session['siteid'])
File "/var/lib/django/.../managers.py", line 16, in getLoadedDict
siteobj = Site.objects.get(pk=agent)
File "/usr/lib/pymodules/python2.6/django/db/models/manager.py", line 120, in get
return self.get_query_set().get(*args, **kwargs)
File "/usr/lib/pymodules/python2.6/django/db/models/query.py", line 305, in get
% self.model._meta.object_name)
DoesNotExist: Site matching query does not exist.
I did debug to see what value 'agent' in siteobj = Site.objects.get(pk=agent) is getting its a valid integer.
Surprisingly both of them are working when tested from shell like this
setup_test_environment()
client = Client()
client.get('/shoppingcart',{'join':'NPO1','siteid':1450})
client.session.save()
oraganisation_list = ['NPO1','NPO2','NPO3']
pay_recursion_list = ['annual','monthly','bi-annual','quarter']
for org in oraganisation_list:
response = client.get('',{'join':org,'siteid':1450})
TestCase.failUnlessEqual(t,response.status_code,200)
for term in pay_recursion_list:
response = client.post('',{'pay-term':term,'submit':'payterm'})
TestCase.failUnlessEqual(t,response.status_code,200)
Sorry for too much info,didn't know how to explain better.
Any ideas would be highly helpful for this newbie. Thanks.
The Django test runner uses a different database than your production data. If your site requires a Site to exist, you should either add it in your Test setUp, or you should require a fixture that loads the site.
Try putting "SITE_ID = 1" in your settings. This is part of the sites framework.