I migrated from google.appengine.ext.webapp and ran into an issue with webapp2.WSGIApplication. I am using Django as the backend
the main part looks like this
application =webapp2.WSGIApplication([
('/warmup', warmupHandler)
('/api/layer', LayerService),
debug=False)
def main():
google.appengine.ext.webapp.util.run_wsgi_app(application)
if __name__ == '__main__':
main()
I have tried using this code snippet provided by google cloud as a replacement and it did no help.
def wsgi_middleware(app):
client = ndb.Client()
def middleware(request):
with client.context()
return app(request)
return middleware
They also have code snippits but i think this is for Flask, and i use django
def ndb_wsgi_middleware(wsgi_app):
def middleware(environ, start_response):
with client.context():
return wsgi_app(environ, start_response)
return middleware
In the current setup I have, all I get is a 404 error. Meaning its not picking up the templates
Related
I want to use scrapy spider in Django views and I tried using CrawlRunner and CrawlProcess but there are problems, views are synced and further crawler does not return a response directly
I tried a few ways:
# Core imports.
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
# Third-party imports.
from rest_framework.views import APIView
from rest_framework.response import Response
# Local imports.
from scrapy_project.spiders.google import GoogleSpider
class ForFunAPIView(APIView):
def get(self, *args, **kwargs):
process = CrawlerProcess(get_project_settings())
process.crawl(GoogleSpider)
process.start()
return Response('ok')
is there any solution to handle that and run spider directly in other scripts or projects without using DjangoItem pipeline?
you didn't really specify what the problems are, however, I guess the problem is that you need to return the Response immediately, and leave the heavy call aka function to run in the background, you can alter your code as following, to use the Threading module
from threading import Thread
class ForFunAPIView(APIView):
def get(self, *args, **kwargs):
process = CrawlerProcess(get_project_settings())
process.crawl(GoogleSpider)
thread = Thread(target=process.start)
thread.start()
return Response('ok')
after a while of searching for this topic, I found a good explanation here: Building a RESTful Flask API for Scrapy
This simple web service works if I run it by hand but in my unit tests I get a 404 not found page as my response, preventing me to properly test the application.
normal behavior:
Folder structure:
/SRC
--web_application.py
/UNIT_TESTS
--test_wab_application.py
web_application.py
from flask import Flask, request, jsonify, send_from_directory
from python.Greeting import Greeting
application = Flask(__name__)
def create_app(test_config=None):
# create and configure the app
app = Flask(__name__, instance_relative_config=True)
app.config.from_mapping(
SECRET_KEY='mega_developer',
DATABASE=os.path.join(app.instance_path, 'web_application.sqlite'),
)
try:
os.makedirs(app.instance_path)
except OSError:
pass
return app
#application.route('/greetings', methods=['GET', 'POST'])
def hello():
# GET: for url manipulation #
if request.method == 'GET':
return jsonify(hello = request.args.get('name', 'world', str))
test_web_application.py
import tempfile
import pytest
import web_application
class TestWebApplication:
app = web_application.create_app() # container object for test applications #
#pytest.fixture
def initialize_app(self):
app = web_application.create_app()
app.config['TESTING'] = True
app.config['DEBUG'] = False
app.config['WTF_CSRF_ENABLED'] = False
app.config['DATABASE'] = tempfile.mkstemp()
app.testing = True
self.app = app
def test_hello_get(self, initialize_app):
with self.app.test_client() as client:
response = client.get('/greetings?name=Rick Sanchez')
assert response.status_code == 200
test results (most relevant part only):
Launching pytest with arguments test_web_application.py::TestWebApplication::test_hello_get in C:\Users\Xrenyn\Documents\Projekte\Studium_Anhalt\QA&Chatbots Exercises\Exercise 2 - Web Service Basics\UNIT_TESTS
============================= test session starts =============================
platform win32 -- Python 3.8.0, pytest-5.2.2, py-1.8.0, pluggy-0.13.0 -- C:\Users\Xrenyn\Documents\Projekte\Studium_Anhalt\QA&Chatbots Exercises\Exercise 2 - Web Service Basics\VENV\Scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\Xrenyn\Documents\Projekte\Studium_Anhalt\QA&Chatbots Exercises\Exercise 2 - Web Service Basics\UNIT_TESTS
collecting ... collected 1 item
test_web_application.py::TestWebApplication::test_hello_get FAILED [100%]
test_web_application.py:21 (TestWebApplication.test_hello_get)
404 != 200
Expected :200
Actual :404
So far I have tested various alternative routing paths for the client.get() method in test-web_application.py , including combinations like '/SRC/greetings?name=Rick Sanchez' or '../SRC/greetings?name=Rick Sanchez', but all to no different effect.
Do you have any idea on what I might be doing wrong or how I could get access to my web services' functions from within unit tests?
I think the problem is that you are creating two Flask instances. One with the name application that you add hello route to, and the second one using the create_app function. You need to create a test client using the application instance (the one you added the hello route to).
Can you import application and then obtain the client using application.test_client()?
Sample solution:
import pytest
from web_application import application
#pytest.fixture
def client():
with application.test_client() as client:
yield client
class TestSomething:
def test_this(self, client):
res = client.get('/greetings?name=Rick Sanchez')
assert res.status_code == 200
Checkout the official docs on testing.
I am attempting to create a Flask middleware in order to make py2neo transactions atomic. First I got a working outside of application context error, and I tried to apply this solution, as seen in the following code:
from flask import g
from py2neo import Graph
def get_db():
return Graph(password="secret")
class TransactionMiddleware(object):
def __init__(self, app):
self.app = app
with app.app_context(): # Error raises here.
g.graph = get_db()
g.transaction = g.graph.begin()
def __call__(self, environ, start_response):
try:
app_status = self.app(environ, start_response)
# some code
return app_status
except BaseException:
g.transaction.rollback()
raise
else:
g.transaction.commit()
But I got this error: AttributeError: 'function' object has no attribute 'app_context'.
I don't know if the solution I'm trying is not suitable for my case, or what is the problem.
You are in a WSGI middleware, so what gets passed in as an app is actually a method called Flask.wsgi_app the results of which you a later supposed to return from your __call__ handler.
Perhaps you can simply import your actual flask app and use app_context on that, as in:
from app import app # or whatever module it's in
# and then
class TransactionMiddleware(object):
...
def __call__(self, environ, start_response):
with app.app_context():
...
I would also consider just instantiating your Graph somehere on a module level. Perhaps next to your flask app instantiation and not attaching it to g. This way you can use it without application context:
# app.py
flask = Flask(__main__)
db = Graph(password="secret")
You can use it by directly importing:
# middleware.py
from app import db
I have the following three files.
app.py
from flask_restful import Api
from lib import globals
from flask import Flask
from flask.ext.cache import Cache
globals.algos_app = Flask(__name__)
#cache in file system
globals.cache = Cache(globals.algos_app, config={'CACHE_TYPE': 'filesystem', 'CACHE_DIR': '/tmp'})
api = Api(globals.algos_app)
api.add_resource(Test, '/test')
if __name__ == '__main__':
globals.algos_app.run(host='0.0.0.0', debug=True)
globals.py
global algos_app
global cache
Test.py
from flask_restful import Resource
from lib import globals
from flask_restful import Resource
import time
class Test(Resource):
def get(self):
return self.someMethod()
def post(self):
globals.cache.clear()
return self.someMethod()
#globals.cache.cached()
def someMethod(self):
return str(time.ctime())
I have a GET method which needs to the value from the cache and a POST method which updates the cache by first clearing the cache.
However, no matter I call the GET or the POST method, it always gives me the value from the cache.
PS: At the moment I am simply testing on the development server however I do need to deploy it using WSGI later.
I am not sure if it is the best way, but I did it using the following way.
class Test(Resource):
def get(self):
return globals.cache.get('curr_time')
def post(self):
result = self.someMethod()
globals.cache.set('curr_time', result, timeout=3600)
def someMethod(self):
return str(time.ctime())
I can run the code below both locally and on openshift with success, but with very different results. When I load '/' in both browsers I see that the code is working.
When I load '/main' in the browser it works locally, and it throws a 500 error(with no debug information) in openshift.
when I load '/error' in the browser I get debug information locally, and just 500 again in openshift.
How can I see debug information instead of a useless error message on openshift?
import watercore
from flask import Flask
app = Flask(__name__)
#app.errorhandler(404)
def page_not_found(e):
return '404'
#app.errorhandler(500)
def errorhandle(e):
return '500'
#app.route('/error')
'force a 500 error'
def testerrors(e):
return watercore.thisdefinitiondoesntexist()
#app.route('/')
def index():
return 'working'
#app.route('/main')
def mainpage():
return watercore.listall()
#app.route('/site/<sitename>')
def site(sitename):
if watercore.listall(sitename) == True:
return sitename
if __name__ == '__main__':
app.debug = True
app.run()
Openshift probably runs your app using wsgi, so your main is never run. To enable debug set the it outside if __name__ == '__main__':.
def create_app():
app = Flask(__name__)
app.debug = True
return app
app = create_app()
Note that you should never do this for an actual production app.
I found this on the flask quickstart:
Attention
Even though the interactive debugger does not work in forking
environments (which makes it nearly impossible to use on production
servers)
Which leads me to believe that the debugger will not work on openshift at all.
I've already switched to django!