Logging module is not working for Flask API on Apache - flask

I have deployed FLASK API along with Apache 2.0 server on Windows, and which is working fine.
Now I am in process of introducing the logs on application level with the help of Python logging module, which is also working as expected when I run Flask alone without Apache. But it is not working along with Apache.
I have 3 loggers to log the details for application restart, successful request and responses, and unsuccessful requests, error details and the responses. Everything works fine without Apache, so I am missing something or doing something wrong in the configuration level.
I have browsed for solution but couldn't get any solution material, I have already invested 3 to 4 days without any result.
Please throw some light on this, this is my first API deployment in production. Having limited knowledge in the production configurations.
API structure
daai_api
-> apps
-> customers
- name_matching_program.py
- routes.py
-> wsgi_scripts
- customers.wsgi
- __init__.py ----> this is where the logging configuration invoked
-> logs
- api_access
- api_error
- api_restart
- .env
- config.py
- **logging.yml**
Am using the YAML file (logging.yml) for configuration which is listed below
##### logging.yml file #####
version: 1
disable_existing_loggers: true
formatters:
standard:
format: '%(asctime)s - %(levelname)s - [%(filename)s:%(lineno)d] - %(message)s'
access:
format: '%(asctime)s - %(levelname)s - [%(filename)s:%(lineno)d] - %(threadName)s - %
(message)s'
error:
format: '%(asctime)s - %(levelname)s - [%(filename)s:%(funcName)s():%(lineno)d] - %
(threadName)s - %(message)s'
handlers:
api_start_file_handler:
class: logging.FileHandler
level: DEBUG
formatter: standard
filename: logs/api_restart/api_restart.log
#filename: api_restart.log
encoding: utf8
access_file_handler:
class: logging.handlers.TimedRotatingFileHandler
level: INFO
formatter: access
filename: logs/api_access/api_access.log
#filename: access.log
when: 'D'
interval: 1
backupCount: 30
error_file_handler:
class: logging.handlers.TimedRotatingFileHandler
level: ERROR
formatter: error
filename: logs/api_error/api_error.log
#filename: error.log
when: 'D'
interval: 1
backupCount: 30
loggers:
access_logger:
level: INFO
handlers: [access_file_handler]
error_logger:
level: ERROR
handlers: [error_file_handler]
restart_logger:
level: DEBUG
handlers: [api_start_file_handler]
Using the app factory to start the Flask application, and setting the logging configuration details in the init.py file
##### daai_api/apps/__init__.py #####
from flask import Flask
from apps.customers.routes import customer
from config import config
import yaml
import logging
from logging.config import dictConfig
import os
def create_app(apps_config='production'):
logging_config = os.path.abspath('../daai_api/logging.yml')
if os.path.exists(logging_config):
with open(logging_config, 'r') as f:
#lgconfig = yaml.safe_load(f.read())
lgconfig = yaml.load(f, Loader=yaml.FullLoader)
dictConfig(lgconfig)
app = Flask(__name__)
app.config.from_object(config[apps_config])
app.register_blueprint(customer)
logger = logging.getLogger('restart_logger')
logger.info('App started as app factory...')
return app
The WSGI file of the API
##### customers.wsgi #####
import sys
sys.path.insert(0, 'D:\\DAAI\daai_api')
try:
from apps import create_app
except Exception as e:
print('Import error: ', e)
raise
application = create_app('production')
Apache configuration:
##### cutomers.conf #####
<VirtualHost *:1234>
ServerName 12.345.678.900
WSGIScriptAlias / D:/DAAI/daai_api/apps/wsgi_scripts/customers.wsgi
WSGIApplicationGroup %{GLOBAL}
<Directory D:/DAAI/daai_api/apps>
#WSGIProcessGroup D:/DAAI/daai_api/apps
WSGIApplicationGroup %{GLOBAL}
Require all granted
</Directory>
</VirtualHost>
I am not sure what information needs to be provided from Apache httpd.conf. Please reply if any specific information is required from httpd.conf.

Related

flask routes not working on gae application

I have the following directory structure for my GAE project:
endpoints/
insights/
insights.py
init.py
lib/
__init__.py
insights.yaml
so in order to get access to the third libraries within lib folder I add the next code in __init__.py file.
import os
import sys
def add_lib_path():
lib_directory = os.path.dirname(os.path.abspath(__file__))
if lib_directory not in sys.path:
sys.path.insert(0, lib_directory)
and I added this code in the insights.py file before the import statements:
from lib import add_lib_path
add_lib_path()
the problem is that now I can import third libraries correctly but my #app.route('/something', methods=['POST']) are not working.
I send a post request and it returns status 200 but it doesn't go inside my #app.route code, I can actually send any route and it just pass out returning 200 but not data and not error.
My imports look like this:
from lib import add_lib_path
add_lib_path()
from flask import Flask, request
And my code inside #app.route('/something', methods=['POST']) looks like this:
def someDef():
some code ...
return response
my yaml file looks like this:
runtime: python27
api_version: 1
threadsafe: false
service: insights
handlers:
- url: /.*
script: endpoints/insights/insights.py
libraries:
- name: ssl
version: latest
Any suggestions about this? Thanks in advance!
The wildcard URL handler in your app.yaml is intercepting the /something post:
handlers:
- url: /.*
script: application.app
You will need to either map out individual urls in your app, or make unique url set for your insights, like /insights/.* in app.yaml. Either way, you can't have a catch-all url handler in your app.yaml if there are other urls you want to give special treatment to, like sending to a separate service.
ok I just changed in the app.yaml file the threadsafe to true, then I added the script like this endpoints.insights.insights.app, using Python module path (with dots not slash) and in my insights.py file I changed #app.route('/something', methods=['POST']) to #app.route('/insights/something', methods=['POST']) .. I added the complete URL that I defined in the app.yaml file and now it's working.
Thanks to #GAEfan for the help, I'll accept the GAEfan answer because it helped me a lot

Django Static Files on Google Cloud Storage

I copied google cloud sample Django project for google appengine. Then, I modified settings.py for storing all static files on google storage. I added this settings
DEFAULT_FILE_STORAGE = 'storages.backends.gs.GSBotoStorage'
GS_ACCESS_KEY_ID = 'YourID'
GS_SECRET_ACCESS_KEY = 'YourKEY'
GS_BUCKET_NAME = 'YourBucket'
STATICFILES_STORAGE = 'storages.backends.gs.GSBotoStorage'
On local machine everything is working perfectly but when I deploy it to google appengine static file problem appears. Django can't find static files which in special google storage bucket. This is the error message:
ImportError at /admin/login/ cannot import name SpooledTemporaryFile
in app.yaml file, settings about static files:
handlers:
- url: /static
static_dir: static/
- url: .*
script: mysite.wsgi.application
I am new at google cloud and i can't figure it out where I am doing wrong.
Edit: I think, I must modify app.yaml file to say appengine "Hey, use another bucket don't serve static file yourself".
Tere is more detailed error output:
ImportError at /admin/ cannot import name SpooledTemporaryFile Request
Method: GET Request URL: https://sbminteractive.appspot.com/admin/
Django Version:
1.8 Exception Type: ImportError Exception Value: cannot import name SpooledTemporaryFile Exception Location:
/base/data/home/apps/e~sbminteractive/20170414t175601.400554512166425273/lib/storages/backends/s3boto.py
in , line 6 Python Executable:
/base/data/home/runtimes/python27/python27_dist/python Python Version:
2.7.5 Python Path: ['/base/data/home/apps/e~sbminteractive/20170414t175601.400554512166425273',
'/base/data/home/apps/e~sbminteractive/20170414t175601.400554512166425273/lib',
'/base/data/home/runtimes/python27/python27_dist/lib/python27.zip',
'/base/data/home/runtimes/python27/python27_dist/lib/python2.7',
'/base/data/home/runtimes/python27/python27_dist/lib/python2.7/plat-linux2',
'/base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-tk',
'/base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-old',
'/base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-dynload',
'/base/data/home/runtimes/python27/python27_dist/lib/python2.7/site-packages',
'/base/data/home/runtimes/python27/python27_lib/versions/1',
'/base/data/home/runtimes/python27/python27_lib/versions/third_party/MySQLdb-1.2.5',
'/base/data/home/runtimes/python27/python27_lib/versions/third_party/protorpc-1.0',
'/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3',
'/base/data/home/runtimes/python27/python27_lib/versions/third_party/webob-1.1.1',
'/base/data/home/runtimes/python27/python27_lib/versions/third_party/yaml-3.10']
Server time: Fri, 14 Apr 2017 14:57:51 +0000

running cron in google app engine gives 404 error

My cronjob does not run when I click Run Now in http://localhost:8000/cron
My app contains the following files
hello.py
import webapp2
class MainPage(webapp2.RequestHandler):
def get(self):
self.response.headers["Content-Type"] = "text/plain"\
self.response.write("Congratulations, it's a web app!")
routes = [('/', MainPage)]
my_app = webapp2.WSGIApplication(routes, debug=True)
app.yaml
application: hello
version: 1
runtime: python27
api_version: 1
threadsafe: false
handlers:
- url: /.*
script: hello.my_app
- url: /tasks/summary
script: hello.application
cron_script.py
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
class CronTask(webapp.RequestHandler):
def get(self):
f = open("test.txt","w") #opens file with name of "test.txt"
f.write("I am a test file.")
f.write("Maybe someday, he will promote me to a real file.")
f.write("Man, I long to be a real file")
f.write("and hang out with all my new real file friends.")
f.close()
application = webapp.WSGIApplication([('/tasks/summary', CronTask)],
debug=True)
if __name__ == '__main__':
run_wsgi_app(application)
cron.yaml
cron:
- description: daily summary job
url: /tasks/summary
target: beta
schedule: every 1 minutes
Your cron.yaml file will make requests to /tasks/summary.
Your app.yaml directs them to your hello.application.
But your matching hello.py file doesn't have a matching route for /tasks/summary.
You could expand your existing route pattern to match that path:
routes = [('/.*', MainPage)]
But more likely you'll add specific route for it (and you'll need to add the matching handler code for it as well):
routes = [('/tasks/summary', CronHandler)]

The app returned an error when the Google Cloud Endpoints server attempted to communicate with it

Hi I'm new to Google App Engine and Cloud Endpoints.
I have an app engine that serves web application.
And I want to use it with Cloud Endpoints.
So I just added source codes from here:
https://cloud.google.com/appengine/docs/python/endpoints/getstarted/backend/write_api
But I cannot deploy app and get this error message:
Checking if Endpoints configuration has been updated.
07:13 AM Failed to update Endpoints configuration. The app returned an error when the Google Cloud Endpoints server attempted to communicate with it.
07:13 AM See the deployment troubleshooting documentation for more information: https://developers.google.com/appengine/docs/python/endpoints/test_deploy#troubleshooting_a_deployment_failure
And this is my source codes
import endpoints
from protorpc import messages
from protorpc import message_types
from protorpc import remote
package = 'Hello'
class Greeting(messages.Message):
"""Greeting that stores a message."""
message = messages.StringField(1)
class GreetingCollection(messages.Message):
"""Collection of Greetings."""
items = messages.MessageField(Greeting, 1, repeated=True)
STORED_GREETINGS = GreetingCollection(items=[
Greeting(message='hello world!'),
Greeting(message='goodbye world!'),
])
#endpoints.api(name='helloworld', version='v1')
class HelloWorldApi(remote.Service):
"""Helloworld API v1."""
#endpoints.method(message_types.VoidMessage, GreetingCollection,
path='hellogreeting', http_method='GET',
name='greetings.listGreeting')
def greetings_list(self, unused_request):
return STORED_GREETINGS
ID_RESOURCE = endpoints.ResourceContainer(
message_types.VoidMessage,
id=messages.IntegerField(1, variant=messages.Variant.INT32))
#endpoints.method(ID_RESOURCE, Greeting,
path='hellogreeting/{id}', http_method='GET',
name='greetings.getGreeting')
def greeting_get(self, request):
try:
return STORED_GREETINGS.items[request.id]
except (IndexError, TypeError):
raise endpoints.NotFoundException('Greeting %s not found.' %
(request.id,))
APPLICATION = endpoints.api_server([HelloWorldApi])
exactly same with tutorial
and my app.yaml
application: my application id
version: application version
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /data
script: MLP.app
login: admin
- url: /insert_db
script: MLP.app
login: admin
- url: /validete
script: MLP.app
login: admin
- url: /stylesheets
static_dir: stylesheets
- url: /js
static_dir: js
- url: /.*
script: MLP.app
- url: /_ah/spi/.*
script: MLP_mobile_backend.APPLICATION
libraries:
- name: webapp2
version: latest
- name: jinja2
version: latest
- name: pycrypto
version: latest
- name: endpoints
version: 1.0
MLP.py is just web service with webapp2
How can I solve the problem?
From the app.yaml line script: MLP_mobile_backend.APPLICATION, it means your code sample must live in a MLP_mobile_backend.py file at the root of your app engine project
.
├── app.yaml
├── MLP.py
└── MLP_mobile_backend.py
with an endpoint api server named APPLICATION defined in that file just as in your code sample above.
APPLICATION = endpoints.api_server([HelloWorldApi])
After those requirements are fulfilled, this line points to how to access your endpoints:
#endpoints.api(name='helloworld', version='v1')
So for example, let's say you're running your devserver's module on port 8080 and you want to access the hellogreeting path:
#endpoints.method(message_types.VoidMessage, GreetingCollection,
path='hellogreeting', http_method='GET',
name='greetings.listGreeting')
you can use cURL or httpie or your browser to locally navigate to the devserver's default module at http://localhost:8080/_ah/api/helloworld/v1/hellogreeting or with Google's API explorer at http://localhost:8080/_ah/api/explorer which should result in this response body:
{
"items": [
{
"message": "hello world!"
},
{
"message": "goodbye world!"
}
]
}
Once you get it set up locally, you can then deploy it.
More here

Django/Mod_WSGI 'client denied by server configuration'

I'm attempting to set up Apache 2.2, Django 1.1.2 and Gentoo. I wish to serve my project with the address: /comics
I followed the mod_wsgi directions in the django documentation to the letter, coming up with these files:
/etc/apache2/modules.d/70_mod_wsgi.conf
<IfDefine WSGI>
LoadModule wsgi_module modules/mod_wsgi.so
</IfDefine>
WSGIScriptAlias /comics /home/****/django/comicky/apache/django.wsgi
and
/home/****/django/comicky/apache/django.wsgi
import os
import sys
sys.path.append('/home/****/django')
os.environ['DJANGO_SETTINGS_MODULE'] = 'comicky.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
However, when I attempt to load the page, I get this in /var/log/apache2/error_log:
client denied by server configuration: /home/****/django/comicky/apache/django.wsgi
Any ideas?
Sounds like a permissions issue from your conf file. See, for example:
Apache Error: Client denied by server configuration
Make sure your conf file matches this: conf sample file. I had to add the wsgi.py at the end to make mine work