I have a function in the views.py of a Django app, in which I am setting up setup function of PyCaret regression module:
from pycaret.regression import setup, compare_models, pull, get_logs
s = setup(data=df.copy(), target=selectedTarget, train_size=selectedTrainingPercentage,numeric_features= selectedFeatureColumns, silent =True, log_experiment=True)
print("Pycaret setup run")
and I get the following output where setup is not fully run, neither giving any error:
What can I try to resolve this?
Related
This is a follow-up to this question: How to stop flask application without using ctrl-c . The problem is that I didn't understand some of the terminology in the accepted answer since I'm totally new to this.
import dash
import dash_core_components as dcc
import dash_html_components as html
app = dash.Dash()
app.layout = html.Div(children=[
html.H1(children='Dash Tutorials'),
dcc.Graph()
])
if __name__ == '__main__':
app.run_server(debug=True)
How do I shut this down? My end goal is to run a plotly dashboard on a remote machine, but I'm testing it out on my local machine first.
I guess I'm supposed to "expose an endpoint" (have no idea what that means) via:
from flask import request
def shutdown_server():
func = request.environ.get('werkzeug.server.shutdown')
if func is None:
raise RuntimeError('Not running with the Werkzeug Server')
func()
#app.route('/shutdown', methods=['POST'])
def shutdown():
shutdown_server()
return 'Server shutting down...'
Where do I include the above code? Is it supposed to be included in the first block of code that I showed (i.e. the code that contains app.run_server command)? Is it supposed to be separate? And then what are the exact steps I need to take to shut down the server when I want?
Finally, are the steps to shut down the server the same whether I run the server on a local or remote machine?
Would really appreciate help!
The method in the linked answer, werkzeug.server.shutdown, only works with the development server. Creating a view function, with an assigned URL ("exposing an endpoint") to implement this shutdown function is a convenience thing, which won't work when deployed with a WSGI server like gunicorn.
Maybe that creates more questions than it answers:
I suggest familiarising yourself with Flask's wsgi-standalone deployment docs.
And then probably the gunicorn deployment guide. The monitoring section has a number of different examples of service monitors, which you can use with gunicorn allowing you to run the app in the background, start on reboot, etc.
Ultimately, starting and stopping the WSGI server is the responsibility of the service monitor and logic to do this probably shouldn't be coded into your app.
What works in both cases of
app.run_server(debug=True)
and
app.run_server(debug=False)
anywhere in the code is:
os.kill(os.getpid(), signal.SIGTERM)
(don't forget to import os and signal)
SIGTERM should cause a clean exit of the application.
Whenever a file imports import greengrasssdk the unit tests fail, because the module greengrass_commondoesn't exists on my local machine and I cannot install it via pip.
I am executing the tests with PyCharm. The Greengrass lambda, I try to test, doesn't execute locally because of the same dependency problem (same exception). But as soon as the lambda is pushed to greengrass it works fine.
Here is the exception:
import greengrasssdk
File "C:\Python27\lib\site-packages\greengrasssdk\__init__.py", line 6,
in <module>
from .Lambda import StreamingBody
File "C:\Python27\lib\site-packages\greengrasssdk\Lambda.py", line 10, in
<module>
from greengrass_common.function_arn_fields import FunctionArnFields
ImportError: No module named greengrass_common.function_arn_fields
A simplified code example is this:
import greengrasssdk
import logging
greengrass_iot_client = greengrasssdk.client('iot-data')
logger = logging.getLogger('logger')
def handler(event, context):
logger.info('Event handler invoked with event: ' + str(event))
I get the following error message on the test (The test is excluded in a test folder, but no other dependency issues have been shown yet - I write this because some developers put their tests into the python code file. I heard that that tests outside of the source code file could result in import issues. Though this case is different since it happens also in the original code file.)
import unittest
import mock
import function
class SimpleTest(unittest.TestCase):
# NONE OF THE THREE PATCH WORK Not in combination nor single
#mock.patch('greengrass_common')
#mock.patch('greengrass_common.function_arn_fields')
#mock.patch('greengrasssdk')
def test_that(self):
pass
The test case is empty for simplification.
I expect the greengrass_common code to be existing outside the Greengrass code for me to write unit tests.
I come from the java world, but spoke to a few python developers. We didn't really find a solution. (Except try catching the import in the production code) , but that seems like the first step into bad software quality in the whole project.
I am very grateful for ideas/solutions/approaches and guidance.
Thank you very much :).
You can import greengrasssdk conditionally if you detect that you're running on Greengrass and use boto3 if you're not. I wrote a quick tests that is working for me here that looks like this:
import socket
host = socket.gethostname()
client = None
# Create an IoT data client with Greengrass SDK on Greengrass, boto3 locally
if host == 'sandbox':
import greengrasssdk
client = greengrasssdk.client('iot-data')
else:
import boto3
client = boto3.client('iot-data')
Essentially check if the hostname is reported to be 'sandbox' and then use the Greengrass SDK, otherwise use boto3.
Had this problem as well and use the patcher in my conftest.py file to patch over the Greengrass SDK e.g.
conftest.py
import pytest
from mock import MagicMock, patch
MockGreengrassSdk = MagicMock()
modules = {
"greengrasssdk": MockGreengrassSdk
}
patcher = patch.dict("sys.modules", modules)
patcher.start()
Once you have defined MockGreengrassSdk as a MagicMock you can then mock out any method you want on it. The only thing I am not sure on is how you would change that per test if you wanted different results from the SDK per test.
The way I have got around that is to create a facade over the greengrasssdk which I can then mock out the methods on. In this case that facade must import greengrasssdk which is where the code above will stop your tests from failing.
In summary:
Create a facade over the greengrasssdk methods you want to use.
In the facade import the greengrasssdk.
Use the patcher code in a conftest.py file (as above) to patch over the Greengrass SDK.
Hope that helps.
I just ran into the same problem and for me it worked by surrounding the import with a try except statement.
try:
import greengrasssdk
client = greengrasssdk.client('iot-data')
except Exception as e:
import boto3
client = boto3.client('iot-data')
Django 1.7+ has AppConfig.ready (docs), however it seems to be running multiple times with Django Channels. How can I ensure that the code runs exactly once, even with multiple workers? I'm searching for a solution that works both with the dev server and with daphne.
Here is something I want to achieve:
from django.apps import AppConfig
from channels import Channel
class MyAppConfig(AppConfig):
name = 'myapp'
def ready(self):
# if thisIsTheFirstWorker:
Channel('mychannel').send({
'text': 'message to be sent only once',
})
This is not a very clean solution, but the following works: make mychannel handled by a single worker (use --exclude-channels=mychannel on other workers) and create a global variable to check if it's the first invocation of the setup code. It works, since it's the same process.
I'm still looking for a cleaner solution.
I've noticed that on occasions where I've run my django project without the PostgreSQL server available, the errors produced are fairly cryptic and often appear to be generated by deep django internals as these are the functions actually connecting to the backend.
Is there a simple clean(and DRY) way to test the server is running.
Where is the best place to put project level start up checks?
You can register a signal on class-prepared.
https://docs.djangoproject.com/en/dev/ref/signals/#class-prepared
Than try executing custom sql directly.
https://docs.djangoproject.com/en/dev/topics/db/sql/#executing-custom-sql-directly
If it fails raise your custom exception.
import time
from django.db import connections
from django.db.utils import OperationalError
self.stdout.write('Waiting for database...')
db_conn = None
while not db_conn:
try:
db_conn = connections['default']
except OperationalError:
self.stdout.write('Database unavailable, waiting 1 second...')
time.sleep(1)
self.stdout.write(self.style.SUCCESS('Database available!'))
you can use this snippet where you need to.
befor accessing database and making any queries you must check if the database is up or not
I'm running some basic test code, with web.py and GAE (Windows 7, Python27). The form enables messages to be posted to the datastore. When I stop the app and run it again, any data posted previously has disappeared. Adding entities manually using the admin (http://localhost:8080/_ah/admin/datastore) has the same problem.
I tried setting the path in the Application Settings using Extra flags:
--datastore_path=D:/path/to/app/
(Wasn't sure about syntax there). It had no effect. I searched my computer for *.datastore, and couldn't find any files, either, which seems suspect, although the data is obviously being stored somewhere for the duration of the app running.
from google.appengine.ext import db
import web
urls = (
'/', 'index',
'/note', 'note',
'/crash', 'crash'
)
render = web.template.render('templates/')
class Note(db.Model):
content = db.StringProperty(multiline=True)
date = db.DateTimeProperty(auto_now_add=True)
class index:
def GET(self):
notes = db.GqlQuery("SELECT * FROM Note ORDER BY date DESC LIMIT 10")
return render.index(notes)
class note:
def POST(self):
i = web.input('content')
note = Note()
note.content = i.content
note.put()
return web.seeother('/')
class crash:
def GET(self):
import logging
logging.error('test')
crash
app = web.application(urls, globals())
def main():
app.cgirun()
if __name__ == '__main__':
main()
UPDATE:
When I run it via command line, I get the following:
WARNING 2012-04-06 19:07:31,266 rdbms_mysqldb.py:74] The rdbms API is not available because the MySQLdb library could not be loaded.
INFO 2012-04-06 19:07:31,778 appengine_rpc.py:160] Server: appengine.google.com
WARNING 2012-04-06 19:07:31,783 datastore_file_stub.py:513] Could not read datastore data from c:\users\amy\appdata\local\temp\dev_appserver.datastore
WARNING 2012-04-06 19:07:31,851 dev_appserver.py:3394] Could not initialize images API; you are likely missing the Python "PIL" module. ImportError: No module named _imaging
INFO 2012-04-06 19:07:32,052 dev_appserver_multiprocess.py:647] Running application dev~palimpsest01 on port 8080: http://localhost:8080
INFO 2012-04-06 19:07:32,052 dev_appserver_multiprocess.py:649] Admin console is available at: http://localhost:8080/_ah/admin
Suggesting that the datastore... didn't install properly?
As of 1.6.4, we stopped saving the datastore after every write. This method did not work when simulating the transactional model found in the High Replication Datastore (you would lose the last couple writes). It is also horribly inefficient. We changed it so the datastore dev stub flushes all writes and saves its state on shut down. It sounds like the dev_appserver is not shutting down correctly. You should see:
Applying all pending transactions and saving the datastore
in the logs when shutting down the server (see source code and source code). If you don't, it means that the dev_appserver is not being shut down cleanly (with a TERM signal or KeyInterrupt).