I have a race condition in Celery. Inspired by this - http://ask.github.io/celery/cookbook/tasks.html#ensuring-a-task-is-only-executed-one-at-a-time I decided to use memcache to add locks to my tasks.
These are the changes I made:
python-memcached
# settings for memcache
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
'LOCATION': '127.0.0.1:11211',
}
}
After this I login to my shell and do the following
>>> import os
>>> import django
>>> from django.core.cache import cache
>>> os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings.base')
>>> cache
<django.core.cache.DefaultCacheProxy object at 0x101e4c860>
>>> cache.set('my_key', 'hello, world!', 30) #display nothing. No T/F
>>> cache.get('my_key') #Display nothing.
>>> from django.core.cache import caches
>>> caches['default']
<django.core.cache.backends.memcached.MemcachedCache object at 0x1048a5208>
>>> caches['default'].set('my_key', 'hello, world!', 30) #display nothing. No T/F
>>> caches['default'].get('my_key') #Display nothing.
also did pip install python-memcached
Using Python 3.6, Django==1.10.5
What am I doing wrong? Any help will be appreciated.
The problem was, memcached was killed for some reason and I was assumed it was still running. My bad.
Now it works all perfectly.
For anyone who is stuck on a similar problem you want to make sure you are still running memcached, try memcached -vv
Keeping this here for reference.
If your install is looking like mine is starting to look (in which case, you have my sympathies; for RHEL is one platform where django seems thinner on-the-ground than this-week's python):
# yum upgrade -y
yum-config-manager --add-repo=https://dl.fedoraproject.org/pub/epel/7/x86_64/
# yum install -y emacs-nox
yum install -y python{-sqlparse,36{,-{devel,pip,pytz,bcrypt}}} memcached
service memcached start
chkconfig memcached on
python36 \
-m pip install \
django \
pymemcache \
--no-deps --upgrade
then checking the status on memcache is as easy with the same commands we've been using for 20 years:
service memcached status
There are other commands one can use on RHEL/EL7 (#fridgeArt), but I prefer compatible workflows in spite of resume-driven differentiation. >:-(
Here's how to launch it with -vv under EL and debuntus too: https://stackoverflow.com/a/22239764/2066657
Related
I want to unit test a bunch of code that looks like this:
# First command. This will open reddit in your browser.
# -------------------------------------------------------------
if 'open reddit' in command:
url = 'https://www.reddit.com/'
if not runtest:
webbrowser.open(url)
print('Done!')
talktome.talkToMe('reddit is opening.')
if runtest:
return url
# -------------------------------------------------------------
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
I've written this unit test:
import unittest
import subprocess
from GreyMatter import julibrain
from SpeakAndHear import talktome
class TestBrain(unittest.TestCase):
def test_open_reddit(self):
test = True
testurl = julibrain.assistant('open reddit', 1, 2, test)
#subprocess.call(['pip', 'list', '|', 'grep', 'webbrowser'])
self.assertEqual(testurl, 'https://www.reddit.com/')
I invoke it on the command line:
python -m unittest test_julibrain.py
It gives me this output:
python -m unittest test_julibrain.py
Package Version
----------------- ----------
autopep8 1.5.2
beautifulsoup4 4.9.0
certifi 2020.4.5.1
chardet 3.0.4
click 7.1.1
future 0.18.2
gTTS 2.1.1
gTTS-token 1.1.3
idna 2.9
isort 4.3.21
lazy-object-proxy 1.4.3
mccabe 0.6.1
mock 4.0.1
MouseInfo 0.1.3
mypy 0.770
mypy-extensions 0.4.3
numpy 1.18.1
Pillow 7.1.1
pip 20.0.2
psutil 5.7.0
PyAudio 0.2.11
PyAutoGUI 0.9.50
pycodestyle 2.5.0
PyGetWindow 0.0.8
pylint 2.4.4
PyMsgBox 1.0.7
pyperclip 1.8.0
PyQt5 5.14.2
PyQt5-sip 12.7.2
PyRect 0.1.4
PyScreeze 0.1.26
python3-xlib 0.15
PyTweening 1.0.3
requests 2.23.0
setuptools 45.2.0
six 1.14.0
soupsieve 2.0
subprocess.run 0.0.8
typed-ast 1.4.1
typing-extensions 3.7.4.1
urllib3 1.25.9
vosk 0.3.3
wheel 0.34.2
wikipedia 1.4.0
wrapt 1.11.2
.
----------------------------------------------------------------------
Ran 1 test in 0.285s
OK
So I can test that the url is being set, but I don't know how to see if the webbrowser module is working. If I run the code, it the webbrowser opens. I don't want to open it. I'd like to get back some sort of status that says the module has been imported, a webbroser object can be instantiated, and there are no problems. After looking at the webbrowser doc, I have no idea how to do this. Any ideas? Thank you. I'm on Ubuntu 18.04, using conda and python 3.6.1. Cheers!
so for this web-scraping project i'm working on, I've been trying to separate some results from results.
basically if the title contains 指定されたページが見つかりません , i'll want to copy the url and write it to one fail.csv file. Anything else i'll want to copy the url and write it to sucess.csv
html = 'www.abc.com'
url = BeautifulSoup(html,'html.parser').title.string
pattern = re.compile(r' 指定されたページが見つかりません')
if pattern.finditer(url):
with open('fail.csv','w') as f:
cw=csv.writer
cw.writerow([url])
else:
move on, run some other codes and write to sucess.csv
However it seems that regex isn't recognising 指定されたページが見つかりません
Am i doing something wrong here or missing something here?
Thanks
Try
sudo pip3 install requests
sudo pip3 install beautifulsoup4
sudo pip3 install re
and under python3
import requests
import re
from bs4 import BeautifulSoup
r = requests.get('https://corp.rakuten.co.jp/careers/life/')
r.encoding='utf-8'
pattern = re.compile(r' 指定されたページが見つかりません')
url = BeautifulSoup(r.text,'html.parser').title.string
pattern.findall(url)
I am getting this import error from the scipy package. Please see requirements.txt for versions. This is only happening on Flask and not when I run the same code on ipython. Is there any reason why statsmodels doesn't work on flask but would work on ipython?
File "/env/lib/python3.7/site-packages/statsmodels/api.py", line 9, in
from . import regression
File "/env/lib/python3.7/site-packages/statsmodels/regression/init.py", line 1, in
from .linear_model import yule_walker
File "/env/lib/python3.7/site-packages/statsmodels/regression/linear_model.py", line 39, in
from scipy.linalg import toeplitz
File "/env/lib/python3.7/site-packages/scipy/init.py", line 156, in
from . import fft
File "/env/lib/python3.7/site-packages/scipy/fft/init.py", line 81, in
from ._helper import next_fast_len
File "/env/lib/python3.7/site-packages/scipy/fft/_helper.py", line 4, in
from . import _pocketfft
File "/env/lib/python3.7/site-packages/scipy/fft/_pocketfft/init.py", line 3, in
from .basic import *
File "/env/lib/python3.7/site-packages/scipy/fft/_pocketfft/basic.py", line 8, in
from . import pypocketfft as pfft
ImportError: SystemExit: 1
*This happens only in Flask. When i use statsmodels
I'm running it in flask and here are my complete list of packages:
attrs==19.3.0
autopep8==1.5
backcall==0.1.0
bleach==3.1.0
cachetools==4.0.0
cattrs==1.0.0
certifi==2019.11.28
chardet==3.0.4
Click==7.0
cloudstorage==0.10.0
cycler==0.10.0
decorator==4.4.1
defusedxml==0.6.0
entrypoints==0.3
Flask==1.1.1
Flask-Assets==2.0
Flask-CacheBuster==1.0.0
Flask-Cors==3.0.8
Flask-SQLAlchemy==2.4.1
flask-talisman==0.7.0
google-api-core==1.16.0
google-auth==1.11.0
google-cloud==0.34.0
google-cloud-core==1.2.0
google-cloud-storage==1.25.0
google-cloud-tasks==1.3.0
google-resumable-media==0.5.0
googleapis-common-protos==1.51.0
grpc-google-iam-v1==0.12.3
grpcio==1.26.0
idna==2.8
importlib-metadata==1.5.0
inflect==3.0.2
inflection==0.3.1
ipykernel==5.1.4
ipython==7.11.1
ipython-genutils==0.2.0
ipywidgets==7.5.1
itsdangerous==1.1.0
jedi==0.16.0
Jinja2==2.11.0
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==5.3.4
jupyter-console==6.1.0
jupyter-contrib-core==0.3.3
jupyter-contrib-nbextensions==0.5.1
jupyter-core==4.6.1
jupyter-highlight-selected-word==0.2.0
jupyter-latex-envs==1.4.6
jupyter-nbextensions-configurator==0.4.1
kiwisolver==1.1.0
looker-sdk==0.1.3b6
lxml==4.5.0
MarkupSafe==1.1.1
matplotlib==3.1.2
mistune==0.8.4
natural==0.2.0
nbconvert==5.6.1
nbformat==5.0.4
nltk==3.4.5
notebook==6.0.3
numpy==1.18.1
pandas==0.25.3
pandas-datareader==0.8.1
pandocfilters==1.4.2
parso==0.6.0
patsy==0.5.1
pexpect==4.8.0
pg8000==1.13.2
pickleshare==0.7.5
prometheus-client==0.7.1
prompt-toolkit==3.0.3
protobuf==3.11.2
psycopg2==2.8.4
ptyprocess==0.6.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.5.0
Pygments==2.5.2
pyparsing==2.4.6
pyrsistent==0.15.7
python-dateutil==2.8.1
python-magic==0.4.15
pytz==2019.3
PyYAML==5.3
pyzmq==18.1.1
qtconsole==4.6.0
requests==2.22.0
rsa==4.0
scipy==1.4.1
scramp==1.1.0
seasonal==0.3.1
Send2Trash==1.5.0
simple-salesforce==0.74.3
simplenlg==0.2.0
six==1.14.0
SQLAlchemy==1.3.13
statsmodels==0.11.0
stripe==2.42.0
terminado==0.8.3
testpath==0.4.4
tornado==6.0.3
traitlets==4.3.3
tzlocal==2.0.0
urllib3==1.25.8
wcwidth==0.1.8
webapp2==2.5.2
webassets==2.0
webencodings==0.5.1
Werkzeug==0.16.1
widgetsnbextension==3.5.1
zipp==2.1.0
no error in ipython, but i get an error when i run it in flask
Checkout this issues for more detail.
Turns out that statsmodels is dependent upon several packages being installed before it so that it can key on them to compile its own modules. Although I don't exactly know what causes this particular conflict with flask, I have faced this on python3.7 before and here is what worked for me:
If you need to clean out what you already have, you can uninstall with the following command:
pip3 uninstall statsmodels
Now install Flask
pip3 install Flask
Install the individual dependencies of statmodels first:
pip3 install numpy scipy patsy pandas
Now install statmodels:
pip3 install statsmodels
Important!!!
Make sure you are running a fresh virtual environment. To open a new one in linxux system:
python3 -m venv venv
source venv/bin/activate
Now install the dependencies as instructed above. Let me know if that fixed your problem. If not, you might need a better dependency resolver than pip. Poetry is one of them.
Just restart the server and the problem is solved!
I want to know as title says: how to download a package using python-apt API? Only download it, don't to install it, just like does the command:
apt-get download ${package_name}
I'm using python v2.7.5-5ubuntu3 with python-apt v0.9.3.5 on Ubuntu 14.04.
You can use this script
#!/usr/bin/python
import apt
import os, sys
if len(sys.argv) != 2:
print 'Usage: apt-download pkg_name'
sys.exit()
cache = apt.Cache()
pkg = cache[sys.argv[1]]
cmd = 'wget ' + pkg.candidate.uri
print cmd
os.system(cmd)
It works well with Python 2.7.8 and python-apt 0.9.3.11 but it should also work with your configuration.
I believe I've installed all of the dependencies correctly, but when I try to query the DB, the cursor returns zero results always.
>>> import pymssql
>>> conn = pymssql.connect(host='some_host', user='blah', password='blah', database='some_db')
>>> crsr = conn.cursor()
>>> crsr.execute('SELECT TOP 10 * from d_user')
>>> print crsr.fetchall()
[]
I know the query executed correctly against the DB, because I can see the query results in the freetds log files.
What's going on here?
Setup: python 2.7.5, current version of pymssql (pip installed), brew install freetds, Mac OSX 10.8.3.