Plone 4.3.3
I am trying to get some extra python modules available to my plone scripts. 'Net search led me to collective.localfunctions, which supposedly demonstrates how to lighten up restricted python. I installed per the instructions:
git clone https://github.com/collective/collective.localfunctions
easy_install collective.localfunctions
# not installed where plone can get to it, so...
copy from system python lib to the plone build-cache/eggs director
added collective.localfunctions to eggs = and zcml =
Running buildout and restarting results in a non-responsive site, nothing is listening on 8080. There is nothing useful in var/log, either.
So, bottom line question is, how do I enable python modules, specifically re and datetime?
The proper way to allow extra modules to be used in restricted python is outlined in the answer to this Importing ping module in RestrictedPython script in Plone
Related
We are considering using Django for an application which can be seen mostly as a desktop application. That is, it is not to be installed on our servers, but directly on our customer's laptops (but still accessed through their browser on localhost.)
Whilst it is OK to require Python to be installed, it is not to require to install Django, or other dependencies.
It would probably run using the internal runserver. And most likely not use SQL (or only SQLite if necessary.)
Is it possible to "package" the application with Django and all dependencies? What are the options here? Ideally, the install guide would read:
install Python, if not done yet
download and unzip our application archive
go to that dir and execute python3 manage.py runserver
Note I: The goal is NOT to package a publicly available website as an app.
Note II: The only related questions I have found are several years old (10 and 7 years old resp.)
Note III: The customers use Linux, Mac and Windows.
Problem
I am attempting to install Python 2.7.16, openpyxl, and pyinstaller onto a Windows 10 machine that is offline for security reasons. To clarify, I have a mapped network drive on there from which I can transfer the files I need to use.
Question
What is the best way to go about this? I currently have a .msi Python installation file directly from their website. The packages I need are packaged as .tar.gz files. I currently have those on my windows machine, but do not want to proceed until I know for sure what I need to do. Also, do I need to do anything for dependencies? If so, how do I find the dependencies for the packages I need?
Side Notes
The version of Python (2.7.16) comes with pip. Not sure if that makes a difference. Downloading and transferring things requires me to ask my admin, for him to download the files, and then transfer them to my drive so I can have them on my computer. If able, I would like to do this in as little attempts as possible.
Useful links
Python: https://www.python.org/downloads/release/python-2716/
openpyxl: https://pypi.org/project/openpyxl/#files
pyinstaller: https://pypi.org/project/PyInstaller/#files
My solution would be to seek out the offline versions of the python and pip installer and follow this guide
Also a great tip: try the complete procedure (the installing of the required software) on a seperate pc which you have disconnected and do the installation. Note everything you have to do to get it working and use those instruction on your originally intended machine. This will prevent you from having to go back and forth and scratch your head while installing on the target machine.
Please note that I have NO idea how python works and this is just a hunch from me as a programmer.
Installing Python and packages on an Offline Machine: A Comprehensive Guide
The Environment
Let us begin by defining the environment in which this guide may be of some great use. If your situation can be described by one or more of the following, you might have great results following this guide...
The machine you are developing on is offline. (No connection to the internet)
You need to develop and run Python on the machine that is completely offline.
If this sounds like you, read the following cases in which a few minor details may make a big difference in getting you started.
Case1:
You are not allowed to plug in any external media devices into the offline machine. This includes but is not limited to a USB, CD, floppy disk, or any other removable media that may be of some use in helping you transfer Python files to the offline machine.
You are allowed to map a network drive (somewhere else on the local network). This would fix the problem mentioned in number one with removable media.
Answer: In this case, just proceed with the guide, as this was my case and I will explain in detail how I solved my problem.
Case2:
There is no physical way to transfer files onto the development machine that is offline.
Answer: If this is your case, you need to get in touch with the admin team who handles the software on your development machine. Direct them to this guide to proceed.
Let's Get Started
Warning A:
The following must be performed on a computer with an internet connection. It is impossible to download things from any website without an internet connection.
Warning B:
There is a longer way, and there is a shorter way to do the following. To avoid the longer way, you must be able to install python on a different machine that is online. This can be the same machine that you are using to download the packages and python version, or it can even be a home machine. This can be any machine in the world that is on the internet. It's sole purpose will be to help you identify the dependencies of each package.
Installing Python
Visit the python website and identify the version you want. 2.7.9 and up is recommended for this guide. Download the file for your specific system.
Python 2.7.9 : https://www.python.org/downloads/release/python-279/
Python 3.7.3 : https://www.python.org/downloads/release/python-373/
The reason I provided Python 2.7.9 is because that is the earliest 2.7.x version that comes with pip (a package manager).
Visit the python package index to locate the packages you will be using in your python project. https://pypi.org/
Search the package you need, go to the downloads, and get the (.tar.gz) file. Not the .whl files unless you know what you are doing with those.
Tip: If you want to keep track of the packages you are installing, I suggest you put them all in one folder somewhere you can find, or just write them down on paper.
Unpack the .tar.gz package files. You can get rid of the .tar.gz once you unpack them as they will not needed any longer.
Install the version of python that you downloaded for your system in step 1 above.
(This may just be running the .msi file for windows or unpacking some files for linux) If you are not sure how, just look at this brilliant guide
https://realpython.com/installing-python/
Now you should be able to go to your terminal and type "python" and get the python interpreter to open up. If you get a "cannot find python command" you need to setup your path variable.
Windows guide: https://geek-university.com/python/add-python-to-the-windows-path/
Linux guide: https://www.tutorialspoint.com/python/python_environment.htm
Your python installation is done! And your packages should also be ready to install!
Installing Python Packages
What you need to know here is that MOST all python packages have dependencies, which are other packages which packages need installed before they can be installed. If you need more explanation on dependencies, read here: https://www.fullstackpython.com/application-dependencies.html
Before proceeding be sure to add the Python/Scripts folder to your path variable too or pip will not work. Follow this link for instructions. https://appuals.com/fix-pip-is-not-recognized-as-an-internal-or-external-command/
Install packages using pip install [package_name] for every package you need, on your machine that is on the internet, and then do a pip freeze to see all the packages installed.
Once you can see all the packages installed, which will include the dependencies for the ones you ran pip install on, you need to manually download these dependencies from the python package index https://pypi.org/ just like you did with the regular packages.
Moving Offline
Once you have identified all the packages you will need, and all of their dependencies, you will need to download them, unpack all of them, and move them into one folder, which I will call "OFFLINE_SETUP_FOLDER".
To be clear:
The packages we installed before was only to find out the dependencies we were going to need. You do not have to re-download the packages you have already downloaded before running pip install. You should only need to download the dependencies you have found during the pip freeze command.
Finally you need to copy into the "OFFLINE_SETUP_FOLDER" your python installation file, be it a .msi file for windows, or the .tar file for linux.
Your "OFFLINE_SETUP_FOLDER" should contain the following...
In the following, package can be the name of any package that you downloaded, and the a and b inpackage1a and package1b just represent dependencies for that package. These file names are just examples for packages
python.msi (installation file for python)
/package1 (normal package folder)
/package1a (package dependency folder)
/package1b (package dependency folder)
/package2 (normal package folder)
/package3 (normal package folder)
/package3a (package dependency folder)
Once this is complete, you need to move that folder onto the machine that is completely offline form the network.
Then run the installation for python as you did before and install it on the machine. Do no forget to setup the path variable. Refer back to the Installing Python section if needed.
Open your terminal or CMD and CD into the "OFFLINE_SETUP_FOLDER".
Now you need to CD into each individual package folder, and run this command: python setup.py install and let it run.
If the package install fails, it will be because one of the dependencies has not been installed. If this is the case, CD into the dependency that is says is missing, and run python setup.py install in there first.
Keep repeating these steps until all packages and dependencies have been installed.
This is the end of this python guide for installing python on an offline machine. I hope this helped :)
I installed Python 2.7 on SLES 11 box that previously was running Python 2.6. To do so I used a script described in this post and run it as a root user. Everything went well but when it was done I discovered few issues:
No symbolic links were created and no path updated so I had to manually update the path to link to the new installation bin directory /opt/python2.7/bin
Everything runs good until I switch from root to the normal user at which point Python shell runs but some modules I installed such as PyYAML are missing. Again, these are OK when I run Python as root
As a regular user I'm not able to run pip, easy_install and wheel. For pip I get ImportError: No module named pkg_resources
P.S. Following #user suggestion I tried adding the following path taken from sys.path of the root user to .bashrc which did not fix the problem
export PYTHONPATH=$PYTHONPATH:/opt/python2.7/lib/python27.zip:/opt/python2.7/lib/python2.7:/opt/python2.7/lib/python2.7/plat-linux2:/opt/python2.7/lib/python2.7/lib-tk:/opt/python2.7/lib/python2.7/lib-old:/opt/python2.7/lib/python2.7/lib-dynload:/opt/python2.7/lib/python2.7/site-packages:/opt/python2.7/lib/python2.7/site-packages/PyYAML-3.11-py2.7-linux-x86_64.egg:/opt/python2.7/lib/python2.7/site-packages/pexpect-4.2.0-py2.7.egg:/opt/python2.7/lib/python2.7/site-packages/ptyprocess-0.5.1-py2.7.egg
Credible / official sources: no reply from official forum. Apart from the SO-link you mentioned, there is also https://unix.stackexchange.com/questions/7644/how-to-do-a-binary-install-of-python-2-7-on-suse-linux-enterprise-server-11, which sketches the way to do it described in Installing Python 2.7 on SLES 11 (SO is not official, is it? ;-)
Concerning your problem: both 2. and 3. might be caused by elements lacking in sys.path.
To test this, type
import sys; sys.path
both in user and root python and check for differences. These need to be merged. Try using PYTHONPATH first to test this, but be aware that there are different methods how to adjust sys.path.
If you just need to fix this for normal (non-daemon) users, adjusting the system-wide bash profile would be an easy solution.
(Any questions/feedback is welcome... :-)
I have been learning Django in the development mode for a two months and I am up to speed with most basic aspects of python + django now. However, I was using the built-in runserver till this time
Now, I have got a Webfaction hosting account and wanted to know the following
1) Webfaction sets up the project with a certain Django version (say 1.6.4) and Python version (say 2.7) initially
The project directory (for say project MYAPP) is /<>/webapps/MYAPP
When the site is running in production mode, how does the apache server know which Python version, and which site-packages versions to use with the MYAPP source code to render the site?
I can see that the MYAPP folder has a lib/python2.7 folder, however when I am connected to SSH terminal, and do a "which python", i see :
which python
/usr/local/bin/python
so, do I take that this is the Python executable that is being used for rendering the website instead of the one in webapps/MYSITE/lib/python2.7 folder? How does the information/data flow about which programs to use during rendering the site with apache mod_wsgi work?
2) I was using a virtualenv in the development mode during testing. How do I use this on webfaction in production mode?
3) I am using Pycharm IDE. It worked well for the development mode. I can see that it has a remote interpreter configuration and a Deployment setting/option.
The python path that the remote interpreter settings tool detects automatically is the python executable at /usr/local/bin/python
Is this fine, or should I be pointing it to the more local python2.7 in the webapps/lib folder?
Thanks a lot of the answers and pls let me know if you need any supplemental info
Note to the OP: This should really be three separate questions.
1) For WebFaction, your Django app will use the Python version (and libraries, etc.) defined in:
~/webapps/<appname>/apache2/conf/httpd.conf
Specifically, you'll want to look at what is defined for WSGIPythonPath (which should mostly match up with WSGIDaemonProcess unless you modify the config and are doing something strange).
Note that which python just tells you what the default python is in your shell. This has nothing to do with the config file for the webapp.
2) You may want to expand on this as to exactly what your use case is and why the default Django webapp created by WebFaction doesn't fit your needs. But the short answer is:
Create a virtualenv on your WebFaction account.
Install Django, etc. into the virtualenv.
Edit the httpd.conf file I mentioned above to use your virtualenv instead.
I've done this with both a Django webapp made via the WebFaction control panel and via a custom mod_wsgi webapp. So it does work. Just make sure to use the right Python version when making your virtualenv.
3) I don't use PyCharm so I can't answer this (one reason why this question should be split up).
I've recently started playing around with Mezzanine, a django-based CMS. I recently just managed to configure Fabric to get it uploading to my host, webfaction.com, as its a bit more involved automatically creating the website on the shared hosting, and I wanted to automate that process.
Altogether, that system uses Fabric for template uploads of config files, and pip + virtualenv for handling the python packages.
However, I just recently read about buildout, and how some people swear by it for deployment, and others don't. See here: Django remote deployment with buildout and Fabric and here: http://labs.creativecommons.org/2011/07/29/not-panicking-switching-to-virtualenv-for-deployment/
While I've googled and found a ton of results for buildout vs. pip, there's not much information about buildout + fabric vs. pip + fabric. It seems like some of the features of buildout (uploading config templates, handling supervisor) can be done through fabric. Can someone tell me the advantages and disadvantages of either approach?
Note: As I'm using shared hosting for the foreseeable future, I can't sudo, which it seems buildout may require for a number of existing recipes.
Summary: Pip only installs python packages and there's more you need to do, obviously. You can do most of the extra work in buildout with the advantage that buildout does it for you both locally and on the server. Fabric has to do less, that way. The drawback is buildout's extra complexity, so if a couple of custom fabric commands is enough for you, that might be preferable for you. So: how does the trade-off work for you?
The long version:
Pip is good at installing python packages for your project. Buildout is good at setting up almost everything for a project (including python packages). That's the difference in goals.
Now... you bring fabric into the mix. With pip+fabric, you can call pip from within fabric to grab all the python packages and then you use fabric itself to set up everything else. An apache/nginx config file, creation of a couple of directories ("var/log/"), etc.
With buildout+fabric, you'll have configured buildout already to do a lot of the things like creating directories and generating files from templates and setting up supervisor and setting up a cronjob to fire up supervisor upon #reboot. So the fabfile has to do less.
So... you swap responsibilities. Everything you can do in buildout, you can do in fabric. Everything you can do in buildout, you can do with custom python (or shell) scripts in combination with pip ("read the README for the extra commands you have to do").
Buildout is a good place to do things if it is an integral part of your project. Think about it like this: if you need it both in production on the server and locally on your development machine, you're better off doing it in buildout. Otherwise you have to run fabric on your local machine, too. You can do it, but...
I use fabric in combination with buildout, myself. Buildout is for setting up the project itself, fabric for everything around it. Some examples:
Actually cloning the buildout from git on the production server.
Git pull (and checkout of the proper tag).
Restarting supervisor.
My suggestion: look on pypi for buildout recipes to see if they are handy for you. Do they save you enough work to make it worthwhile to dive into the extra complexity that a full buildout configuration means? If you don't get enough out of buildout, you might be better off with just fabric+pip and a bunch of custom commands in your fabric file.
Take a look at fabtools which adds a lot of nice buildout functionality into your fabfile. I've worked with all sorts, Chef, Puppet (sledgehammer for a walnut) Ansible and Fabric. I find Ansible great for devops teams who are stuck, but don't want to learn a language, but personally, a well organised Fabric project wins hands down.