How to add user made scripts to EC2 ubuntu? - amazon-web-services

I have created an EC2 instance on AWS and I want to transfer some scripts into the site-packages of python3. I can't seem to find a way to do this. Some functions from these files are used by the scripts I am looking to run.
I tried SFTP but it wont allow me to get into the folder just tried to download it.
I am on Ubuntu.
Thanks

Best option to install any service or program in python on a linux server is to create a regular user and then create a virtualenv on a private directory. There, you can freely install any script or package and give the proper rights to specific services/directories of your server.
Install also basic packages in order to run python environment:
sudo apt-get install -y build-essential python-dev python-setuptools zlib1g-dev python-pip

Related

How to Enable PHP 7.0 Zip module on Amazon Linux AMI

Using the command line, I have installed PHP 7.0 on the Amazon Linux AMI.
However, I am unable to install the PHP Zip Module using the command
sudo yum install php7.0-zip
I get the response
No package php7.0.zip available.
I am excited to start using PHP7 on all my PHP websites, but am a little confused why I am unable to install this particular module. Any help would be much appreciated!
Welp,
After trying several variations of the same command...
sudo yum install php7.0-zip
sudo yum install php7-zip
THIS ONE WORKS
sudo yum install php70-zip

How to install Docker from the Source code?

I am trying to install docker from the source code downloaded from github.com/docker/docker
I am unable to install it from the source code .
The Makefile present creates a image , but i want to install it in my system.
Can anyone suggest solution ?
I am using UBUNTU 14.04
Well, idk if this works for your linux distro. (looks like it is ubuntu) but i run kali linux and even if we have different commands to use the process is just as same in every linux distro.
first, before we jump on, we need to update our linux repos.(repositories)
sudo apt update
and,
sudo apt-get update
then,
sudo apt install git
[This installs git]
Now we can start cloning git repos. into our system
go to your desired folder/working directory and type:
sudo git clone "link of the git repo. without the commas"
i would better suggest you to just:
sudo apt install docker.io
[To install docker by apt]
it's better to install it via the docker package and update it to the last version. This is the best way to install docker.

How to get Django 1.7 working on Ubuntu 14.04 with nginx and virtualenv using python 2.7 while having python 3.4 installed?

I am a newbie to Django and Python installation. Intermediate with Ubuntu 14.04.
These are my installations so far in my Ubuntu 14.04.
apt-get install python3-setuptools --force-yes -y ## for python3
easy_install3 pip ## for python3
apt-get install python-setuptools --force-yes -y ## for python2.7 or above
easy_install pip ## for python2.7 or above
apt-get install python-dev --force-yes -y ## because ubuntu 14.04 does not have dev version of python 2
apt-get install python3-dev --force-yes -y ## because ubuntu 14.04 does not have dev version of python 3.4
apt-get install links --force-yes -y ##a command line web browser
apt-get install python-flup --force-yes -y ## connects python to uwsgi)
apt-get install build-essential --force-yes -y ##
pip2 install django uwsgi virtualenv ## use pip to install django and uwsgi and virtualenv for python2
pip3 install django uwsgi ## use pip to install django and uwsgi for python3
For the full list, please look at https://gist.github.com/simkimsia/41c55e0c08eda42e2cb3#file-install-sh-L88
I am confused about the use of virtualenv.
I want to prepare my ubuntu 14.04 server edition for a production level of Django 1.7 as much as possible.
The reason why I installed multiple Python environment because I may have other Python apps running which require 3.4.
My Django files are from bitbucket repository and I have git cloned them into /var/virtual/WebApps/DjangoProject
Inside /var/virtual/WebApps/DjangoProject, I have manage.py and other files and folders.
Please advise on how do I get the Django project running for this situation.
I am currently testing this setup on my virtualbox and vagrant.
EDIT
There will be at least 2 Django applications. 1 requires 2.7 python. The other requires 3.4 python.
Let me add that this is a single server that will host the application, frontend, and database.
EDIT 1
I have restarted with a fresh install of Ubuntu 14.04 and I started with Python 2.7.6 and Python 3.4.0.
I then did sudo apt-get install python-virtualenv which I checked its version: 1.11.4.
I have created ~/virtualenvs/py2.7 and ~/virtualenvs/py3.4.
Inside ~/virtualenvs, I did virtualenv -p /usr/bin/python2 py2.7
and ~/virtualenvs, I did virtualenv -p /usr/bin/python3 py3.4
So how do I install python2 only libraries for the python 2 app?
E.g. are django-adminfiles, sorl-thumbnail, psycopg2
EDIT 5
Use virtualenv --system-site-packages -p /usr/bin/python2 py2.7 instead
I have restarted with a fresh install of Ubuntu 14.04 and I started
with Python 2.7.6 and Python 3.4.0.
Okay, so now in your system you have two base versions of Python. Base version just means, the versions that are supported by your operating system; which you have installed globally.
In other words you have installed them using the operating system's package installers and did not compile them separately.
In practice, this above only matters in Linux, because in Windows you cannot install "locally" without going through a few hoops; all Python installers will register themselves in the registry thus making them global, base Python versions.
I then did sudo apt-get install python-virtualenv which I checked its
version: 1.11.4.
This package is outdated (current version is 12.0.7).
Now you have virtual environment installed against the base Python 2 version because the package requires Python 2.
In practical terms it means if you need to upgrade Python 2, you'll have to make sure python-virtualenv is also updated for the base versions of both Pythons that are supported by your operating system. This means, when you apt-get update and apt-get upgrade, virtualenv will be upgraded.
Usually this doesn't matter as its a rare case if python2 is upgraded and then python-virtualenv is not upgraded to match its dependency.
However this is not recommended because you want to control the versions of critical software "manually" to avoid any surprises. There are ways to control this on Ubuntu and other debian-like distributions by pinning versions; but even if you do so, you may not be getting the latest version of the library which may force you later on to uninstall the version that came with your operating system, and reinstall it from source.
I have created ~/virtualenvs/py2.7 and ~/virtualenvs/py3.4.
Inside ~/virtualenvs, I did virtualenv -p /usr/bin/python2 py2.7 and
~/virtualenvs, I did virtualenv -p /usr/bin/python3 py3.4
So how do I install python2 only libraries for the python 2 app?
E.g. are django-adminfiles, sorl-thumbnail, psycopg2
In order for solr-thumbnail and psycopg2 to be installed correctly, you need to build their dependencies; so
sudo apt-get install libjpeg62 libjpeg62-dev zlib1g-dev
sudo apt-get install libgraphicsmagick++-dev libboost-python1.55-dev
sudo apt-get install libexpat1-dev libpython-dev libpython3-dev libssl-dev libpq-dev
To install libraries for the Python 2 app:
Activate the virtual environment; by typing source ~/virtualenvs/py2.7/bin/activate
Type pip install _____ (name of the library)
To support multiple Python applications with different major versions; your system should have both Python major versions installed (you already have done this).
You then install virtualenv for each Python major version. You can skip this step if your applications are fully contained (that is, they include the Python runtime required - but this is a rare case) or if you have a single purpose server.
You should avoid installing anything but the base Python libraries in your system's global python. That is avoid (as root, or using sudo) to pip install things; because these will be installed for all users of Python and may cause problems (on some systems; like Fedora/RedHat - critical system packages like yum rely on the base system Python).
Next step is to make sure you have a suitable build environment available. This means for Debian-sourced systems to install build-essential and further the support libraries for common Python drivers and modules. The exact libraries you need to install will depend on the applications you are planning to host, but at a minimum you should make sure PIL (or Pillow) can be installed and database drivers' support libraries are available. To do so, you can apt-get build-deb python-imaging psycopg2 python-mysqldb (for PostgreSQL, MySQL and PIL).
Now you have a system ready to support most Python applications. You can then optionally add other utilities, but I would try to avoid assuming too much about what applications will require.
To host an app:
Create a virtual environment with the base version of Python required. So virtualenv-2.7 or virtualenv-3 as the normal non-root user account.
Install required packages into the virtual environment.
Adjust the bootstrap script for your application to use the correct Python binary. This is usually done from whatever process you are using to manage your application server. For example, on my server I use supervisord.
That's all you have to do. Everything else will depend on the individual application's requirements and setup (so if you need to serve static files, you'll have to configure that mapping, etc.)
After reading your shell script, it seems you are trying to build a server that will support both the application, the front end and the database.
In order to support such a system; you will need to install the following:
Database server(s) that you would like to support. As this is a single purpose server, you will also need to install database command line clients.
Source code tools (git, etc.)
A global process manager (like supervisor or circus).
Base Python versions you intend to support; and their development headers (sudo apt-get install python-dev)
setuptools, pip, and then virtualenv. These tools should be installed from source rather than your package manager; to ensure the latest versions are installed. You should install these globally (ie, as root) so they are available for all users.
A build tool chain (ie, "Development Tools" or build-essential)
Support libraries for any extensions (but not the extensions themselves). The easiest way to do this is to use the package manager to build the dependencies and not the packages sudo apt-get build-dep python-imaging psycopg2 python-mysqldb.
The next thing you need to do is decide how you will run your application servers (the "django code"). You can use uwsgi, gunicorn, etc. as these are the ones most tested with django.
You need to be able to support multiple versions of these runtimes, so instead of installing them globally across the system; just build their dependencies and install the specific version required for each application in its own isolated environment.
The next thing you need to install is a front end proxy for your applications. You can install whatever proxy suits your needs (nginx is the most popular); but please install from source rather than the packages as those are almost always out of date.
Once all this is setup, the process of hosting a django application is the following:
Create a separate user account with a non-login shell.
Create a virtual environment in this user's home directory. I recommend keeping some standard here, like env for the virtual environments.
Download/copy the source code of your application.
Create a standard directory where you will store the static files. For example I use $HOME/www/static.
Create an entry in your process manager.
Create an entry in your proxy for front end routing.
Reload your proxy server.
Reload your process manager.
You can automate/script a lot of the above. For example, you can create a custom skeleton directory to create the base directories for you when adding new users; and you can create custom templates for other areas using tools like cookiecutter.
Understanding execution path
The first principle to understand is that when you install packages on your operating system (virtual machine, whatever) with sudo is that you are installing all those packages in a global location, that is, a file directory which your system knows about via the $PATH (PATH) environment variable.
This $PATH variable is often set-up by default in your .bashrc or your .bash_profile or your .profile whenever a new linux/unix user is created. Whether you have any of these files depends on what is inside the /etc/skel on your system. /etc/skel holds a "template" of these files that gets duplicated whenever a new user with a home directory is created.
When you type echo $PATH in command line, you would see a list of execution search path delimited by :, for example:
/usr/local/sbin:/usr/local/bin:/usr/bin
This is an example I will use for the purpose of this discussion. Different OS will give you slightly different default $PATH but the simple idea here is that the binaries of your globally installed packages gets thrown into one of these directories and the reason you are able to run these binaries is because these binaries (programs) are now available in the execution search path (simply called $PATH).
Ok great, but what does this mean for the python 2 and python 3 interpreters I installed?
So in relation to your question, this means that when you sudo apt-get or sudo aptitude install python2 or python3 binaries (python interpreters), they are both available and to differentiate which interpeter you are using at any moment, you would run python2.7 or python3.4 that correspondingly calls /usr/bin/python2.7 or /usr/bin/python3.4
You can always check this easily by using the which linux/unix command. which python3.4 will return you the exact path where your python3.4 binary is installed.
Similarly, when type pip2.7 in your command line, you are asking your system to execute the pip2.7 program that came installed with your python2.7 package. And naturally which pip2.7 will reveal to you where this pip2.7 binary has been installed.
All this is possible simply because these binaries are have been installed by you using sudo apt-get and are placed in the directories listed in $PATH. If you move one of these binaries forcibly to another directory not listed in $PATH, you will realize that you can no longer run the binary in your command line without typing out the specific path to the binary.
Additional python specific information
The Python Interpreter has another attribute call $PYTHONPATH. This - as you will rightly deduce - is a variable that holds a list of directories ("search path") where the python interpreter will search for python modules to load. If you want to know where your python interpreter is currently looking for modules (your own python modules or 3rd party python modules), run
python -c "import sys; print(sys.path)"
where python is your specific python interpreter. If /usr/bin/python is symlinked to /usr/bin/python2.7, then you are in fact calling python2.7.
When would python path matter? It matters inside your own .py source code when you ask to import other modules. The import line of code in your .py source code is where you are asking the python interpreter to go forth and search for the module that you want to import. As you can imagine, if your sys.path (in python) is empty, you will not be able to import any 3rd party modules.
Beyond the gory details
Now that we have a clear understanding of the underlying principles behind $PATH and $PYTHONPATH, we can now understand what the virtualenv (and additionally, virtualenvwrapper is useful) does for us.
When we create a new virtualenv giving it a directory, what we are saying is that we want to symlink a specific python interpreter (via -p python2.7 flag for the virtualenv command) for example to that virtualenv.
When we source activate that virtualenv we created, we are in fact invoking shell scripts that come with virtualenv to dynamically modify the $PATH. Try running echo $PATH after you have source activated a virtualenv you created. What do you see?
That's right, you will be seeing something like this:-
/Users/calvin/.virtualenvs/myproject/bin:/usr/local/sbin:/usr/local/bin:/usr/bin
And if you type which python, what do you get?
/Users/calvin/.virtualenvs/myproject/bin/python
That's right, the python interpreter we are using is the one that is inside the virtualenv directory and no longer from the /usr/bin directory.
If you run
ls -la /Users/calvin/.virtualenvs/myproject/bin
what do you see?
Ah! A symlink to the specific global python interpreter you specified when you created this virtualenv. If you don't see a symlink, this means that your entire python interpreter was copied over from the globally installed one.
So this is what our virtualenv tool does. It lets us isolate specific projects and choose which python interpreter to use for a specific project.
Once you have source activated a virtualenv, your pip is also the pip that is located in your virtualenv directory. This pip is python path-aware. When you issue pip install commands, python packages now gets installed into your
/Users/calvin/.virtualenvs/myproject/lib/site-packages
python packages directory; and are available ONLY to your project when you source activate that particular project.
What if you install packages with sudo pip?
sudo pip calls the pip tool that comes with your globally installed python interpreter. When you use sudo pip, you are installing your python packages into a global location (which is not jailed within a virtualenv).
I did not give you specific answers to your how-can-I-make-two-projects-use-two-different-python-interpreters question. I explained the principles and now that you know the principles, it is clear what you need to do. Cheers!
:D
you should have to try following steps to make your django work with python-2.7.
As Ubuntu-14 already come with Python-2.7 installed. so no need to install python-2.7.
First of all install python setuptools for python-2.7:
https://pypi.python.org/packages/source/s/setuptools/setuptools-12.0.5.tar.gz
tar -zxvf setuptools-12.0.5.tar.gz
cd setuptools-12.0.5/
sudo python2.7 setup.py install
after install setuptools install pip for python-2.7
wget https://pypi.python.org/packages/source/p/pip/pip-6.0.8.tar.gz
tar -zxvf pip-6.0.8.tar.gz
cd pip-6.0.8/
sudu python2.7 setup.py install
after installing pip now we have to go for install python virtual environment:
sudo pip2.7 install virtualenv
then to folder where you want to create new virtualenv using python 2.7
virtualenv-2.7 "name for new virtualenv for example '/var/virtual/DjangoProject'"
cd /var/virtual/
virtualenv-2.7 DjangoProject
Note just created virtualenv name is "DjangoProject"
Now time to activate new virtualenv
cd DjangoProject
source bib/activate
Now time to clone your project here.
git clone "your git url"
install rest of apps and django here. if you have created simply install by requirement.txt
cd "projectFolder"
pip install -r requirement.txt
after doing all these your new python2.7 django with virtualenv are ready to use. just run django wsgi server for test is every thing working:
python manage.py runserver.
Now you can use this virtualenv and project to setup for web server.
Hope this will help-full to you.
You should use pyenv (and pyenv virtualenvwrapper). With these tools you can have multiple Python versions and switch between them.
Here is how you setup one environment with Python 3.4.2 and one with Python 2.7.8 (and each environment can have its own Python packages)
# install pyenv
git clone https://github.com/yyuu/pyenv.git ~/.pyenv
# setup pyenv and load when shell is loaded
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init -)"' >> ~/.bashrc
# install pyenv virtualenvwrapper
git clone https://github.com/yyuu/pyenv-virtualenvwrapper.git ~/.pyenv/plugins/pyenv-virtualenvwrapper
# setup pyenv virtualenvwrapper and load when shell is loaded
echo 'export PYENV_VIRTUALENVWRAPPER_PREFER_PYVENV="true"' >> ~/.bashrc
echo "pyenv virtualenvwrapper" >> ~/.bashrc
Now everything is setup. Restart your shell.
# install python 2.7.8
pyenv install 2.7.8
pyenv rehash
# install python 3.4.2
pyenv install 3.4.2
pyenv rehash
# switch the global python version to 2.7.8
pyenv global 2.7.8
# create a virtual environment using python 2.7.8
mkvirtualenv project278
# -- now you can install you python 2 modules
# leave the virtual environment
deactivate
# switch the global python to 3.4.2
pyenv global 3.4.2
# create a virtual environment using python 3.4.2
mkvirtualenv project342
# -- now you can install you python 2 modules
# leave the virtual environment
deactivate
# switch the global python to the system python.
pyenv global system
# swith to environment project278 and check the python version
workon project278
python --version
# swith to environment project342 and check the python version
workon project342
python --version
VoilĂ ! You now have two environments with different Python versions. Just use the workon (and the deactivate) commands to switch between them.
Make sure you read the documentation of the used packages, so you understand what is going on!

Installing numpy on Amazon EC2 virtualenv

I'm trying to build a virtualenv that uses python27 in a virtualenv on an Amazon EC2. Research on the web indicated messing with the preinstalled Python26 is a bad idea so I wanted to work in a safe virtual env. Here are all of my commands starting from a fresh instance on EC2
sudo easy_install python27
sudo easy_install virtualenv
yum groupinstall "Development Tools"
mkdir virt_env
virtualenv -p python27 virt_env/py27
And this all seems to work great. When I activate my virtualenv Python27 is the default, and outside of it Python26 is the default. So far so good. Next I tried iPython in the virtualenv
source virt_env/py27/bin/activate
pip install ipython
This works great, again ipython is available in the virtualenv and not available outside of it. However when I go to install numpy in the virtualenv I get the follwing:
pip install numpy
// Lots of output that I won't paste all of it, main error below
SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.
Not sure what I'm doing wrong because I've always installed numpy on virtualenv this way. Any help would be greatly appreciated thanks
EDIT: I've also tried using:
sudo yum install python-devel
in the virtualenv. Still no help
Was able to answer my own question. To install the python dev correctly I needed:
sudo yum install python27-devel

How do you install Docutils from the Terminal so that Django admindocs will work?

Docutils is a great package. If you are using Django the admindocs package needs docutils. Instructions are available for installing with a web browser, but what if you are remote and logging in with a terminal over SSH? How to install in that case? What if you just want a quick recipe to do the job with the terminal?
I know I'm rather late to this question, but the accepted answer doesn't really reflect the common best practices from Python community members (and even less so from the Django community members.) While the outlined manual installation process does work, is is far more pains taking and error prone than the following:
You really should be using Pip. With Pip installing docutils system wide is as simple as:
$ sudo pip install docutils
This not only works for docutils but nearly any package on the 'Cheese Shop' as well as many other code repositories (Github, Bitbucket, etc.)
You may also want to look into other common Python best practice tools like virtualenv and virtualenvwrapper so that you can avoid global package installation.
To install Pip on Ubuntu/Debain I generally do the following:
$ sudo apt-get install python-pip
BTW: for virtualenv 'sudo apt-get install python-virtualenv' and for virtualenvwrapper 'sudo apt-get install virtualenvwrapper'.
The key to the install is to use the curl utility. The following will install docutils:
mkdir docutilsetup
cd docutilsetup
curl -o docutils-docutils.tar.gz http://docutils.svn.sourceforge.net/viewvc/docutils/trunk/docutils/?view=tar
gunzip docutils-docutils.tar.gz
tar -xf docutils-docutils.tar
cd docutils
sudo python setup.py install
This performs the following steps: Create a directory to download docutils into. cd into the directory just made, and use curl to download the zipped version of docutils. Unzip the file which creates a subdirectory docutils. cd into that directory and install with root permissions.
If you are using Django you will have to restart Django for admindocs to start working.
Although it is an old thread, I want to share the answer I found. To install type command
sudo apt install python-docutils
or
sudo apt install python3-docutils
This will install the dependencies too. Yesterday, I installed docutils using this command for Geany editor and it is working fine.