I want to install one of the jupyter containers with python 2.X using docker-stacks.
The documentation at GitHub for jupyter/docker-stacks say:
Python 2.x was removed from all images on August 10th, 2017, starting in tag cc9feab481f7. If you wish to continue using Python 2.x, pin to tag 82b978b3ceeb.
See:
https://github.com/jupyter/docker-stacks
I assume you run this as follows to, say for example, install minimal-notebook:
docker run -it --rm -p 8888:8888 jupyter/minimal-notebook:82b978b3ceeb
But after running I find that python 3.x was installed:
sys.version_info(major=3, minor=6, micro=2, releaselevel='final', serial=0)
Here's the output of the 'docker run' command:
$ docker run -it --rm -p 8888:8888 jupyter/minimal-notebook:82b978b3ceeb
/usr/local/bin/start.sh: line 48: [: missing `]'
/usr/local/bin/start.sh: line 48: : command not found
Execute the command
[I 20:55:49.950 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
[W 20:55:49.979 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.
[I 20:55:50.010 NotebookApp] JupyterLab alpha preview extension loaded from /opt/conda/lib/python3.6/site-packages/jupyterlab
JupyterLab v0.24.1
Known labextensions:
[I 20:55:50.013 NotebookApp] Running the core application with no additional extensions or settings
[I 20:55:50.016 NotebookApp] Serving notebooks from local directory: /home/jovyan
[I 20:55:50.016 NotebookApp] 0 active kernels
[I 20:55:50.017 NotebookApp] The Jupyter Notebook is running at: http://[all ip addresses on your system]:8888/?token=f09a12bf53902cb20aca2f1924011e1e80d51243cc10a390
[I 20:55:50.017 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 20:55:50.017 NotebookApp]
Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://localhost:8888/?token=f09a12bf53902cb20aca2f1924011e1e80d51243cc10a390
There's a recipe for creating a Python 2 conda environment in FROM one of the core images on the GitHub wiki: https://github.com/jupyter/docker-stacks/wiki/Docker-recipes#add-a-python-2x-environment
I notice that the link suggested in previous post is not there anymore, so I put here the some with some modifications.
You could create the following Dockerfile:
# From https://github.com/jupyter/docker-stacks/wiki/Docker-recipes#add-a-python-2x-environment
# Choose your desired base image: you could use another from https://github.com/busbud/jupyter-docker-stacks
FROM jupyter/all-spark-notebook:latest
# Create a Python 2.x environment using conda including at least the ipython kernel
# and the kernda utility. Add any additional packages you want available for use
# in a Python 2 notebook to the first line here (e.g., pandas, matplotlib, etc.)
RUN conda create --quiet --yes -p $CONDA_DIR/envs/python2 python=2.7 ipython ipykernel kernda && \
conda clean -tipsy
USER root
# Bundle requirements
# You can change the libraries in the file
# requirements.txt
ADD requirements.txt /requirements.txt
# Create a global kernelspec in the image and modify it so that it properly activates
# the python2 conda environment.
RUN $CONDA_DIR/envs/python2/bin/python -m ipykernel install && \
$CONDA_DIR/envs/python2/bin/kernda -o -y /usr/local/share/jupyter/kernels/python2/kernel.json && \
pip install -r /requirements.txt && \
rm /requirements.txt && \
USER $NB_USER
or if you do not have requirements
# From https://github.com/jupyter/docker-stacks/wiki/Docker-recipes#add-a-python-2x-environment
# Choose your desired base image: you could use another from https://github.com/busbud/jupyter-docker-stacks
FROM jupyter/all-spark-notebook:latest
# Create a Python 2.x environment using conda including at least the ipython kernel
# and the kernda utility. Add any additional packages you want available for use
# in a Python 2 notebook to the first line here (e.g., pandas, matplotlib, etc.)
RUN conda create --quiet --yes -p $CONDA_DIR/envs/python2 python=2.7 ipython ipykernel kernda && \
conda clean -tipsy
USER root
# Create a global kernelspec in the image and modify it so that it properly activates
# the python2 conda environment.
RUN $CONDA_DIR/envs/python2/bin/python -m ipykernel install && \
$CONDA_DIR/envs/python2/bin/kernda -o -y /usr/local/share/jupyter/kernels/python2/kernel.json &
USER $NB_USER
Then go to the folder:
To create image:
docker build -t wm/ubuntupython2jupyterpyspark:v1.0 .
wm/ubuntupython2jupyterpyspark:v1.0 is just an example you could put another
To run container
docker run -p 8888:8888 wm/ubuntupython2jupyterpyspark:v1.0
Related
We have followed this doc to spin up notebook running with interactive sessions. We want to add a few python packages to the environment to assist with development (i.e. pyright). I have added the pip install at the bottom, stopped the instance, restart instance, run "import pyright", but I get "ModuleNotFoundError: No module named 'pyright'"
#!/bin/bash
set -ex
sudo -u ec2-user -i <<'EOF'
ANACONDA_DIR=/home/ec2-user/anaconda3
# Create and Activate Conda Env
echo "Creating glue_pyspark conda enviornment"
conda create --name glue_pyspark python=3.7 ipykernel jupyter nb_conda -y
echo "Activating glue_pyspark"
source activate glue_pyspark
# Install Glue Sessions to Env
echo "Installing AWS Glue Sessions with pip"
pip install aws-glue-sessions
# Clone glue_pyspark to glue_scala. This is required because I had to match kernel naming conventions to their environments and couldn't have two kernels in one conda env.
echo "Cloning glue_pyspark to glue_scala"
conda create --name glue_scala --clone glue_pyspark
# Remove python3 kernel from glue_pyspark
rm -r ${ANACONDA_DIR}/envs/glue_pyspark/share/jupyter/kernels/python3
rm -r ${ANACONDA_DIR}/envs/glue_scala/share/jupyter/kernels/python3
# Copy kernels to Jupyter kernel env (Discoverable by conda_nb_kernel)
echo "Copying Glue PySpark Kernel"
cp -r ${ANACONDA_DIR}/envs/glue_pyspark/lib/python3.7/site-packages/aws_glue_interactive_sessions_kernel/glue_pyspark/ ${ANACONDA_DIR}/envs/glue_pyspark/share/jupyter/kernels/glue_pyspark/
echo "Copying Glue Spark Kernel"
mkdir ${ANACONDA_DIR}/envs/glue_scala/share/jupyter/kernels
cp -r ${ANACONDA_DIR}/envs/glue_scala/lib/python3.7/site-packages/aws_glue_interactive_sessions_kernel/glue_spark/ ${ANACONDA_DIR}/envs/glue_scala/share/jupyter/kernels/glue_spark/
echo "Changing Jupyter kernel manager from EnvironmentKernelSpecManager to CondaKernelSpecManager"
JUPYTER_CONFIG=/home/ec2-user/.jupyter/jupyter_notebook_config.py
sed -i '/EnvironmentKernelSpecManager/ s/^/#/' ${JUPYTER_CONFIG}
echo "c.CondaKernelSpecManager.name_format='conda_{environment}'" >> ${JUPYTER_CONFIG}
echo "c.CondaKernelSpecManager.env_filter='anaconda3$|JupyterSystemEnv$|/R$'" >> ${JUPYTER_CONFIG}
# Install python modules to env
pip install "pyright"
EOF
systemctl restart jupyter-server
Am I missing something in the script? I assumed just "pip install "pyright"" would've worked.
Update:
I have included the following under the pip install aws-glue-sessions:
pip install "pyright"
and
pip install pyright
When I check the CloudWatch logs, I see that the package is being downloaded... I would assume it means it's installed.
[1]: https://i.stack.imgur.com/JeKce.png
I am trying to use a post-startup script to create a Vertex AI User Managed Notebook whose Jupyter Lab has a dedicated virtual environment and corresponding computing kernel when first launched. I have had success creating the instance and then, as a second manual step from within the Jupyter Lab > Terminal, running a bash script like so:
#!/bin/bash
cd /home/jupyter
mkdir -p env
cd env
python3 -m venv envName --system-site-packages
source envName/bin/activate
envName/bin/python3 -m pip install --upgrade pip
python -m ipykernel install --user --name=envName
pip3 install geemap --user
pip3 install earthengine-api --user
pip3 install ipyleaflet --user
pip3 install folium --user
pip3 install voila --user
pip3 install jupyterlab_widgets
deactivate
jupyter labextension install --no-build #jupyter-widgets/jupyterlab-manager jupyter-leaflet
jupyter lab build --dev-build=False --minimize=False
jupyter labextension enable #jupyter-widgets/jupyterlab-manager
However, I have not had luck using this code as a post-startup script (being supplied through the console creation tools, as opposed to command line, thus far). When I open Jupyter Lab and look at the relevant structures, I find that there is no environment or kernel. Could someone please provide a working example that accomplishes my aim, or otherwise describe the order of build steps that one would follow?
Post startup scripts run as root.
When you run:
python -m ipykernel install --user --name=envName
Notebook is using current user which is root vs when you use Terminal, which is running as jupyter user.
Option 1) Have 2 scripts:
Script A. Contents specified in original post. Example: gs://newsml-us-central1/so73649262.sh
Script B. Downloads script and execute it as jupyter. Example: gs://newsml-us-central1/so1.sh and use it as post-startup script.
#!/bin/bash
set -x
gsutil cp gs://newsml-us-central1/so73649262.sh /home/jupyter
chown jupyter /home/jupyter/so73649262.sh
chmod a+x /home/jupyter/so73649262.sh
su -c '/home/jupyter/so73649262.sh' jupyter
Option 2) Create a file in bash using EOF. Write the contents into a single file and execute it as mentioned above.
This is being posted as support context for the accepted solution from #gogasca.
#gogasca's suggestion (I'm using Option 1) works great, if you are patient. Through many attempts, I discovered that inconsistent behavior was based on timing of access. Using Option 1, the User Managed Notebook appears available for use in Vertex AI Workbench (green check and clickable "OPEN JUPYTERLAB" link) before the installation script(s) have finished.
If you open the Notebook too soon, you will find two things: (1) you will be prompted for a recommended Jupyter Lab build, for instance:
Build Recommended
JupyterLab build is suggested:
#jupyter-widgets/jupyterlab-manager changed from file:../extensions/jupyter-widgets-jupyterlab-manager-3.1.1.tgz to file:../extensions/jupyter-widgets-jupyterlab-manager-5.0.3.tgz
and (2) while the custom environment/kernel is present and accessible, if you try to use ipyleaflet or ipywidget tools, you will see one of several JavaScript errors, depending on how quickly you try to use the kernel, relative to the build that is (apparently) continuing to take place in the background: Error displaying widget: model not found, and/or a broken page icon with a JavaScript error, that, if clicked, will show you something like:
[Open Browser Console for more detailed log - Double click to close this message]
Failed to load model class 'LeafletMapModel' from module 'jupyter-leaflet'
Error: No version of module jupyter-leaflet is registered
at f.loadClass (https://someURL.notebooks.googleusercontent.com/lab/extensions/#jupyter-widgets/jupyterlab-manager/static/134.bcbea9feb6e7c4da7530.js?v=bcbea9feb6e7c4da7530:1:74856)
at f.loadModelClass (https://someURL.notebooks.googleusercontent.com/lab/extensions/#jupyter-widgets/jupyterlab-manager/static/150.3e1e5adfd821b9b96340.js?v=3e1e5adfd821b9b96340:1:10729)
at f._make_model (https://someURL.notebooks.googleusercontent.com/lab/extensions/#jupyter-widgets/jupyterlab-manager/static/150.3e1e5adfd821b9b96340.js?v=3e1e5adfd821b9b96340:1:7517)
at f.new_model (https://someURL.notebooks.googleusercontent.com/lab/extensions/#jupyter-widgets/jupyterlab-manager/static/150.3e1e5adfd821b9b96340.js?v=3e1e5adfd821b9b96340:1:5137)
at https://someURL.notebooks.googleusercontent.com/lab/extensions/#jupyter-widgets/jupyterlab-manager/static/150.3e1e5adfd821b9b96340.js?v=3e1e5adfd821b9b96340:1:6385
at Array.map ()
at f._loadFromKernel (https://someURL.notebooks.googleusercontent.com/lab/extensions/#jupyter-widgets/jupyterlab-manager/static/150.3e1e5adfd821b9b96340.js?v=3e1e5adfd821b9b96340:1:6278)
at async f.restoreWidgets (https://someURL.notebooks.googleusercontent.com/lab/extensions/#jupyter-widgets/jupyterlab-manager/static/134.bcbea9feb6e7c4da7530.js?v=bcbea9feb6e7c4da7530:1:77764)
The solution here is to keep waiting. In my demo script, I transfer a file at the end of the build process. If I wait long enough for this file to actually appear in the Instance directories, the recommendation for a rebuild is absent and the extensions work properly.
I didn't know what is the benefit of Docker over virtual environment. Is it necessary to use virtual environment and docker at the same time.
Before today, Basically I used virtual environment to crate Django project. But today My friend recommended me to use docker. I'm confused what should i use ?
I used this command to create virtual environment
python3 -m venv virtual_environment_name
Is this is best way to create virtual environment or should i use another way to create virtual environment
I would rather use pipenv to replace virtualenv at local development environment, and docker without virtual environment at production. Here is my Dockerfile(run django with gunicorn):
FROM python:3.6
ENV PYTHONUNBUFFERED 1
# switch system download source
RUN python -c "s='mirrors.163.com';import re;from pathlib import Path;p=Path('/etc/apt/sources.list');p.write_text(re.sub(r'(deb|security)\.debian\.org', s, p.read_text()))"
RUN apt-get update
# aliyun source for pip
RUN python -c "s='mirrors.aliyun.com';from pathlib import Path;p=Path.home()/'.pip';p.mkdir();(p/'pip.conf').write_text(f'[global]\nindex-url=https://{s}/pypi/simple\n[install]\ntrusted-host={s}\n')"
# Optional: install and conf vim, install ipython
RUN apt-get install -y vim
RUN wget https://raw.githubusercontent.com/waketzheng/carstino/master/.vimrc
RUN pip install ipython
# copy source code to docker image
WORKDIR /carrot
ADD . .
# required packages for carrot
RUN apt-get install -y ruby-sass
# install gunicorn and Pipfile
RUN pip install pipenv gunicorn
RUN pipenv install --system
RUN python manage.py collectstatic --noinput
# database name and rpc server ip
ENV POSTGRES_HOST=db
ENV RPC_SERVER_IP=172.21.0.2
EXPOSE 9000
# the PASSWORD env should be replace to a real one
CMD ["gunicorn", "--bind", ":9000", "--env", "PASSWORD=123456", "--error-logfile", "gunicorn.error", "--log-file", "gunicorn.log", "carrot.wsgi:application"]
I am following the below instructions to install superset. In this step "superset runserver -d" getting an error below. How do I fix this issue. Thanks
[DEPRECATED] As of Flask >=1.0.0, this command is no longer supported, please use flask run instead, as documented in our CONTRIBUTING.md
[example]
flask run -p 8080 --with-threads --reload --debugger
# Install superset
pip install superset
# Create an admin user (you will be prompted to set a username, first and last name before setting a password)
fabmanager create-admin --app superset
# Initialize the database
superset db upgrade
# Load some data to play with
superset load_examples
# Create default roles and permissions
superset init
# To start a development web server on port 8088, use -p to bind to another port
superset runserver -d
Yesterday commit https://github.com/apache/incubator-superset/pull/5966 fixed similiar issue with docker install. Try to clone repo from scratch and go trough the following steps:
git clone https://github.com/apache/incubator-superset/
cd incubator-superset/contrib/docker
# prefix with SUPERSET_LOAD_EXAMPLES=yes to load examples:
docker-compose run --rm superset ./docker-init.sh
# you can run this command everytime you need to start superset now:
docker-compose up
New installation doc: https://github.com/apache/incubator-superset/blob/master/docs/installation.rst
Try the below steps:
$ pip install flask==1.0.0
$ pip install sqlalchemy==1.2.18
$ pip uninstall pandas
$ pip install pandas==0.23.4
I have an aws code pipeline which currently successfully deploys code to my EC2 instances.
I have a Docker image that has the necessary setup to run my code, Dockerfile provided below. When I run docker run -t it just loads up an interactive shell on my docker but then hangs on any command (eg: ls)
Any advice?
FROM continuumio/anaconda2
RUN apt-get install git
ENV PYTHONPATH /app/phdcode/panaxeaA1
# setting up venv
RUN conda create --name panaxea -y
RUN /bin/bash -c "source activate panaxea"
# Installing necessary packages
RUN conda install -c guyer pysparse
RUN conda install -c conda-forge pympler
RUN pip install pysparse
RUN git clone https://github.com/usnistgov/fipy.git
RUN cd fipy && python setup.py install
RUN cd ~
WORKDIR /app
COPY . /app
RUN cd panaxeaA1/models/alpha04c/launchers
RUN echo "launching..."
CMD python launcher_260818_aws.py
docker run -t simply starts a docker container with a pseuodo-tty connection to the container's stdin. However, just running this command does not establish an interactive shell to the container. You will need this to be able to have run commands within your container.
You need to also append the -i command line flag along with the shell you wish to use. For example, docker run -it IMAGE_NAME bash will launch a container from the image you provide using bash as your interactive shell. You can then run Bash commands as you normally would.
If you are looking for a simple way to run containers on EC2 instances in AWS, I highly recommend AWS EC2 Container Service (ECS) as an option. It is a very simple service for running containers that abstracts and manages much of the server level work involved in running containers.