im using EMR and wanted to use jupyter(ipython) so i added to the cluster the bootstrap action:
s3://elasticmapreduce.bootstrapactions/ipython-notebook/install-ipython-notebook
I performed the port tunelling to access jupyter from my local host and works fine, but it is asking for a login password, tried empty, tried hadoop, but no luck, does any body knows what is the jypyter password?
I ran into this problem as well when I used the same bootstrap action. I tried adding in Args=[--password, jupyter] which I also could not get working. That was from this aws forum:
Name='Install Jupyter notebook',Path="s3://aws-bigdata-blog/artifacts/aws-blog-emr-jupyter/install-jupyter-emr5.sh",Args=[--r,--julia,--toree,--torch,--ruby,--ds-packages,--ml-packages,--python-packages,'ggplot nilearn',--port,8880,--password,jupyter,--jupyterhub,--jupyterhub-port,8001,--cached-install,--notebook-dir,s3://<your-s3-bucket>/notebooks/,--copy-samples]
What I did instead was to follow these instructions for installing anaconda directly in the EMR instance using the CLI. If you follow the first part you should be able to get it up and running. To summarize here:
ssh into your master emr instance using the .pem file you saved
once there's you'll want to install anaconda using super user priveledges: sudo wget http://repo.continuum.io/archive/Anaconda3-4.1.1-Linux-x86_64.sh. Then bash Anaconda3–4.1.1-Linux-x86_64.sh
Make sure you're using the anaconda version of python: which python
If you're not, specify your source: source .bashrc
Now make a jupyter config file: jupyter notebook --generate-config
cd into the jupyter folder: cd ~/.jupyter/
update the config file: vi jupyter_notebook_config.py
In the config file add the following lines:
c = get_config()
c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False
c.NotebookApp.port = 6789 <---pick whichever port you want
exit out of the config editor and run jupyter via: jupyter notebook
this should run a notebook with no active kernels (for now). But it will give you the token you're looking for: http://localhost:6789/?token=xxxxxx
Leave this running, and open a new terminal window. Now you'll want to tunnel to the EMR instance per this aws blog post (make the port the same as the one you specified in the config file). ssh -o ServerAliveInterval=10 -i <<credentials.pem>> -N -L 8192:<<master-public-dns-name>>:8192 hadoop#<<master-public-dns-name>>
Opening localhost:6789 in the browser should prompt you with the jupyter page to enter your password or token. Enter the token that was generated in the above step and you should be good to go.
Hope this helps! There might be a less convoluted way, but this is what ended up working for me.
Related
Hi I have installed Spark and Python and Jupyter Notebook in Amazon AWS EC2 but when I run "jupyter notebook" in the command prompt it just provides an address for "jupyter notebook" when I open the jupyter notebook, I can't run pyspark commands. I just can run python commands.
I googled it and found these commands:
export SPARK_HOME=/home/ubuntu/spark-3.0.1-bin-hadoop3.2
export PATH=$SPARK_HOME/bin:$PATH
export PYTHONPATH=$SPARK_HOME/python:$PYTHONPATH
After executing them, when I typed "jupyter notebook" I access a "jupyter notebook" that can support pyspark.
However, when I close my commands prompt and logged in later, I have to type the above commands again to be able to have "pyspark" in the "jupyter notebook".
My question: how permanently save those variables in "PATH" of environment variable. And how can see all of the environment variables including the ones that I entered through above commands.
It's can be a similar case for environment variables and your local terminal. To have some extra variables exported, you need to save those export commands in your EC2 machine .bashrc file
More details can be easily found on the Internet, for example here
I am using gcloud ssh to connect gce.
> gcloud compute --project "first-medium-2****8" ssh --zone "us-east1-b" "instance-2"
I entered the above command to powershell ,but it replies
>Using username "hogehoge".
>Authenticating with public key "DESKTOP-****hogehoge"
and stops. Nothing happened after all.
Yesterday I did the same thing and there was no problem.
But today, I can't. I tried gcloud init and reinstalled the gcloud.
But nothing changed. What should I do to solve this problem?
Additonal information.
OS Windows10
Google Cloud SDK 237.0.0
PowerShell 5.1.17134.590
Putty 0.70 (only one installation)
note1:I found I could use cloud shell without problem.
But, cloud shell has timeout.So I prefer gcloud to cloud shell.
note2:When I use cloudshell, it connects as "tomotomo".
Not "hogehoge" which username when I use gcloud.
When I run "gcloud compute ssh VM_NAME --verbosity=debug --log-http"
it replies
>DEBUG: SSH Known Hosts File [C:\Users\hogehoge\.ssh\google_compute_known_hosts] could not be opened: Unable to read file
[C:\Users\hogehoge\.ssh\google_compute_known_hosts]: [Errno 2] No such file or directory: u'C:\\Users\\hogehoge\\.ssh\\goo
gle_compute_known_hosts'
DEBUG: Current SSH keys in project: [u'tomotomo:ssh-rsa AAAAB***
DEBUG: Running command [C:\Users\hogehoge\AppData\Local\Google\Cloud SDK\google-cloud-sdk\bin\sdk\putty.exe -t -i C:\User
s\hogehoge\.ssh\google_compute_engine.ppk hogehoge#3*****].
DEBUG: Executing command: [u'C:\\Users\\hogehoge\\AppData\\Local\\Google\\Cloud SDK\\google-cloud-sdk\\bin\\sdk\\putty.ex
e', u'-t', u'-i', u'C:\\Users\\hogehoge\\.ssh\\google_compute_engine.ppk', u'hogehoge#3*****']
It was very long, so I only extract which I think important.
Running
putty -cleanup
solves this problem.
Putty saves some information in registry.(IP address,public key and so on)
This command removes those registries and random seed file.
Running "putty -cleanup" as per #redpawn fixed the issue.
in AWS I am trying to join ubuntu EC2 instance to AWS simple directory. I used steps described here to manually join instance.
First i Connected to instance using PuTTY
In step 4 i executed the following command
$ sudo apt-get -y install sssd realmd krb5-user samba-common packagekit adcli
As mentioned in the article, i was presented with several pop-up configuration screens and i left the fields in these screens blank.
then i tried to join the instance to the directory with the following command.
$ sudo realm join -U user#INTERNAL.MYCOMPANY.COM internal.mycompany.com --verbose
Since instance has Ubuntu 16.04, i entered the domain name portion of the username with all capital letters
However i am getting the No such realm found error
I am not linux expert, infact this is the first time i am using linux. All other instances in our network are windows.
Can anyone guide me what could be the issue here?
found it
To configure an EC2 instance running Linux to use static DNS server entries, use a text editor such as vim to edit the file /etc/dhcp/dhclient.conf and add the following line to the end of the file:
supersede domain-name-servers xxx.xxx.xxx.xxx, xxx.xxx.xxx.xxx;
Where xxx.xxx.xxx.xxx is the IP address of a DNS server that you want the instance to use
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-static-dns-ubuntu-debian/
and then try to join using
$ sudo realm join -U user#INTERNAL.MYCOMPANY.COM internal.mycompany.com --verbose
I am baking an image on top of Amazon linux image.
I need to run a service as ec2-user.
Is it possible to run a launch script of any kind as user other than root?
I'm assuming you're going to put the command under UserData.
Scripts entered as user data are executed as the root user, so do not use the sudo command in the script. Remember that any files you create will be owned by root; if you need non-root users to have file access, you should modify the permissions accordingly in the script. Also, because the script is not run interactively, you cannot include commands that require user feedback (such as yum update without the -y flag).
Here's the full documentation discussing topic
Use this:
su ec2-user -c 'your commands go here'
I am very new to the Amazon Web Services and have been trying a learn-by-doing approach with them.
In summary I was trying to set up Git with the elastic beanstalk command line interface for my web-app. However, I wanted to use my SSH key-pair to authenticate (aws-access-id, secret) and in my naivety and ignorance, I just supplied this information (the SSH key files) and now I can't get it to work. More specifically stated below.
I have my project directory with Git set up so that it works. I then open the git bash window MINGW64 (I am on Windows 10) and attempt to set up eb.
$ eb init
It then tells me that my credentials are not set up and asks me for aws-access-id and the secret. I had just set up the SSH key-pair and try to enter these files; what's the harm in trying? EB failure, it turns out. Now, the instances seem to run fine still, looking at their status on the AWS console website. However, whatever I type into the bash:
$ eb init
$ eb status
$ eb deploy
$
There is no output. Not even an error. It just silently returns to awaiting a new command from me.
When using the --debug option with these commands, a long list of operations is returned, ending with
botocore.parsers.ResponseParserError: Unable to parse response (no element found: line 1, column 0), invalid XML received:
b''
I thought I would be able to log out or something the like, so that I could enter proper credentials which I messed up from the beginning. I restarted the web-app from the AWS webpage interface and restarted my PC. No success.
Thanks in advance.
EDIT:
I also tried reinstalling awscli and awsebcli:
pip uninstall awsebcli
pip uninstall awscli
pip install awscli
pip install awsebcli --upgrade --user
Problem persists, but now there is one output (previously seen only upon --debug option):
$ eb init
ERROR: ResponseParserError - Unable to parse response (no element found: line 1, column 0), invalid XML received:
b''
$
It sounds like you have replaced your AWS credentials in ~/.aws/credentials and/or ~/.aws/config file(s) with your SSH key. You could manually replace these or execute aws configure if you have the AWS CLI installed.