I am working on my graduation project and It's my first time dealing with spark and EC2
so I am following the steps in this blog
http://www.supergloo.com/fieldnotes/apache-spark-cluster-amazon-ec2-tutorial/#comment-3843
The problem is he is working on MAC and I don't know how to make these commands work on Windows (CMD).
for example this command
ec2/spark-ec2 –key-pair=courseexample –identity-file=courseexample.pem launch spark-cluster-example
Any Help ?!
try running it in the following way (from the same folder):
python -Wdefault "ec2\spark_ec2.py" –key-pair=courseexample –identity-file=courseexample.pem launch spark-cluster-example
if you don't know how to open console in windows - just press "Start->Run", type cmd and hit enter, then you would need to havigate to your Spark home folder, and execute the above command.
NOTE: I don't currently own a Windows machine, so I haven't tried this command myslef.
Related
I am stuck in a technical issue on a project and I think you the forum could help me out.
I have an EC2 Instance Type:p2.xlarge running on AWS, I cloned a repository in this instance which requires pytorch and cuda dependencies(this point has been taken care of).
Now, The issue is that I wanna work & run this code-base(which is is AWS instance now) somehow in my local pyCHARM IDE. In short, I didn't have proper resources on my laptop to run the repository, so I have to run in an AWS instance but for debugging purposes the local IDE would be a great option.
Is it possible to do that?. In other words, we can do SSH into AWS instance and run code, but all will be done through command line, if we could SSH through PYCHARM and can see the code in AWS here in local machine within PYCHARM and change, debug or run it as it was local but actually it gets executed in the instance.
Please suggest a solution to it.
Thanks in advance.
EDIT-1:
After following, #Cromulent suggestion, I have arrived here
Setting the remote:
Upload happening within the local & remote repo.
I still didn't understand the requirement of syncing the local and remote folders, when I only want to open the remote folder in my PYCHARM IDE and work on it.
I think after this setup, I have to change the code in local copy and the PYCHARM will sync the code in remote copy. How will I be running(using resources-GPUs of the remote Instance, not my local machine.) the remote code in PYCHARM in this scenario, I am just syncing it, for running again I have to ssh through command line and run the script(This does not serve the purpose)?
EDIT-2:
After #Cromulent suggestions.
Actually, it did work, but still, I am not able to run the remote code locally.
I am getting the below error while running any remote script. If I run the same script using ssh in the terminal, the scripts run normally. I tried to fix the problem using this post on StackOverflow, but it didn't work too.
ssh://ubuntu#ec2-52-41-247-169.us-west-2.compute.amazonaws.com:22/home/ubuntu/anaconda3/bin/python -u <08ad9807-3477-4916-96ce-ba6155e3ff4c>/home/ubuntu/InsightProject/scripts/download_flownet2.py
/home/ubuntu/anaconda3/bin/python: can't open file '<08ad9807-3477-4916-96ce-ba6155e3ff4c>/home/ubuntu/InsightProject/scripts/download_flownet2.py': [Errno 2] No such file or directory
The below is the screenshot for the above problem:
PyCharm Professional supports remote Python interpreters (either the globally installed Python interpreter or a virtualenv). It works by creating an SSH connection to the server and then running the code on the remote host. The results are then displayed locally in PyCharm Professional. You can also do remote debugging as well.
You MUST be using the professional version of PyCharm though. The free community version does not support this feature.
You can find the documentation here:
https://www.jetbrains.com/help/pycharm/configuring-remote-interpreters-via-ssh.html
One more solution is to deploy a Jupyter Notebook on your remote server. Then you will be able to use it from PyCharm Professional Edition.
Don't forget to make rules for the jupyter ports (e.g. allow all 8888) in your AWS console and in your instance.
To configure a remote interpreter for your notebook do this (source):
Open the Jupyter Notebook page of the Settings/Preferences dialog.
On this page, select or clear the Markdown cells rendering enabled option, and specify the username and password. Note that for the
single-user notebooks these fields are optional - leave them blank.
Fill in the username (for JupyterHub) and password.
Click the link Configure remote interpreter. You'll find yourself at the Project Interpreter page.
Configure the remote interpreter, as described in the section Configuring Python Interpreter.
You will want to configure a remote interpreter.
I tried the above approach but it didn't work for me. I have edited my post so that I can get additional input from the community, but I didn't any after the first answer was posted.
My friend actually figured out a secondary way to fix the issue. He actually uses "NOMACHINE" on the local machine and open connection to the remote desktop. Then you can directly install PYCHARM in the remote machine and work in there. I hope this will help others.
The solution is in his blog post. (Thanks to Shaobo Guan)
Another solution would be to use VNC instead of NoMachine
My GCP Compute Engine instance running WS 2012 does not execute an executable command while using using a startup script.
Summary of steps followed:
I created a Compute Engine instance with WS 2012.
Set Metadata keys windows-startup-script-cmd with value = `"c:/prog/1.exe".
copied all needed files to local disk.(exact folder c:/prog/).
Shut down VM. (by GCP console).
Started instance from the GCP console.
And it seems that the command "c:/prog/1.exe" did not run
I saw "1.exe" process running in the Task Manager but it did not do anything . (It should have created a specific file to show that it ran successfully).
Please, tell me if I did something wrong, Or if I missed anything.
Thanks for your help .
The windows-startup-script-cmd is metadata key to run cmd startup scripts on Windows instances. The cmd startup scripts are those with .cmd or .bat extensions. You can run your executable files through a batch file. How? visit this answer on stackoverflow.
Thank you all who use their time to help others. Thank you in advance.
I am taking a class on EDX for Linux command line. I have attached the instructions I have followed to set up Red hat using Windows 10. But the SSH for isn't working it says that RSA Key not found. (Attachment didn't work)
I paste it here
"The recommended way to access your Red Hat Enterprise Linux cloud instance for this course is
to use Secure Shell (ssh) to get an interactive shell on the system.
Step 1: Getting SSH
If you are using macOS or Linux: The OpenSSH command (ssh) should already be installed on
your computer. Start a terminal program to get a shell, and proceed to the next step.
If you are using Microsoft Windows, you will need to get an SSH client as follows:
● Option 1: Download Git for Windows from https://git-scm.com/download/win and install
it using the default settings. When it is installed, right-click on your desktop (not an icon
or a file) and select Git Bash Here to open a Git Bash command prompt. An SSH client is
provided with Git Bash and you can continue with Step 2.
● Option 2: If you have the PuTTY client, which can be downloaded from
http://www.chiark.greenend.org.uk/~sgtatham/putty/, and you are familiar with that
tool, you can use that to connect to your instance. AWS provides a tutorial on this at
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html.
Step 2: Connecting to your instance with SSH
From the Amazon EC2 console, in the left-side menu under INSTANCES, click Instances to
display a list of the instances you have running. This is probably just the one you set up for this
course. Select its checkbox and click Connect:"
Also, I have another PC I built running on Ubuntu. Can I use that PC with the RSA by logging into AWS account if there is no solution.
I have researched this question and I am stuck here trying to find a solution so I can start class.
I have python script that execute android emulator
how I can execute this python script on startup?
the problem is that I open ssh session execute the android emulator and when I close the ssh session the emulator stop
You meant to say you want a script to be executed every time you logged in to a machine. If I understood your requirement correctly, then have a look at the link
I have a Jenkins server on OS X 10.7, which polls a subversion server, builds the code and packages the app. The last step that I need to complete is deploying the app on a remote host, which is a windows share. Note that my domain account has write access to the target folder and the volume is mounted. I've tried using a shell script build step:
sudo cp "path/to/app" "/Volumes/path/to/target"
However i get a "no tty" response. I was able to run this command succesfully in Terminal, but not as a build step in Jenkins.
Does this have something to do with the user being used when starting up Jenkins? As a side note, the default user.name is jenkins and my JENKINS_HOME resides in /Users/Shared/Jenkins. I would appreciate any help as to how to achieve this.
Your immediate problem seems to be that you are running Jenkins in the background and sudo wants to input a password. Run Jenkins in the foreground with $ java -jar jenkins.war.
However, this most probably won't solve your problem as you'll be asked to enter a password when the command runs - from the terminal you started Jenkins from (presumably it's not what you want). You need to find a way to copy your files without needing root permissions. In general, it it not a good idea to rely on administrative permissions in your builds (there are exceptions, but your case is not it).