Google recently added support for GPUs in their cloud service.
I'm trying to follow the instructions found here to start a machine with a GPU. Running this script on Windows:
gcloud beta compute instances create gpu-instance-1^
--machine-type n1-standard-2^
--zone us-east1-d^
--accelerator type=nvidia-tesla-k80,count=1^
--image-family ubuntu-1604-lts^
--image-project ubuntu-os-cloud^
--maintenance-policy TERMINATE^
--restart-on-failure^
with gcloud command line tool version 146.0.0 fails, saying:
ERROR: (gcloud.beta.compute.instances.create) unknown collection [compute.acceleratorTypes]
Any ideas?
Was never able to get the gcloud utility working. Using the API did work. Of note, when posting the API request (instructions on the same page as the gcloud instructions, here) the key that creates an instance with a GPU is guestAccelerators. This key does not have an analogous option in gcloud.
Copying the API request as it appears on the instructions page linked above.
POST https://www.googleapis.com/compute/beta/projects/[PROJECT_ID]/zones/[ZONE]/instances?key={YOUR_API_KEY}
{
"machineType": "https://www.googleapis.com/compute/beta/projects/[PROJECT_ID]/zones/[ZONE]/machineTypes/n1-highmem-2",
"disks":
[
{
"type": "PERSISTENT",
"initializeParams":
{
"diskSizeGb": "[DISK_SIZE]",
"sourceImage": "https://www.googleapis.com/compute/beta/projects/[IMAGE_PROJECT]/global/images/family/[IMAGE_FAMILY]"
},
"boot": true
}
],
"name": "[INSTANCE_NAME]",
"networkInterfaces":
[
{
"network": "https://www.googleapis.com/compute/beta/projects/[PROJECT_ID]/global/networks/[NETWORK]"
}
],
"guestAccelerators":
[
{
"acceleratorCount": [ACCELERATOR_COUNT],
"acceleratorType": "https://www.googleapis.com/compute/beta/projects/[PROJECT_ID]/zones/[ZONE]/acceleratorTypes/[ACCELERATOR_TYPE]"
}
],
"scheduling":
{
"onHostMaintenance": "terminate",
"automaticRestart": true
},
"metadata":
{
"items":
[
{
"key": "startup-script",
"value": "[STARTUP_SCRIPT]"
}
]
}
}
Sometimes you need to ensure you have the latest version of the gcloud utility installed in order to use certain GCP features.
Try running this command or read the below docs on how to update your gcloud utility:
gcloud components update
https://cloud.google.com/sdk/gcloud/reference/components/update
Related
Im using hashicorp's Packer to create machine images for the google cloud (AMI for Amazon). I want every instance to run a script once the instance is created on the cloud. As i understand from the Packer docs, i could use the startup_script_file to do this. Now i got this working but it seems that the script is only runned once, on image creation resulting in the same output on every running instance. How can i trigger this script only on instance creation such that i can have different output for every instance?
packer config:
{
"builders": [{
"type": "googlecompute",
"project_id": "project-id",
"source_image": "debian-9-stretch-v20200805",
"ssh_username": "name",
"zone": "europe-west4-a",
"account_file": "secret-account-file.json",
"startup_script_file": "link to file"
}]
}
script:
#!/bin/bash
echo $((1 + RANDOM % 100)) > test.log #output of this remains the same on every created instance.
I am very new to AWS Step Functions and AWS Lambda Functions and could really use some help getting an EMR Cluster running through Step Functions. A sample of my current State Machine structure is shown by the following code
{
"Comment": "This is a test for running the structure of the CustomCreate job.",
"StartAt": "PreStep",
"States": {
"PreStep": {
"Comment": "Check that all the necessary files exist before running the job.",
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:XXXXXXXXXX:function:CustomCreate-PreStep-Function",
"Next": "Run Job Choice"
},
"Run Job Choice": {
"Comment": "This step chooses whether or not to go forward with running the main job.",
"Type": "Choice",
"Choices": [
{
"Variable": "$.FoundNecessaryFiles",
"BooleanEquals": true,
"Next": "Spin Up Cluster"
},
{
"Variable": "$.FoundNecessaryFiles",
"BooleanEquals": false,
"Next": "Do Not Run Job"
}
]
},
"Do Not Run Job": {
"Comment": "This step triggers if the PreStep fails and the job should not run.",
"Type": "Fail",
"Cause": "PreStep unsuccessful"
},
"Spin Up Cluster": {
"Comment": "Spins up the EMR Cluster.",
"Type": "Pass",
"Next": "Update Env"
},
"Update Env": {
"Comment": "Update the environment variables in the EMR Cluster.",
"Type": "Pass",
"Next": "Run Job"
},
"Run Job": {
"Comment": "Add steps to the EMR Cluster.",
"Type": "Pass",
"End": true
}
}
}
Which is shown by the following workflow diagram
The PreStep and Run Job Choice tasks use a simple Lambda Function to check that the files necessary to run this job exist on my S3 Bucket, then go to spin up the cluster provided that the necessary files are found. These tasks are working properly.
What I am not sure about is how to handle the EMR Cluster related steps.
In my current structure, the first task is to spin up an EMR Cluster. this could be done through directly using the Step Function JSON, or preferably, using a JSON Cluster Config file (titled EMR-cluster-setup.json) I have located on my S3 Bucket.
My next task is to update the EMR Cluster environment variables. I have a .sh script located on my S3 Bucket that can do this. I also have a JSON file (titled EMR-RUN-Script.json) located on my S3 Bucket that will add a first step to the EMR Cluster that will run and source the .sh script. I just need to run that JSON file from within the EMR Cluster, which I do not know how to do using the Step Functions. The code for EMR-RUN-SCRIPT.json is shown below
[
{
"Name": "EMR-RUN-SCRIPT",
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "s3://us-east-1.elasticmapreduce/libs/script-runner/script-runner.jar",
"Args": [
"s3://PATH/TO/env_configs.sh"
]
}
}
]
My third task is to add a step that contains a spark-submit command to the EMR Cluster. This command is described in a JSON config file (titled EMR-RUN-STEP.json) located on my S3 Bucket that can be uploaded to the EMR Cluster in a similar manner to uploading the environment configs file in the previous step. The code for EMR-RUN-STEP.json is shown below
[
{
"Name": "EMR-RUN-STEP",
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args": [
"bash", "-c",
"source /home/hadoop/.bashrc && spark-submit --master yarn --conf spark.yarn.submit.waitAppCompletion=false --class CLASSPATH.TO.MAIN s3://PATH/TO/JAR/FILE"
]
}
}
]
Finally, I want to have a task that makes sure the EMR Cluster terminates after it completes its run.
I know there may be a lot involved within this question, but I would greatly appreciate any assistance with any of the issues described above. Whether it be following the structure I outlined above, or if you know of another solution, I am open to any form of help. Thank you in advance.
You need a terminate cluster step,
as documentation states:
https://docs.aws.amazon.com/step-functions/latest/dg/connect-emr.html
createCluster uses the same request syntax as runJobFlow, except for the following:
The field Instances.KeepJobFlowAliveWhenNoSteps is mandatory,
and must have the Boolean value TRUE.
So, you need a step to do this for you:
terminateCluster.sync - for me this is preferable over the simple terminateCluster as it waits for the cluster to actually terminate and you can handle any hangs here - you'll be using Standard step functions so the bit of extra time will not be billed
Shuts down a cluster (job flow).
terminateJobFlows The same as terminateCluster, but waits for the cluster to terminate.
ps.: if you are using termination protection you'll need an extra step to turn if off before you can terminate your cluster ;)
'KeepJobFlowAliveWhenNoSteps': False
add the above configurations to emr cluster creation script. it will auto terminate emr clusters when all the steps are completed emr boto3 config
That is what I get when I follow the instructions at the official Docker tutorial here: tutorial link
I uploaded my Dockerrun.aws.json file and followed all other instructions.
The logs show nothing even when I click Request:
If anyone has a clue as to what I need to do, ie. why would not having a default VPC even matter here? I have only used my AWS account to set up Linux Machine EC2 instances for a Deep Learning nanodegree at Udacity in the past (briefly tried to set up a VPC just for practice but am sure I deleted/terminated everything when I found out that is not included in the free tier).
The author of the official tutorial forgot to add that you have to add the tag to the image name in the Dockerrun.aws.json file per below in gedit or other editor where :firsttry is the tag:
{
"AWSEBDockerrunVersion": "1",
"Image": {
"Name": "hockeymonkey96/catnip:firsttry",
"Update": "true"
},
"Ports": [
{
"ContainerPort": "5000"
}
],
"Logging": "/var/log/nginx"
}
It works:
It seems that by default EMR deploys the Spark driver to one of the CORE nodes, resulting in the MASTER node being virtually un-utilized. Is it possible to run the driver program on the MASTER node instead? I have experimented with the --deploy-mode arguments to no avail.
Here is my instance groups JSON definition:
[
{
"InstanceGroupType": "MASTER",
"InstanceCount": 1,
"InstanceType": "m3.xlarge",
"Name": "Spark Master"
},
{
"InstanceGroupType": "CORE",
"InstanceCount": 3,
"InstanceType": "m3.xlarge",
"Name": "Spark Executors"
}
]
Here is my configurations JSON definition:
[
{
"Classification": "spark",
"Properties": {
"maximizeResourceAllocation": "true"
},
"Configurations": []
},
{
"Classification": "spark-env",
"Properties": {
},
"Configurations": [
{
"Classification": "export",
"Properties": {
},
"Configurations": [
]
}
]
}
]
Here is my steps JSON definition:
[
{
"Name": "example",
"Type": "SPARK",
"Args": [
"--class", "com.name.of.Class",
"/home/hadoop/myjar-assembly-1.0.jar"
],
"ActionOnFailure": "TERMINATE_CLUSTER"
}
]
I am using aws emr create-cluster with --release-label emr-4.3.0.
Setting the location of the driver
With spark-submit, the flag --deploy-mode can be used to select the location of the driver.
Submitting applications in client mode is advantageous when you are debugging and wish to quickly see the output of your application. For applications in production, the best practice is to run the application in cluster mode. This mode offers you a guarantee that the driver is always available during application execution. However, if you do use client mode and you submit applications from outside your EMR cluster (such as locally, on a laptop), keep in mind that the driver is running outside your EMR cluster and there will be higher latency for driver-executor communication.
https://blogs.aws.amazon.com/bigdata/post/Tx578UTQUV7LRP/Submitting-User-Applications-with-spark-submit
I don't think it is a waste. When running Spark on EMR, the master node will run Yarn RM, Livy Server, and maybe other applications you selected. And if you run in client mode, the majority of the driver program will run on the master node as well.
Note that the driver program could be heavier than the tasks on executors, e.g. collecting all results from all executors, in which case you need to allocate enough resources to your master node if it is where the driver program is running.
Try as I might, I cannot get the import-image task to work. I'm looking for a working example that I can reproduce, preferably starting with a "raw" disk image.
Most recent problems:
"Unsupported kernel version" when using an image that works fine when converted with the mouse instead of the API (posted to EC2 forum, no response: https://forums.aws.amazon.com/thread.jspa?threadID=221844)
"No valid partitions" when using a VirtualBox VMDK image that boots just fine in VirtualBox.
I ran into a similar issue when I tried importing FreeBSD bundled OVAs to it. According to the pre-requisites/checklist, Amazon does not yet support vmimporting of FreeBSD. That produces the "No valid partitions".
Also, if you use LUKS encrypted partitions it produced that same error for me, (Ubuntu).
For "Unsupported kernel version", here is my output of that same error:
c:\Users\XXXXX\Documents>aws ec2 describe-import-image-tasks --import-task-ids "import-ami-fgacu4yu"
{
"ImportImageTasks": [
{
"Status": "deleted",
"SnapshotDetails": [
{
"UserBucket": {
"S3Bucket": "myautomationbucket",
"S3Key": "ubuntu14.04-patched.ova"
},
"DiskImageSize": 843476480.0,
"Format": "VMDK"
}
],
"Description": "Optimus Custom Ubuntu14.04",
"StatusMessage": "ClientError: Unsupported kernel version 4.2.0-36-generic",
"ImportTaskId": "import-ami-XXXXXXXX"
}
]
}
According to AWS they posted a list of known good kernels however they are not verbose for my favorite flavor, Ubuntu.
http://docs.amazonaws.cn/en_us/AWSEC2/latest/WindowsGuide/VMImportPrerequisites.html
So what I had done is downgrade the kernel to their acceptable ones.
I obtained how to get what was "acceptable" by performing this command on an existing, known good running instance in my EC2:
c:\Users\XXXXXX\Documents>aws ec2 describe-instance-attribute --instance-id i-12345678 --attribute kernel --region us-east-1
{
"InstanceId": "i-12345678",
"KernelId": {
"Value": "aki-825ea7eb"
}
}
So this aki-824ea7eb is the supported kernel ID. That isn't very helpful, so after some research I realized that AWS may only have a list of supported kernels due to the limitation of their existing platform -- they are not running ESXi you know. ;)
I had searched and found this to be useful and followed the instructions for 13.04 https://www.linode.com/docs/tools-reference/custom-kernels-distros/run-a-distributionsupplied-kernel-with-pvgrub
I performed 1,2,3,4, but I had skipped steps 5,6,7,8... performed 9 and then 15.
And then when I performed them on my VM, repackaged the VM to an OVA and ran my vmimport, it successfully imported with an instance.
Hope this helps.