I can automatically shutdown a sagemaker instance as indicated here, by using lifecycle configuration
https://stackoverflow.com/questions/53609409/automatically-stop-sagemaker-notebook-instance-after-inactivity
Is there a way to achieve this programmatically, by means of any ''image terminal'' command?
By image terminal, it is meant the Linux shell that can be opened via ''Launcher'' in ''Sagemaker Studio''
My use case is large computational jobs, where the idle inactive time in the automatic solution would be quite expensive.
It would be useful to have a shutdown_instance(), to be added as last command in lengthy shell scripts
Related
I get disconnect every now and then when running a piece of code in Jupyter Notebooks on Sagemaker. I usually just restart my notebook and run all the cells again. However, I want to know if there is a way to reconnect to my instance without having to lose my progress. At the minute, it shows that there is "No Kernel" at the bottom bar, but my file seems active in the kernel sessions tab. Can I recover my notebook's variables and contents? Also, is there a way to prevent future kernel disconnections?
Note that I reverted back to tornado = 5.1.1, which seems to decrease the number of disconnections, but it still happens every now and then.
Often, disconnections will be caused by inactivity because a job is running for a long time with no user input. If it's pre-processing that's taking a long time, you could increase the instance size of the processing job so that it executes faster, or increase the instance count. If you're using EMR, you can now run an EMR Spark query directly on the EMR cluster since December 2021:
https://aws.amazon.com/about-aws/whats-new/2021/12/amazon-sagemaker-studio-data-notebook-integration-emr/
There's a useful blog here https://aws.amazon.com/blogs/machine-learning/build-amazon-sagemaker-notebooks-backed-by-spark-in-amazon-emr/ which is helpful in getting you up and running.
Please let me know if you need more information, or vote for the answer if it's useful. :-)
For me a quick solution was to open a Terminal instead, save the notebook file as a Pytohn file, and run it from the terminal within Sagemaker.
I am looking for a way to execute a script at instance launch in Google cloud platform similar to user data in AWS. I check 'Startup script' but it is executed at every boot. is there any way to achieve it?
Yes, accordingly to the documentation startup script runs on each boot and there's no option available to change this behavior:
Compute Engine lets you create and run your own startup scripts on
your virtual machine (VM) instances to perform automated tasks every
time your instance boots up. Startup scripts can perform actions such
as installing software, performing updates, turning on services, and
any other tasks defined in the script.
To solve this issue you can use this workaround:
Set up a flag, such as a file on the disk, when your startup script runs first time.
Check existence of this flag in your startup script and exit without any action if the flag exists.
I have a AWS EC2 Windows (2008 R2 instance) which I want to start-stop using command/script from my local machine and schedule as per my usage.
I also want couple of my programs running on the EC2 instance to get start when instance starts. These programs currently are started using a bat file present in the instance.
I did following till now for the same:
1- I have an AWS user created in AWS IAM and using auth_id and key for that user for using EC2 apis and command line utilities.
2- To start and stop instance I'm using command line utilities from EC2 Util.
start ->ec2-start-instances i-instanceID
stop ->ec2-stop-instances i-instanceID
3- To schedule it I've added this to my windows scheduler.
4- Added user data for the instance in the AWS management console. My user data looks like this:
<script>
C:\Services\my_application.lnk
</script>
5- I can see the user data is present in my EC2 instance at C:\ProgramFiles\Amazon\Ec2ConfigServer\Scripts\UserScript
6- In C:\Program Files\Amazon\Ec2ConfigService\Settings\confi.xml the values of Ec2SetPassword and Ec2HandleUserData were changed to enabled and added true was added as well.
I'm facing following issues:
1- The user data scripts does not execute every time the instance is started. I'm not able to figure out why.
2- The changes made in Ec2ConfigService\Settings\confi.xml are getting reverted to the default values when the instance is restarted.
I feel this is common use case, and would like to know the best practices and approach taken for automating EC2 operations.
I also need help in starting programs on my instance- where am I going wrong or missing, what else needs to be done etc?
userdata is only executed the very first time that the instance is created. This is by design.
You've got a couple of options - all of which use your userdata script
Copy the my_application.lnk to the startup folder
Register the application in the registry "run" start key (http://blogs.msdn.com/b/powershell/archive/2006/04/25/how-to-access-or-modify-startup-items-in-the-window-registry.aspx)
Register it with the task scheduler to configure it to execute on startup (http://technet.microsoft.com/en-us/library/bb490996.aspx)
I have a simple pure C program, that takes an integer as input, runs for a while (let's say an hour) and then returns me a text file. I want to run this program 1000 times with input integers from 1 to 1000.
Currently I'm running this program in parallel (4 processors), what takes 250 hours. The program is such that it fits in a AWS micro instance (I've tested it). Would it be possible to use 1000 micro instances in AWS to do the whole job in one hour? (at a cost of ~20$ - $0.02/instance)?
If it is possible, does anybody have some guidlines on how to do that?
If it is not, does anybody have an also low-budget alternative to that?
Thanks a lot!
In order to achieve this, you will need to:
Create a S3 bucket to store your bootstrapping scripts, application, input data and output data
Custom Lightweight AMI: you might want to create a custom lightweight AMI which knows about downloading the bootstrapping script
Bootstrapping Script: will download your software from your S3 bucket, parse a custom instance tag which will contain the integer [1..1000] and download any additional data.
Your application: does the processing stuff.
End of processing script is which uploads the result to a another result S3 bucket and terminates the instance, you might also want to send a SNS notification to communicate the end of processing status.
If you need result consolidation you might want to create another instance and use it as a coordinator, waiting for all "end of processing" notifications in order to finish the processing. In this case you might consider using in the future the Amazon's Hadoop map reduce engine, since it will do almost all this heavy lifting for free.
You didn't specify what language you'd like to do this in and since the app you use to deploy your program doesn't have to be in the same language I would recommend C#. There are a number of examples around the web on how to programmatically spawn new Amazon Instances using the SDK. For example:
How to start an Amazon EC2 instance programmatically in .NET
You can create your own AMIs with the program already present but that might end up being a pain if you want to make adjustments to it since it will require you to recreate the entire AMI. I'd recommend creating an extra instance or simply hosting the program in a location which is accessible from a public URL. Then I would create some kind of service which would be installed on the AMI ahead of time to allow me to specify the URL to download the app from along with whatever command line parameters I wanted for that particular instance.
Hope this helps.
Be carefull by default you can only spawn 20 instance by zone. You need to ask amazon in order to use more instances.
If you want low cost but don't care about delay you should use spot instances.
I am currently using AWS ElasticBeanStalk and I was curious as to how (as in internally) it knows that when you fire up an instance (or it automatically does with scaling), to unpack the zip I uploaded as a version? Is there some enviroment setting that looks up my zip in my S3 bucket and then unpacks automatically for every instance running in that environment?
If so, could this be used to automate a task such as run an SQL query on boot-up (instance deployment) too? Are these automated tasks changeable or viewable at all?
Thanks
I don't know how beanstalk knows which version to download and unpack, but running a task on start-up is trivial. Check out cloud-init, a tool written by Ubuntu that's now packaged in Amazon Linux. It allows you to pass arbitrary shell scripts into the UserData section of the instance configuration, and those shell scripts will run on startup.
It's a great way to bootstrap instances on startup, which avoids the soul-sucking misery of managing AMIs.
A quick (possibly non-applicable) warning: If you're running a SQL query on a database that lives on the beanstalk AMI, you're pretty much guaranteed to lose your database at some point. Those machines are designed to be entirely transient. Do not put databases on them. See this answer for more details.
Since your goal seems to be to run custom configuration tasks, the answer is yes, there is a way to do that. You can define custom actions in an .ebextensions file packaged with your app. For example, you can configure a command to run every time a new machine is deployed:
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html#linux-commands