I'm preparing to get in to the world of cloud computing.
My first question is:
Is it possible to programmatically create a new, or duplicate an existing VM from my server?
Project Background
I provide a file processing service, and as it's been growing I need to offer a better service.
Project Requirement
Machine specs:
HDD: Min 16gb
CPU: Min 1 core
RAM: Min 2
GB GPU: Min CUDA 10.1 compatible
What I'm thinking is the following steps:
User uploads a file
A dedicated VM is created for that specific file inside Google Cloud Compute
The file is sent to the VM
File is processed using a Anaconda environment
Results are downloaded to local server
Dedicated VM is removed
Results are served to user
How is this accomplished?
PS: I'm looking for resources and advice. Not code.
Your question is a perfect formulation of the concept of Google Cloud Run. At the highest level concept, you create a Docker image (think of it like a VM) and then register that Docker image with GCP Cloud Run. When a trigger occurs, GCP will spin up an instance of that Docker container and pass in information about the cause of that trigger (a file created in GCS or a REST request or others ...). What you do in your container is up to you. You have full power of the Linux environment (under Docker) to do as you like. When your request ends, the container is spun down. You are only billed for the compute resources you use. If your container (VM) isn't being used, you pay nothing until the next trigger.
An alternative to Cloud Run is Cloud Functions. This is a higher level abstraction where instead of providing a Docker container, you provide the body of a function (JavaScript, Java, Python or others) and the request is passed to that function when a trigger occurs. Which you use is mostly personal choice (you didn't elaborate on "File is processed").
References:
Cloud Run
Cloud Functions
Related
I am planning to use Azure VMSS for deploying a set of spring boot apps. I am planning to create a custom linux VM image with all the required softwares/utilities as well as the required directory structure and configure this image in VMSS. We use jenkins as CI/CD tool and Git as source code repo. What is the best way to build and deploy these spring boot apps on VMSS?
I think one way is to write a custom script extension which downloads code from Git repo and then starts these spring boot apps. I believe this script will then get executed every time a new VM is provisioned.
But what about cases where already multiple VMs are running on top of minimum scale instance count. I believe a manual restart will not trigger the CSE script to run on these already running VMs right?
Could anyone advise the best way to handle this?
Also once a VM is deallocated due to auto scale down, what is the best/cost optimal way to back up the log files from VM to storage (blob or file share)?
You could enable Automatically tear down virtual machines after every use in the organization settings/project setting >> agent pool >> VMSS agent pool >> settings. Then, a new VM instance is used for every job. After running a job, the VM will go offline and be reimaged before it picks up another job. The Custom Script Extension will be executed on every virtual machine in the scaleset immediately after it is created or reimaged. Here is the reference document: Create the scale set agent pool.
To back up the log files from VM, you could refer to Troubleshoot and support about related file path on the target virtual machine.
I have a software that process some files. What I need is:
start a default image on google cloud (I think docker should be a good solution) using an API or a run command
download files from google storage
process it, run my software using those downloaded files
upload the result to google storage
shut the image down, expecting not to be billed anymore
What I do know is how to create my image hehe. But I can't find any info saying me what google cloud service should I use or even if I could do it like I'm thinking. I think I'm not using the right keywords to find what i need.
I was looking at Kubernetes, but i couldn't figure out how to manipulate those instances to execute a one time processing.
[EDIT]
Explaining better the process I have an app that receive images and send it to Google storage. After that, I need to process that images, apply filters, georeferencing, split image etc. So I want to start a docker image to process it and upload the results to google cloud again.
If you are using any of the runtimes supported by Google Cloud Functions, they are easiest way to do those kind of operations (i.e. fetch something from Google Cloud Storage, perform some actions on those files and upload them again). The Cloud Functions will be triggered by an event of your choice, and after the job, it will die.
Next option in terms of complexity would be to deploy a Google App Engine application in standard environment. It allows you to deploy your own application written in any of the supported languages for this environment. While there is traffic in your application, you will have instances serving, but the number of instances running can go down to 0 when they are not serving, which would mean less cost.
Another option would be Google App Engine in flexible environment. This product allows you to deploy your application in any custom runtime. This option has always at least one instance running, so it would never shut down.
Lastly, you can use Google Compute Engine to "create and run virtual machines on Google infrastructure". Otherwise than GAE, this is not that managed by Google, which means that most of the configuration is up to you. In this case, you would need to programmatically indicate your VM to shut down after you have finished your operations.
Based on your edit where you stated that you already have an app that is inserting images into Google Cloud Storage, your easiest option would be to use Cloud Functions that are triggered by additions, changes, or deletions to objects in Cloud Storage buckets.
You can follow the Cloud Functions tutorial for Cloud Storage to get an idea of the generic process and then implement your own code that handles your specific tasks. There are other tutorials like the Imagemagick tutorial for Cloud Functions that might also be relevant to the type of processing you intend to do.
Cloud Functions is probably your lightest weight approach. You could of course do more full scale applications, but that is likely overkill, more expensive, and more complex. You can write your processing code in Node.js, Python, or Go.
I am very new to Cloud Foundry/Bosh and have a set of basic questions.
1) Droplet vs Garden container: I understand that droplet contains source code + build pack and these droplets are executed in garden container. IMHO, containers are good to transport to other system. Why there is intermediate notion of droplets? Should container by them-self not create the droplets?
2) Diego cell: What is the role of Diego cell (I assume that its job is to only start/stop garden containers) ? Are Diego cell platform dependent (eg. particular cell can run only windows-garden or other can run linux-garden container) ? Do we need one cell per container?
3) In the description of Diego cell, I read "Each application VM has a Diego Cell that executes application start and stop actions locally, manages the VM’s containers, and reports app status and other data to the BBS and Loggregator."
What is application VM mentioned here? Does it mean container?
4) Lets assume, I use Bosh to create my cloud foundry instance. After some time, I need to scale my system to two VMs (due to increase in load). Do I need to create a new Manifest for second VM (As the earlier manifest will also deploy the entire CF on this VM also)?
A container is, roughly speaking, a root file system image together with some things like resource limits and metadata about what volumes to mount, what processes to run, etc.
Garden is an API for creating and running container specifications. Anyone can write a server that implements the Garden API, the core Cloud Foundry teams maintain garden-linux, garden-runC, and garden-windows implementations.
A droplet is a "built" artifact created from source code that is typically mounted or streamed into a Garden container and then run. There are times where you do not want a separate droplet, and want to have your root file system as well as all source code and/or built artifacts baked into a single image. However, often, you do want this separation between the droplet, which represents your code, and the root file system. One major benefit is that CVEs present in lower level dependencies that are common to most containers can be uniformly repaired across all tenants and all running applications on the Cloud Foundry platform without any developers having to re-push their code. E.g. if there is a new patch required for something like openssl, and your Cloud Foundry installation has thousands of developers and hundreds of thousands of running Garden containers, it would be much better if an operator could roll out the openssl patch to all containers with a single command.
The Diego cell is a VM that's part of the Cloud Foundry architecture. Cloud Foundry itself is a distributed system with different components responsible for different things. There is a component responsible for user authorization and authentication, there are components for aggregating logs from applications, there is a component responsible for providing the developer-facing API for creating, scaling, and managing applications, etc. The Diego cell's are responsible for essentially taking requests to run containerized workloads, and running them. User requests to run an application are consumed by the user-facing API, and translated to a request to the Diego backend. Diego itself has several components, including a scheduler, and the scheduler's job is to select which cell to do a given piece of work.
You can think of the cell has having two components: (1) a Garden server for running containers, and (2) a representative that can represent that Garden server to the Diego scheduler, so rather than Garden having any Diego-specific knowledge (Garden can function in a standalone manner), the scheduler instead talks to each Garden's Diego representative on that same cell.
I'm not sure what "application VM" means in the quote you pulled out. Each application running on Cloud Foundry can be run with multiple parallel instances (for fault tolerance, better concurrency, etc.). Each application instance is running as a Garden container in some Diego cell. A production deployment of Cloud Foundry will have many Diego cells. Each cell can run many (up to hundreds) of Garden containers. For better fault tolerance, the Diego scheduler will attempt to place the instances of a given application on different Diego cells, rather than cramming them all into the same cell, since if that single cell goes down, your whole application goes down.
You do not need to create a new manifest to scale up BOSH deployments. Just change the instances value of whatever job/instance group you want to have more VMs of, and re-run bosh deploy.
1) The droplet is a container image that is stored persistently when you upload your application with a cf push. This image will be re-used whenever the system creates a new container with the same image (for example, when you restart your application, or scale to multiple instances). Without the droplet, you would have to re-upload your application and create the container image every time you wanted a new instance.
2) The Diego cell runs in a large VM that hosts many containers. A typical Diego cell might be 32G in size, while a typical container might be 1G. Diego cells currently only run Linux-Garden containers.
3) The application VM is just the VM that is hosting the Diego cell. I found the sentence a bit confusing because I tend to use the term Diego cell to refer both to the cell software and to the "application VM" which is hosting it.
4) Bosh will use multiple VMs to deploy Cloud Foundry. Single-VM deployments do exist (see for example, http://pivotal.io/pcf-dev) but they are not deployed with Bosh.
I Have successfully implemented predictionIO engine templates. I can deploy an engine with
$pio deploy -- --driver-memory xG
But How can I run recommendation (or any other) engine as a service? I want to log all entries to a file specified for reference if some issues occur.
Also its mentioned that for small deployments it is better not to use a distributed setup. I have a json formatted dataset for text classification template about 2MB in size and it requires about 8GB of memory for the engine to train and deploy. Does this fit into small deployment category?
pio deploy
Will run the engine as a service. Logging goes to pio.log and you could add your own custom logging. I'd say so long as you have more than 8GB in RAM to spare, stick to the single machine aka "small deployment".
Let's say that I setup my own cloud using the open source cloud foundry implementation provided on cloudfoundry.org. Will each app that I deploy be run as a separate user? Or is there any of VMWare's virtualization technology in use here? E.g. would each app run in a separate virtual machine or anything like that? How can I configure the memory, cpu, and disk resource limits for each app?
I asked this on the mailing list. Here's the response I got:
If your DEA is configured to run in secure mode, then each app runs as its own user and process isolation is used to protect them. We are moving toward a model of using linux cgroups http://en.wikipedia.org/wiki/Cgroups when on linux, using the warden cgroup wrappers that are already in our source tree.
VM based isolation for a single app is pretty heavy weight, but we have long term plans to provide this for apps that need/desire it. (As opposed to the warden/cgroup work which is a near term project)
Since this is related to the open source for cloud foundry, you can try asking your question on https://groups.google.com/a/cloudfoundry.org/group/vcap-dev
You should get a quick response there!