I want to create a new Google Cloud instance with Hardenedbsd iso. Hardenedbsd is a FreeBSD based OS. I checked public documentation on https://cloud.google.com/compute/docs/images/import-existing-image but I couldn't see FreeBSD on supported OS section.
Is there a way to do that?
FreeBSD works pretty well in GCE, the upload procedure of a custom image or making your own is quite easy I would say even better than with AWS, therefore high are the changes the same could apply for Hardenedbsd, the only "trick" is that after you have your raw disk you need to use gnu tar to upload the image:
gtar -cSzf freebsd.tar.gz disk.raw
To create the disk.raw I use this script https://github.com/fabrik-red/images/blob/master/fabrik.sh (root on ZFS) to read more about the procedures you could check: https://fabrik.red/post/google/
For testing or getting an idea, you could try FreeBSD 12.0
https://github.com/fabrik-red/images/releases/download/12.0/disk.tar.gz
I haven't tried working with any *BSD on Google Cloud Platform so take my words with a grain of salt.
You could try booting the instance in rescue mode (if supported) and perform a dd to write the Hardenedbsd to the main disk.
You could also take a look on Packer from Hashicorp which is meant to create OS images to be deployed on the cloud.
https://www.packer.io/docs/builders/googlecompute.html
Related
I'm new to Azure-Pipeline and struggling to put together a C++ oriented pipeline that uses camke which properly compiles, run tests and build documentation on Ubuntu, macOS, and Windows.
I managed the macOS and Ubuntu cases rather easily but am struggling with the Windows case not knowing what's installed and what's in system PATH for the given image & container I've selected.
Not being super familiar with the Azure-Platform I'm basically relying on commit-push-run-pipeline every single little change to my YAML file thus wasting time and resources.
I can't imagine that the only way is to blindly try out commands by commit, push and run the pipeline.
I managed to find a basic description of the currently (hopefully) available images here following the included software link for Windows link yoou end up on a comprehensive list of what's supposedly installed (I have some doubts on whether this documentation actually matches the content of the image). Calling some of those tools like cmake and choco, present in the above list, failed. Whether or not they're actually installed and in system PATH I have no idea.
Q1: Is there any way to locally test out an Azure-Pipeline YAML?
Q2: Is there any way to figure what is actually installed on a given image/container (without issuing a DIR /s from the root folder??)
Q3: Is it possible to connect to a running container (or is it a VM???) instance and directly tinker with it?
Q4: Alternatively, is it possible to run such an image locally (Docker)? Does it imply execution on a Windows machine or is that a standalone VM image?
EDIT: Found out about this question, although doesn't quite answer mine: Is there a tool to validate an Azure DevOps Pipeline locally?
Q1: Is there any way to locally test out an Azure-Pipeline YAML?
The answer is yes. You could create your private agent to execute the Azure-Pipeline YAML.
Self-hosted agents
Q2: Is there any way to figure what is actually installed on a given
image/container (without issuing a DIR /s from the root folder??)
Just as you know, we could check the document Software for the software installed on the agent. If you want to know the install the path of some software, you could check the debug log from the build task. For example, cmake. We could check the build log from the cmake task:
Q3: Is it possible to connect to a running container (or is it a
VM???) instance and directly tinker with it?
For the hosted agent, I am afraid the answer is not.
Q4: Alternatively, is it possible to run such an image locally
(Docker)? Does it imply execution on a Windows machine or is that a
standalone VM image?
The answer is yes, we could Run a self-hosted agent in Docker. And it imply execution on a Windows machine.
I want to use Manjaro in GCP compute engine with GUI but there isn't any image available in compute engine
You need to install Manjaro in your own machine first using something like VirtualBox. Then you can upload the local boot disk to GCS and use that.
See details here: https://cloud.google.com/compute/docs/images/importing-virtual-disks
Seems Manjaro is based on Arch but GCP only supports CentOS/Debian/Red Hat/Ubuntu
(https://cloud.google.com/compute/docs/images/importing-virtual-disks#supported_operating_systems)
Don't if it will work.
For GUI, you need to install X and VNC server then use vnc client to connect.
Besides what Cloud Ace already mentions, there's another resource I believe you might find of interest.
There is actually an ArchLinux dedicated repo in the official GCP GitHub page, which contains some instructions as per how to install ArchLinux into a GCE instance by either using a preconfigured image in the available public images in GCP or building yourself your own custom image. I believe the process of building an image could be attempted with Manjaro, given the fact that it is originally intended for ArchLinux. It has potential to work.
In the end, if the custom image building does not work out with Manjaro, you can always use the Arch Linux public image mentioned in the GitHub I shared (which is the minimal base Arch Linux image), and install the desktop environment you like.
Hope this helps.
I am using the compute engine of the google cloud platform to do computations.
I am using Ubuntu as the OS and every time I create a new instance, I have to install the software I need from scratch, including the build-essential.
I am pretty sure there is a way to specify the software I would like to have in my VM but couldn´t figure out a straightforward way to do it.
You should use GCE custom images to create VM images with pre-installed software that you need.
Alternatively, you can consider using startup scripts in which you can install software during VM startup. But in contrast to custom images it will increase VM startup time, because startup script will be running during VM startup.
I have finally arrived in the cloud to put my NLP work to the next level, but I am a bit overwhelmed with all the possibilities I have. So I am coming to you for advice.
Currently I see three possibilities:
SageMaker
Jupyter Notebooks are great
It's quick and simple
saves a lot of time spent on managing everything, you can very easily get the model into production
costs more
no version control
Cloud9
EC2(-AMI)
Well, that's where I am for now. I really like SageMaker, although I don't like the lack of version control (at least I haven't found anything for now).
Cloud9 seems just to be an IDE to an EC2 instance.. I haven't found any comparisons of Cloud9 vs SageMaker for Machine Learning. Maybe because Cloud9 is not advertised as an ML solution. But it seems to be an option.
What is your take on that question? What have I missed? What would you advise me to go for? What is your workflow and why?
I am looking for an easy work environment where I can quickly test my models, exactly. And it won't be only me working on it, it's a team effort.
Since you are working as a team I would recommend to use sagemaker with custom docker images. That way you have complete freedom over your algorithm. The docker images are stored in ecr. Here you can upload many versions of the same image and tag them to keep control of the different versions(which you build from a git repo).
Sagemaker also gives the execution role to inside the docker image. So you still have full access to other aws resources (if the execution role has the right permissions)
https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/scikit_bring_your_own/scikit_bring_your_own.ipynb
In my opinion this is a good example to start because it shows how sagemaker is interacting with your image.
Some notes on other solutions:
The problem of every other solution you posted is you want to build and execute on the same machine. Sure you can do this but keep in mind, that gpu instances are expensive and therefore you might only switch to the cloud when the code is ready to run.
Some other notes
Jupyter Notebooks in general are not made for collaborative programming. I think they want to change this with jupyter lab but this is still in development and sagemaker only use the notebook at the moment.
EC2 is cheaper as sagemaker but you have to do more work. Especially if you want to run your model as docker images. Also with sagemaker you can easily build an endpoint for model inference which would be even more complex to realize with ec2.
Cloud 9 I never used this service and but on first glance it seems good to develop on, but the question remains if you want to do this on a gpu machine. Because you're using ec2 as instance you have the same advantage/disadvantage.
One thing I'd like to call out first is SageMaker notebook is not the only IDE environment in which you can interact with other components of SageMaker such as training and hosting. In fact you can make API calls to SageMaker training/hosting through Cloud9 or any IDEs you've installed on EC2 or even your laptop, as long as you have AWS SDK or SageMaker Python SDK installed.
Regarding the choice of the IDE, it's really up to your particular needs. SageMaker notebook is Jupyter based (now also supports JupyterLab beta), ML focused, and fully managed. Hundreds of Python packages that are commonly used in ML, as well as Tensorflow, Keras, MxNet, SageMaker Python SDK, etc., are preinstalled and automatically maintained for you. It also integrates more closely with other components of SageMaker as one can imagine.
Cloud9 is a managed IDE too but it is for general purpose rather than ML specific. If you want to use Jupyter on cloud9 it requires extra work from your side. It does not preinstall and maintain the version of common ML/DL related packages like SageMaker notebook does.
I'm working on a project where I think of distributing tasks to VMs that are dynamically managed (created, destroyed, paused, run processes from host, etc). I was wondering what would be a good approach or API to accomplish the management of the VMs. Below are some examples of what I'm thinking but I wanted to get some guidance on the best approach.
Vagrant-binding: it looked perfect but it out of date and not supported.
Oracle Tools: the vagrant module looks interesting but there isn't much documentation and I'm a bit confused on how to actually use it.
VirtualBox sdk: I'm a bit confused on the setup of this.
As you can see from the examples I was thinking in Java but I'm open to working on other languages. This project academic in nature and I'm a student so I know that this might not be the most practical thing to do but I wanted to see if it possible and what would be best way to accomplish it.
I have recently created a program to manage the VMs at my work. I used object-pascal (Delphi) to create the GUI and then did all the heavy lifting using vboxmanage commands that are ran as a process through cmd but called from my program. Oracle has a nice list of available commands here
For example
List running Vms
VBoxManage.exe list runningvms
Import a VM
VBoxManage.exe import (VM_To_Import_Location) --vsys 0 --vmname (Name_of_VM) --unit 11 --disk (Where_You_Want_VM_Stored)
Start a VM
VBoxManage.exe startvm (Name_Of_VM) --type headless
Take a Snapshot
VBoxManage.exe snapshot (VM_Name) take (Snapshot_Name) -desc "My Snapshot Description"
Restore a Snapshot
Vboxmanage.exe snapshot (VM_Name) restore (Snapshot_Name)
There are many more for pretty much everything you would want to do with a VM.