I have been reading for few weeks for different approaches for ML in production. I decided to test Kubeflow and I decided to test it on GCP. I started to deploy Kubeflow on GCP using the guiidline on official kubeflow website(here https://www.kubeflow.org/docs/gke/). I run into a lot of issues and it was quit hard to fix them. I started to look into a better approach and I noticed that GCP AI platform now offers deploying Kubeflow pipelines with just few simple steps. (https://cloud.google.com/ai-platform/pipelines/docs/connecting-with-sdk.)
After easily setting up this, I had few question and doubts. If it is this much easy to set up and deploy Kubeflow why we have to go through such a cumbersome way as suggested in the kubeflow official website. Since creating Kubeflow pipeline on GCP means basically I am deploying Kubeflow on GCP, does that mean I can access other Kubeflow services like Katib?
Elnaz
The kubeflow official website provides the required information in detailed way and where as in google cloud it directly provides you the services with possible ready solution.
Referring to will fuks document it says YES, you can able to access katlib on GCP
The GCP managed service of Kubeflow Pipelines is just that. You won't have a lot of access to the cluster to make changes. I've deployed a Kubeflow cluster that can still reach the AI Hub as well.
I believe they have plans to expand what can be deployed in the AI Platform but if you don't want to wait, the self-deployment is possible (but not easy) IMO.
Related
Recently I started working with GCP workflows, and functions. We are using serverless framework for the functions and we can run them in our on computers with the command serverless invoke local --function <function_name> so we don't have to spend cloud executions.
What I'm looking now is if there is a way to do the same thing with GCP workflows, to run them in our own computers instead of invoking them inside the cloud.
I already read the resources from google and from many different articles but I still not find the trick (if it actually exists)
Today, there is no emulator for Cloud Workflows. But if you can afford to deploy your cloud functions on GCP, Cloud workflows has a generous free tier: 5000 steps for free
I am trying to build an app where the user is able to upload a file to cloud storage. This would then trigger a model training process (and predicting later on). Initially I though I could do this with cloud functions/pubsub and cloudml, but it seems that cloud functions are not able to trigger gsutil commands which is needed for cloudml.
Is my only option to enable cloud-composer and attach GPUs to a kubernetes node and create a cloud function that triggers a dag to boot up a pod on the node with GPUs and mounting the bucket with the data? Seems a bit excessive but I can't think of another way currently.
You're correct. As for now, there's no possibility to execute gsutil command from a Google Cloud Function:
Cloud Functions can be written in Node.js, Python, Go, and Java, and are executed in language-specific runtimes.
I really like your second approach with triggering the DAG.
Another idea that comes to my mind is to interact with GCP Virtual Machines within Cloud Composer through the Python operator by using the Compute Engine Pyhton API. You can find more information in automating infrastructure and taking a deep technical dive into the core features of Cloud Composer here.
Another solution that you can think of is Kubeflow, which aims to make running ML workloads on Kubernetes. Kubeflow adds some resources to your cluster to assist with a variety of tasks, including training and serving models and running Jupyter Notebooks. Please, have a look on Codelabs tutorial.
I hope you find the above pieces of information useful.
I am learning GCP, and came across Kuberflow and Google Cloud Composer.
From what I have understood, it seems that both are used to orchestrate workflows, empowering the user to schedule and monitor pipelines in the GCP.
The only difference that I could figure out is that Kuberflow deploys and monitors Machine Learning models. Am I correct? In that case, since Machine Learning models are also objects, can't we orchestrate them using Cloud Composer? How does Kubeflow help in any way, better than Cloud Composer when it comes to managing Machine Learning models??
Thanks
Kubeflow and Kubeflow Pipelines
Kubeflow is not exactly the same as Kubeflow Pipelines. The Kubeflow project mostly develops Kubernetes operators for distributed ML training (TFJob, PyTorchJob). On the other hand the Pipelines project develops a system for authoring and running pipelines on Kubernetes. KFP also has some sample components, by the main product is the pipeline authoring SDK and the pipeline execution engine
Kubeflow Pipelines vs. Cloud Composer
The projects are pretty similar, but there are differences:
KFP use Argo for execution and orchestration. Cloud Composer uses Apache Airflow.
KFP/Argo is designed for distributed execution on Kubernetes. Cloud Composer/Apache Airflow are more for single-machine execution.
KFP/Argo are language-agnostic - components can use any language (components describe containerized command-line programs). Cloud Composer/Apache Airflow use Python (Airflow operators are defined as Python classes).
KFP/Argo have concept of data passing. Every component has inputs and outputs and pipleine connects them into a data passing graph. Cloud Composer/Apache Airflow do not really have data passing (Airflow has global variable storage and XCom, but it's not the same thing as explicit data passing) and the pipeline is a task dependency graph rather than mostly data dependency graph (KFP can also have task dependencies, but usually they're not needed).
KFP supports execution caching feature that skips execution of tasks that have already been executed before.
KFP records all artifacts produced by pipeline runs in ML Metadata database.
KFP has experimental adapter which allows using Airflow operators as components.
KFP has large fast-growing ecosystem of custom components.
Kubeflow is a platform for developing and deploying a machine learning (ML) systems. Its components are focused on creating workflows aimed to build ML systems.
Cloud Composer provides the infraestructure to run Apache Airflow worflows. Its components are known as Airflow Operators and the workflows are connections between these operators that are known as DAGs.
Both services run on Kubernetes, but they are based on different programming frameworks; therefore, you are correct, Kuberflow deploys and monitors Machine Learning models. See below the answer for your questions:
In that case, since Machine Learning models are also objects, can't we orchestrate them using Cloud Composer?
You would need to find an operator that meet your needs, or create a custom operator with the structure required to create a model, see this example. Even when it can be performed, this could be more difficult that using Kubeflow.
How does Kubeflow help in any way, better than Cloud Composer when it comes to managing Machine Learning models??
Kubeflow hides complexity as it is focused on Machine Learninig models. The frameworks specialized on machine learning makes those things easier than using Cloud Composer which in this context can be considered as a general purpose tool (focused on linking existing services supported by the Airflow Operators).
Taking this straight from kubeflow.org
The Kubeflow project is dedicated to making deployments of machine
learning (ML) workflows on Kubernetes simple, portable and scalable.
Our goal is not to recreate other services, but to provide a
straightforward way to deploy best-of-breed open-source systems for ML
to diverse infrastructures. Anywhere you are running Kubernetes, you
should be able to run Kubeflow.
And as you can see it is a suite made of many software that are useful in the life cycle of a ML model. It comes with tensorflow, jupiter, etc.
Now the real deal, when it comes to Kubeflow is "easy deploy of a ML model at scale on a Kubernetis cluster".
However on GCP you already a ML suite in cloud, datalab, cloud build etc. So I don't know how much efficient will be sinning up a kubernetis cluster if you don't need the "portability" factor.
Cloud Composer is the real deal while taking about orchestration of a workflow. It is a "managed" version of Apache Airflow and it is ideal for any "simple" workflow that changes a lot, since you can change it via a visual UI and with python.
It is also ideal to automate infrastructure operations:
I'm looking into the best way to set up an end-to-end machine learning pipeline, and evaluating the data exploration component options.
I'm trying to figure out the difference between google cloud datalab, and google cloud ai platform notebooks. They both seem to offer similar functionality, so not sure why they both exist, or whether one is a new iteration of the other.
If they are different, what is the benefit of one over the other?
Google Cloud AI Platform Notebooks is effectively the upgraded version of Google Cloud Datalab and gives you benefits like being able to use the notebook directly in your browser without having to setup an ssh tunnel first (which datalab forces you to do).
If you're creating a new notebook, you 100% want to go with AI Platform Notebooks
(Also, Datalab is now deprecated)
Quoting the End-to-End Machine Learning with TensorFlow on GCP course on Coursera (link) -
AI Platform Notebooks is the next generation of hosted notebook on GCP, and has replaced Cloud Datalab.
I finally got my first django project and I need help in deploying it in GCP with kubernetes.
I've never deployed any project before so it's a bit frustrating for me with the client nagging on my head.
it's an E-learning platform so I want to use GCP with kubernetes (for CI/DI since there will be a lot of updates for the project) and Google cloud storage for storing media files.
i'd love to have some help concerning deployment, things to do/don't, and some useful links to start with.
ps: this is my first question so be easy on me
Your question is too wide, try starting first and then asking a question - as we won’t be able to explain an universal way of deploying Django applications in GCP. I recommend starting from getting familiar with GCP services.
There is a really cool but paid course on Coursera platform ("Getting Started with Google Kubernetes Engine") with practical hands-on labs on how to use Kubernetes on GCP in pair with CI/CD tool like Jenkins. You can also find more about Jenkins in GCP in here.
You will also find there how to:
Use different deployment strategies with Kubernetes (Rolling Updates,
Canary and Blue-Green Deployments) with simple hello-world app.
Create a continuous delivery pipeline using Jenkins
You can enroll into this course with free trial account.