I am new to Alibaba Cloud and I have an Apache Beam application running on Google Cloud Dataflow.
Now I want to deploy the same apache beam pipeline to Alibaba Cloud.
I am seeking help on how/what setups are needed to run an apache beam pipeline in Alibaba Cloud?
Is there a resource in Alibaba that is equivalent to Google Cloud Platform Dataflow?
You may try Alibaba DataStudio, in the Data Works section of the Alibaba Cloud Console as it is similar to Data Flow from google cloud .
You can drag and drop nodes to create a workflow, collaborate, integrate with other AC products etc.
Here is how you can create workflow: https://www.alibabacloud.com/help/doc-detail/85554.htm?spm=a2c63.l28256.a3.31.591c5b5aLHvJ68
It is not based on Apache Beam, but I believe that it will be available very soon.
Hope this will help to get idea about your needs, and what is offered right now.
Related
I have used google cloud migrate for migrating VM from AWS to GCP, Azure to GCP, and On-Prem Datacenter (VMware) to GCP. Please share views on migrating Hyperv -V based VM's to be migrated to Google cloud
As near as I can tell there is no tool for on premises Hyper-V to GCP migration. The only documentation I can find that is helpful talks about uploading the VHD to cloud storage and building an Image from that.
https://cloud.google.com/compute/docs/import/importing-virtual-disks
I am currently uploading a VHD now and will edit my answer with the result.
I am in a project and I need to connect Airflow to IA Platform (Artificial Intelligence Google Cloud). Are there any connectors I can use? And I also need to start a job from there. Thanks.
Have you considered using Airflow GCP Operator?
Airflow has extensive support for the Google Cloud Platform.
See the GCP connection type documentation to configure connections to GCP.
All hooks are based on airflow.gcp.hooks.base.GoogleCloudBaseHook.
see: https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#gcp-google-cloud-platform
Alternatively, Google has productise the Apache Airflow into Google Cloud Composer which has inbuilt integrations with Google AI services. So using Cloud Composer might prove beneficial in a long run.
I'm studying how to use GCP, especially focus on the Big Data and analytic functions, I'm not quite sure about their functionality. I did some mapping to understand these components. Could you help to check out my understanding?
Cloud Pub/Sub: Apache Kafka
Cloud Dataproc: Apache Hadoop, Spark
GCS: HDFS compatible
Cloud Dataflow: Apache Beam, Flink
Datastore: MongoDB
BigQuery: Teradata
BigTable: HBase
Memorystore: Redis
Cloud SQL: MySQL, PostgreSQL
Cloud Composer: Informatica
Cloud Data Studio: Tableau
Cloud Datalab: Jupyter notebook
I'm not totally sure what you want to know, your understanding of the GCP products is not far off, but if you are studiying GCP and want to understand them better, you can take a look at the Google Cloud developer's cheat sheet. It has a brief explanation of all the products inside GCP.
Link to the GitHub of the cheat sheet
I'm trying to create .yaml and .jinja files for a full cloud deployment, but have become stuck with deploying Cloud spanner, I am aware that it is a PaaS Application, so how would such application be implemented if it can at all?
I am relatively new to this area and am currently experimenting with cloud technology.
Cloud Spanner does not require a configuration file for deployment. You can create a new Cloud Spanner instance using the Google Cloud Console UI and point your application to it. Follow the instructions in: https://cloud.google.com/spanner/docs/create-manage-instances
I read on wikipedia that cloud foundry open source software is available to anyone whereas the Pivotal Web Services is a commercial product from Pivotal.
I kinda searched a lot on internet but did not find any cloud foundry open source software implementation example. Everything is for Pivotal product which provides a 2 months free trial service.
So can anyone tell me what is the cloud foundry open source software?
And what exactly is the difference between cloud foundry OSS & Pivotal CF?
Cloud Foundry is open source software, but if you are looking to tinker with it for the first time, using the OSS is a bit involved. You will need to have a provisioned cloud environment, you will install it yourself using MicroBosh, and everything will be done through the command line.
Pivotal Cloud Foundry is a commercial implementation that makes it easier to get up and running as you are learning the project. It provides a hosted environment in Pivotal Web Services so that you don't have to install it yourself, a web interface that makes managing the environment easier, and a number of pre-provisioned services including relational databases and messaging queues. This is the best starting point if you are just learning the technology.
To add to the above answer, Pivotal Cloud Foundry offers a public cloud offering called Pivotal Web Services where you can signup and deploy your apps on the cloud which is hosted by Pivotal.
On the other hand they also allow enterprises to host private cloud environment by installing components of the cloud infrastructure on VMWare VSphere, AWS, OpenStack Check this(http://docs.pivotal.io/pivotalcf/installing/pcf-docs.html) link out.