The cloud workflow doesn't come with a scheduling feature. Apart from that, what are all the differences between these two services in terms of features? In which use case should we prefer the workflow over composer or vice versa?
There are some key differences to consider when choosing between the two solutions :
A Composer instance needs to be in a running state to trigger DAGs and you'll also need to size your Cloud Composer instance based on your usage, You do not need to do this in Cloud Workflows as it is a Serverless service and you pay for anytime a workflow is triggered
Another key difference is that Cloud Composer is really convenient for writing and orchestrating data pipelines because of it's internal scheduler and also because of the provided Operators, You can interact with any Data services inside of GCP.
However, Cloud Workflows interacts with Cloud Functions, wich is a task that Composer cannot do really well.
Both Composer and Workflows support orchestrating multiple services and can handle long running workflows. Despite there being some overlap in the capabilities of these products, each has differentiators that make them well suited to particular use cases.
Composer is most commonly used for orchestrating the transformation of data as part of ELT or data engineering. Workflows, in contrast, is focused on the orchestration of HTTP-based services built with Cloud Functions, Cloud Run, or external APIs.
Composer is designed for orchestrating batch workloads that can handle a delay of a few seconds between task executions. It wouldn’t be suitable if low latency was required in between tasks, whereas Workflows is designed for latency sensitive use cases.
While you don’t have to worry about maintaining Airflow deployments in Composer, you do need to specify how many workers you need for a given Composer environment. Workflows is completely serverless; there is no infrastructure to manage or scale.
For further information refer to this google blog article and this one.
I'm working on a SaaS application which at the moment is cloud only. It's a traditional Java web application which we deploy to AWS. We rely on AWS concepts like RDS, S3, ELB, Autoscaling and for infrastructure provisioning AMIs, Cloudformation, Ansible and CodeDeploy.
There is now more and more demand for on-premise deployments by potential clients.
Are there any common approaches to package b2b applications for on-premise deployments?
My first thought would be to containerize the app infrastructure (web server, database, etc) and assume a client would be able run images. What are you guys doing and how do you tackle HA and DR aspects which come with cloud infrastructure like AWS?
I'm tackling a similar problem at the moment and there really is no one-fits all answer. Designing software for cloud-nativity comes with a lot of architectural design decisions to use technologies on offer by the platform (as you have with S3, RDS, etc) which ultimately do not cross-over to majority of on-premise deployments.
Containerising your application estate is great for cross-cloud and some hybrid cloud portability but there is no guarantee that a client is using containerised work-loads on their on-premise data centre which makes the paradigm still a way off the target of supporting both seamlessly.
I find another issue is that the design principles behind cloud-hosted software are vastly different to those on-premise, with static resource requirements, often a lack of ability to scale etcetera (ironically some of the main reasons you would move a software solution to a cloud environment) so trying to design for both is a struggle and I'm guessing we will end up with a sub-optimal solution unless we decide to favour one and treat the other as a secondary concern.
I'm thinking maybe the best cross-breed solution is to concentrate on containerisation for cloud hosts taking into account the products and services on offer (and in the roadmap) - and then for making the same software available to clients who wish to use on-premise datacenters still.... perhaps they could be offered VM Images with the software solution packaged in... then make this available on a client portal for them with instructions on installation/configuration.
... I wish everyone would just use Kubernetes already! :)
I am using Google Data Flow to implement an ETL data ware house solution.
Looking into google cloud offering, it seems DataProc can also do the same thing.
It also seems DataProc is little bit cheaper than DataFlow.
Does anybody know the pros / cons of DataFlow over DataProc
Why does google offer both?
Yes, Cloud Dataflow and Cloud Dataproc can both be used to implement ETL data warehousing solutions.
An overview of why each of these products exist can be found in the Google Cloud Platform Big Data Solutions Articles
Quick takeaways:
Cloud Dataproc provides you with a Hadoop cluster, on GCP, and access to Hadoop-ecosystem tools (e.g. Apache Pig, Hive, and Spark); this has strong appeal if you are already familiar with Hadoop tools and have Hadoop jobs
Cloud Dataflow provides you with a place to run Apache Beam based jobs, on GCP, and you do not need to address common aspects of running jobs on a cluster (e.g. Balancing work, or Scaling the number of workers for a job; by default, this is automatically managed for you, and applies to both batch and streaming) -- this can be very time consuming on other systems
Apache Beam is an important consideration; Beam jobs are intended to be portable across "runners," which include Cloud Dataflow, and enable you to focus on your logical computation, rather than how a "runner" works -- In comparison, when authoring a Spark job, your code is bound to the runner, Spark, and how that runner works
Cloud Dataflow also offers the ability to create jobs based on "templates," which can help simplify common tasks where the differences are parameter values
Here are three main points to consider while trying to choose between Dataproc and Dataflow
Provisioning
Dataproc - Manual provisioning of clusters
Dataflow - Serverless. Automatic provisioning of clusters
Hadoop Dependencies
Dataproc should be used if the processing has any dependencies to tools in the Hadoop ecosystem.
Portability
Dataflow/Beam provides a clear separation between processing logic and the underlying execution engine. This helps with portability across different execution engines that support the Beam runtime, i.e. the same pipeline code can run seamlessly on either Dataflow, Spark or Flink.
This flowchart from the google website explains how to go about choosing one over the other.
https://cloud.google.com/dataflow/images/flow-vs-proc-flowchart.svg
Further details are available in the below link
https://cloud.google.com/dataproc/#fast--scalable-data-processing
Same reason as why Dataproc offers both Hadoop and Spark: sometimes one programming model is the best fit for the job, sometimes the other. Likewise, in some cases the best fit for the job is the Apache Beam programming model, offered by Dataflow.
In many cases, a big consideration is that one already has a codebase written against a particular framework, and one just wants to deploy it on the Google Cloud, so even if, say, the Beam programming model is superior to Hadoop, someone with a lot of Hadoop code might still choose Dataproc for the time being, rather than rewriting their code on Beam to run on Dataflow.
The differences between Spark and Beam programming models are quite large, and there are a lot of use cases where each one has a big advantage over the other. See https://cloud.google.com/dataflow/blog/dataflow-beam-and-spark-comparison .
Cloud Dataflow is a serverless data processing service that runs jobs written using the Apache Beam libraries. When you run a job on Cloud Dataflow, it spins up a cluster of virtual machines, distributes the tasks in your job to the VMs, and dynamically scales the cluster based on how the job is performing. It may even change the order of operations in your processing pipeline to optimize your job.
So use cases are ETL (extract, transfer, load) job between various data sources / data bases. For example load big files from Cloud Storage into BigQuery.
Streaming works based on subscription to PubSub topic, so you can listen to real time events (for example from some IoT devices) and then further process.
Interesting concrete use case of Dataflow is Dataprep. Dataprep is cloud tool on GCP used for exploring, cleaning, wrangling (large) datasets. When you define actions you want to do with your data (like formatting, joining etc), job is run under the hood on Dataflow.
Cloud Dataflow also offers the ability to create jobs based on "templates," which can help simplify common tasks where the differences are parameter values.
Dataproc is a managed Spark and Hadoop service that lets you take advantage of open source data tools for batch processing, querying, streaming, and machine learning. Dataproc automation helps you create clusters quickly, manage them easily, and save money by turning clusters off when you don't need them. With less time and money spent on administration, you can focus on your jobs and your data.
Super fast — Without using Dataproc, it can take from five to 30
minutes to create Spark and Hadoop clusters on-premises or through
IaaS providers. By comparison, Dataproc clusters are quick to start,
scale, and shutdown, with each of these operations taking 90 seconds
or less, on average. This means you can spend less time waiting for
clusters and more hands-on time working with your data.
Integrated — Dataproc has built-in integration with other Google
Cloud Platform services, such as BigQuery, Cloud Storage, Cloud
Bigtable, Cloud Logging, and Cloud Monitoring, so you have more than
just a Spark or Hadoop cluster—you have a complete data platform.
For example, you can use Dataproc to effortlessly ETL terabytes of
raw log data directly into BigQuery for business reporting.
Managed — Use Spark and Hadoop clusters without the assistance of an
administrator or special software. You can easily interact with
clusters and Spark or Hadoop jobs through the Google Cloud Console,
the Cloud SDK, or the Dataproc REST API. When you're done with a
cluster, you can simply turn it off, so you don’t spend money on an
idle cluster. You won’t need to worry about losing data, because
Dataproc is integrated with Cloud Storage, BigQuery, and Cloud
Bigtable.
Simple and familiar — You don’t need to learn new tools or APIs to
use Dataproc, making it easy to move existing projects into Dataproc
without redevelopment. Spark, Hadoop, Pig, and Hive are frequently
updated, so you can be productive faster.
If you want to migrate from your existing Hadoop/Spark cluster to the cloud, or take advantage of so many well-trained Hadoop/Spark engineers out there in the market, choose Cloud Dataproc; if you trust Google's expertise in large scale data processing and take their latest improvements for free, choose DataFlow.
Here are three main points to consider while trying to choose between Dataproc and Dataflow
Provisioning
Dataproc - Manual provisioning of clusters
Dataflow - Serverless. Automatic provisioning of clusters
Hadoop Dependencies
Dataproc should be used if the processing has any dependencies to tools in the Hadoop ecosystem.
Portability
Dataflow/Beam provides a clear separation between processing logic and the underlying execution engine. This helps with portability across different execution engines that support the Beam runtime, i.e. the same pipeline code can run seamlessly on either Dataflow, Spark or Flink.
Cloud Dataproc and Cloud Dataflow can both be used for data processing, and there’s overlap in their batch and streaming capabilities. You can decide which product is a better fit for your environment.
Cloud Dataproc is good for environments dependent on specific Apache big data components:
- Tools/packages
- Pipelines
- Skill sets of existing resources
Cloud Dataflow is typically the preferred option for green field environments:
- Less operational overhead
- Unified approach to development of batch or streaming pipelines
- Uses Apache Beam
- Supports pipeline portability across Cloud Dataflow, Apache Spark, and Apache Flink as runtimes.
See more details here https://cloud.google.com/dataproc/
Pricing comparision:
DataProc
Dataflow
If you want to calculate and compare cost of more GCP resources, please refer this url https://cloud.google.com/products/calculator/
One of the other important difference is:
Cloud Dataproc:
Data mining and analysis in datasets of known size
Cloud Dataflow:
Manage datasets of unpredictable size
see
Cloud Dataflow
Is a serverless data processing service that runs jobs written using
the Apache Beam libraries.
When you run a job on Cloud Dataflow it gets operated like this:
It spins up a cluster of virtual machines
Distributes the tasks in your job to the VMs, and dynamically scale the cluster based on how the job is performing
Dataflow may even change the order of operations in your processing pipeline to optimize your job.
It supports both batch and streaming Jobs. So use cases are ETL (extract, transfer, load) jobs between various data sources/databases.
For example, load big files from Cloud Storage into Big Query.
Streaming works based on subscription to Pub-Sub topic, so you can listen to real-time events (for example from some IoT devices) and then further process the data.
An interesting concrete use case of Dataflow is Data prep.
Data prep is a cloud tool on GCP used for exploring, cleaning, and wrangling (large) datasets. When you define the actions you want to perform on your data (like formatting, joining etc.), the job run under the hood on Dataflow.
Cloud Dataflow also offers the ability to create jobs based on "templates" which can help simplify common tasks where the differences are parameter values.
Data proc
Is a managed Spark and Hadoop service that lets you take advantage of
open-source data tools for batch processing, querying, streaming, and
machine learning.
Data proc automation helps you create clusters quickly, manage them easily, and save money by turning clusters off when you don't need them. With less time and money spent on administration, you can focus on your jobs and your data.
I'm working on a cloud based solution architecture at the moment and want to take the approach of using an immutable server infrastructure (creating and then deploying new AMIs when updating software, OS, etc). There's plenty of good reading and presentations out there in terms of the approach and the various tooling to use, and I'm aware of the virtues of stateless web/application servers and using the likes of AWS RDS to take away some of the headaches in managing the data tier.
However, I was wondering how to approach some of the management services/servers that sit around the application server estate from an immutable perspective. I can appreciate how the likes of AWS RDS take away some of the complications of maintaining the data tier from an immutable perspective, but how you would approach some of the other 'less stateless' servers/services that typically make up a solution? I'm thinking of things like the CI server e.g. Jenkins, Directory Services, Monitoring/Logging Services, etc. I suppose I could look to engage 3rd party services for some of these things, like the approach with AWS RDS, but what other approaches exit? What alternative approaches are there to maintain these servers and services without deviating widely from the immutable server deployment approach?
Thank you.
Last month there was an outage in AWS and some sites had to be taken down because of that. I was wondering if a company is availing both AWS and Google Cloud Platform for hosting, how easy would it be for them to easily transfer their services from the Amazon platform to the Google platform or vice versa ( In case Google Cloud has some outage) . First of all is it possible or not? And also if it's what would be the cost for performing such an activity and how much time will it take to get the services running back again.
In this I also did some digging up and what I came across was each of the providers (Google and Amazon) have tools of their own to do so i.e. for transferring the stored data from other platforms to their platform -
https://cloud.google.com/storage/docs/migrating?hl=en
https://aws.amazon.com/importexport/
Are these the only options available or there is anything else as well. Hope some AWS/Google cloud expert would be able to answer my question.
You would need to run your application in both environments, keep the deployments in sync, keep the databases in sync, etc. That can get complicated and expensive...
Then to automatically fail over from one environment to another you could use a DNS service such as DynDNS Active Failover that monitors the health of your application and starts sending traffic to the other environment if your primary environment becomes unhealthy.
How you manage deployments, how you continually ship data across environments, how much all that will cost, all those questions are extremely specific to the technologies (programming languages, operating systems, database servers) you are currently using. There's no way to give details on how you would accomplish those tasks without having all the details of your system.
Further, if you are using proprietary technologies on a specific platform, such as Amazon Redshift or DynamoDB, you might not find a service on the other platform that provides the same functionality.
I've seen this subject come up a lot since the last AWS outage, but I think maintaining two environments on two different platforms is overkill for all but the most extremely critical applications. Instead, I would look into maintaining a copy of your application in a different AWS region, and use Route53 health checks to fail-over.