My Google Dataflow Job is keep getting failed while creating "Kafka-Bigquery" dataflow with AWS MSK public access bootstrap server. Is there any way to solve this issue?
I tried accessing public access bootstrap server by disabling unauthenticated access and enabling IAM based access, and also updated security groups to access from anywhere. also created a topic using EC2 instance terminal by accessing through private connection string.
In GCP, I created a dataflow using Kafka-Bigquery template and passed public-access-bootstrap-connection-string, and topic created in EC2 also passed bigquery table id.
Expectation: Job creates successfully and table is populated based on kafka message sent to msk server.
Can I connect AWS MSK (Kafka) with Google Dataflow (Kafka-Bigquery) Job (as consumer)?
Can we use source database endpoints which are hosted on other servers than AWS server for using AWS DMS? If yes, can you help me by sharing any tutorials.
I have seen this image in AWS official documentation.
Thankyou!
Yes, you can use DMS to migrate an on-prem (or diff cloud) db to AWS and vice versa.
In same documentation from where you take snapshot for e.g you have part "Migrating an On-Premises Oracle Database to Amazon Aurora MySQL".
If you check your screenshoot you can notice "The source or target database must be on an AWS service"
You have a very nice explanation in this blog article:
https://medium.com/workfall/how-to-do-database-migration-using-aws-database-migration-service-dms-from-on-premise-ec2-to-rds-d46b9144d3cc
AWS Transfer Family supports integration with AD Connector (https://docs.aws.amazon.com/directoryservice/latest/admin-guide/ad_connector_app_compatibility.html). As far as I understand, connectors are deployed in vpn-linked subnets that allows them to proxy calls to an on-premise Active Directory.
What exactly happens (what resources are created/updated under the hood) when I select AD connector as the authenticator for AWS Transfer? I'm specifically curious as to what changes are made in VPC to allow this integration.
In relation to AWS Directory Service, AWS Transfer does not seem to mutate your VPC. If you create an AD and then associate it with AWS Transfer, and take a look at your VPC, there is no new networking resources of any kind. Similar to other applications (https://docs.aws.amazon.com/directoryservice/latest/admin-guide/ms_ad_manage_apps_services.html), AWS Directory Services authorizes AWS Transfer to access your AD (in this case, connector) for Transfer logins.
Is it possible to connect to on-prem SFTP server directly in AWS glue job ?
The SFTP server has restricted access in this case (IP whitelisting)
Thanks
AWS Glue doesn't support connectivity to SFTP servers natively. You might have to use third party drivers or jars for this kind of support. I have added a blog link that might suit your needs here.
Connect to SFTP Data in AWS Glue Jobs Using JDBC
Alternative approach:
Since you mentioned "The SFTP server has restricted access" If there is no security concern to copy the data over to S3, below would be the possible solution for file based kind of source systems.
Setup a scheduler or a process to transfer these files into S3
buckets. Then AWS Glue crawler can be used to create metadata and AWS
Glue jobs for ETL.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
Starting with AWS seems to be a pain in the neck. I've already spent countless hours trying to squeeze out some information about what does what in their ocean of products and brand names. But there are no simple answers. First I have to read through countless pages congratulating me on choosing AWS and confirming how easily I'll be able to begin. Then I have to watch a dozen videos in which some deputy chief architect manager of whatever department explains how excited they are to see me. Yeah, thanks, but will you finally tell what does this crap do?! I don't have all the world's time.
Is there a list somewhere a clear and concise lists of AWS services and products without all the inspirational corporate bullshit, something like this one (entirely fictional):
Daffodil: User management service which can be embedded in your codebase.
Trainwreck: Geospatial database API.
Footsmell: Industrial automation API to control robots and drones.
Wristwatch: Thesaurus and grammar checker.
If there was a similar one for Google's services, the better.
This is a slightly old list from March 2017:
Compute
Amazon EC2: Virtual Servers in the Cloud
Amazon EC2 Container Service: Run and Manage Docker Containers
Amazon EC2 Container Registry: Store and Retrieve Docker Containers
Auto Scaling: Automatic Elasticity
AWS Elastic Beanstalk: Run and Manage Web Apps
Amazon LightSail: Launch and Manage Virtual Private Servers
AWS Lambda: Run your code in response to events
AWS Batch: Run Batch Jobs at any Scale
Storage
Amazon S3 (Simple Storage Service): Scalable Storage in the Cloud
Amazon Glacier: Low-Cost Archive Storage in the Cloud
Amazon EBS (Elastic Block Store): Block Storage for EC2
Amazon EFS (Elastic File System): Managed File Storage for EC2
AWS Storage Gateway: Hybrid Storage Integration
Database
Amazon RDS (Relational Database Service): Managed Relational Database Service
Amazon Aurora: High Performance Managed Relational Database
Amazon DynamoDB: Managed NoSQL Database
Amazon Redshift: Fast, Simple, Cost-Effective Data Warehousing
Amazon ElastiCache: In-Memory Caching System
Migration
Snowball: Petabyte-scale Data Transport
AWS Application Discovery Service: Discover On-Premises Apps
AWS Database Migration Service: Migrate Databases with Minimal Downtime
AWS Server Migration Service: Migrate On-Premises Servers to AWS
Networking & Content Delivery
Amazon Virtual Private Cloud (VPC): Isolate Cloud Resources
AWS Direct Connect: Dedicated Network Connection to AWS
Amazon Route 53: Scalable Domain Name Service
Elastic Load Balancing: High Scale Load Balancing
Amazon CloudFront: Global Content Delivery Network
Developer Tools
AWS CodeCommit: Store Code in Private Git Repositories
AWS CodeBuild: Build and Test Code
AWS CodeDeploy: Automate Code Deployment
AWS CodePipeline: Release Software using Continuous Delivery
AWS X-Ray: Analyze and Debug Your Applications
AWS Command-Line Interface: Unified Tool to Manage AWS Services
Management Tools
AWS CloudFormation: Create and Manage Resources with Templates
AWS Service Catalog: Create and Use Standardized Products
Amazon CloudWatch: Monitor Resources and Applications
AWS CloudTrail: Track User Activity and API Usage
AWS Config: Track Resource Inventory and Changes
AWS OpsWorks: Automate Operations with Chef
Amazon EC2 Systems Manager: Configure EC2 Instances and On-Premises Servers
AWS Trusted Advisor: Optimize Performance and Security
AWS Personal Health Dashboard: Personalized View of AWS service health
Security, Identity & Compliance
AWS Identity & Access Management (IAM): Manage User Access and Encryption Keys
AWS Organizations: Policy-Based Management for Multiple AWS Accounts
AWS Directory Service: Host and Manage Active Directory
AWS Cloud Directory: Create flexible cloud-native directories
AWS Key Management Service (KMS): Creation and Control of Encryption Keys
AWS CloudHSM: Hardware-based Key Storage
AWS Certificate Manager: Provision and Deploy SSL/TLS Certificates
Amazon Inspector: Analyze Application Security
AWS Shield: Managed DDoS Protection
AWS Web Application Firewall (WAF): Filter Malicious Web Traffic
Analytics
Amazon Athena: Query Data in S3 using SQL
Amazon EMR: Hosted Hadoop Framework
Amazon CloudSearch: Managed Search Service
Amazon Elasticsearch Service: Run and Scale Elasticsearch Clusters
Amazon Kinesis: Work with Real-Time Streaming Data
Amazon QuickSight: Fast Business Analytics Service
AWS Data Pipeline: Orchestration Service for periodic Data-Driven Workflows
AWS Glue: Prepare and Load Data
Artificial Intelligence
Amazon Machine Learning: Machine Learning for Developers
Amazon Polly: Turn Text into Lifelike Speech
Amazon Rekognition: Search and Analyze Images
Amazon Lex: Build Voice and Text Chatbots
Mobile Services
Amazon Cognito: User Identity and App Data Synchronization
AWS Device Farm: Test Mobile Apps on Real Devices in the Cloud
AWS Mobile Hub & Mobile SDK: Build, Test and Monitor Mobile Apps
Application Services
Amazon API Gateway: Build, Deploy and Manage APIs
AWS Step Functions: Coordinate Distributed Applications
Amazon Elastic Transcoder: Easy-to-Use Scalable Media Transcoding
Messaging
Amazon Simple Queue Service (SQS): Message Queue Service
Amazon Simple Notification Service (SNS): Push Notification Service
Amazon Simple Email Service (SES): Email Sending and Receiving Service
Amazon Pinpoint: Push Notifications for Mobile Apps
Business Productivity
Amazon Chime: Frustration-free meetings, video calls and chats
Amazon WorkDocs: Enterprise Storage and Sharing Service
Amazon WorkMail: Managed Business Email and Calendaring
Desktop & App Streaming
Amazon WorkSpaces: Desktop Computing Service
Amazon AppStream 2.0: Stream desktop applications to a browser
Internet of Things
AWS IoT Platform: Connect Devices to the Cloud
AWS Greengrass: Local Compute, Messaging, Sync for Devices
AWS IoT Button: Cloud Programmable Dash Button
Game Dev
Amazon GameLift: Dedicated Game Server Hosting
Amazon Lumberyard: Free Cross-Platform 3D game engine
There's even more these days!
It's a fair point, and with so many Amazon Web Service (AWS) services, not an easy one to sum up in a few words.
I'd say start here for a summary of the main services: https://d1.awsstatic.com/whitepapers/aws-overview.pdf
Then I think the Tech Essentials training video from acloud.guru (with 7 day free trial) is a good video to get you going: https://acloud.guru/learn/aws-technical-essentials
Google Cloud Platform is a bit more accessible IMO, their main product page gives a brief description of the products:
https://cloud.google.com/products/
Some context to the services: https://cloud.google.com/docs/overview/cloud-platform-services
And again acloud.guru have an introductory video for CDP: https://acloud.guru/learn/gcp-101
and I might as well complete the trifactor…
Microsoft Azure is a very worthy contender,
High level services: https://azure.microsoft.com/en-gb/services/
Intro Video: https://acloud.guru/learn/intro-to-azure
If you want one liners like you mentioned in your questions then click here
On that page click on the category of the service and it will list out services in that category and one line description.
E.g. Click 'Compute' to see the list of provided compute services, click 'Storage' to list if provided storage services and so on
If you want somewhat detailed explanation, click here
Here also services are grouped in categories, and you click on one of categories and you get to see the services (along with some brief explanation about each of the service) within that category
The documentation page of each product gives explanation in a simple way. Moreover, if you read the FAQ that explains things from scratch.