AWS Glue Timeout: Creating External Schema In Redshift - amazon-web-services

Question

I am trying to create an external schema, and my command is as follows. As of course, I have changed the names of the components/items to non-meaningful names just to hide my production values:
create external schema sb_external
from data catalog
database 'dev'
iam_role 'arn:aws:iam::490412345678:role/aws-service-role/redshift.amazonaws.com/AWSServiceRoleForRedshift'
create external database if not exists;
The query is ran in the Redshift database using "psql" CLI from within an EC2 instance. It is a private subnet, while the EC2 instance and the Redshift Database are in 2 different VPCs joined by VPC Peering. On the VPC where we have the EC2 instance, we have a Glue Endpoint.
While I run the above query from the same VPC where I have the Redshift database, I still get an error as follows, even if in the same VPC I have created an Endpoint Interface for Glue.
Failed to perform AWS request, curlError=Failed to connect to glue.eu-west-1.amazonaws.com port 443: Connection timed out
With or Without the VPC Endpoint, we have the same error.
Any help in this regard would be highly appreciated.

Answer 1

I have also faced the same issue and somehow I managed to resolve it.
This error caused when you enable Enhanced VPC routing in your cluster.
By default Glue endpoint uses default security group.
As error starting "glue.eu-west-1.amazonaws.com", you need to enable DNS hostnames and DNS resolution for your VPC.
Also add inbound rule for port number 443 which is for https in default security group with source as Redshift's security group.
listing few links which helped me:
[+]. https://docs.aws.amazon.com/glue/latest/dg/vpc-interface-endpoints.html
[+]. https://docs.aws.amazon.com/vpc/latest/privatelink/create-interface-endpoint.html#vpce-interface-limitations
[+]. https://docs.aws.amazon.com/redshift/latest/mgmt/spectrum-enhanced-vpc.html#spectrum-enhanced-vpc-considerations
"Access to AWS Glue or Amazon Athena
Redshift Spectrum accesses your data catalog in AWS Glue or Athena. Another option is to use a dedicated Hive metastore for your data catalog.
To enable access to AWS Glue or Athena, configure your VPC with an internet gateway or NAT gateway. Configure your VPC security groups to allow outbound traffic to the public endpoints for AWS Glue and Athena. Alternatively, you can configure an interface VPC endpoint for AWS Glue to access your AWS Glue Data Catalog. When you use a VPC interface endpoint, communication between your VPC and AWS Glue is conducted within the AWS network."

Answer 2

A reason for this error message is having enabled Enhanced VPC on your Redshift Cluster.
As per documentation https://aws.amazon.com/premiumsupport/knowledge-center/redshift-enhanced-vpc-routing/ enabling Enhanced VPC might impact Unload / Copy commands. Here you are trying to create an external schema and one of the potential reason for this error is having enabled this configuration.

Answer 3

If you are using Enhanced VPC:
Create VPC Endpoints Interface for: S3, Glue and, if using: LakeFormation, Athena.
Create a VPC Endpoint Gateway for S3
Ensure all endpoints have ingress 443 from the security group of Redshift
Ensure Redshift has egress to the endpoints
Check the routing table has routing to S3 Gateway prefix (not just IP to the S3 interface)
Check DNS in the VPC

Answer 4

The security group associated with the Redshift cluster needs to have egress configured for enabling outbound traffic.
Example egress configuration:
from port: 0
to port: 0
protocol: -1 (all protocols)
CIDR IP: "0.0.0.0/0"
References
AWS::EC2::SecurityGroupEgress