API Gateway cares about my Authorization header when it shouldn't - amazon-web-services

Question

I created a private REST API in API Gateway (with Lambda proxy integration), which needs to be accessible from a VPC. I've setup a VPC Endpoint for API Gateway in the VPC. The API is accessible from within the VPC, as expected.
The VPC endpoint (and indeed the entire VPC environment) is created via CloudFormation.
The API needs to consume an Authorization header, which is not something I can change. The content of that header is something specific to our company, it's not something standard. The problem is that when I add an Authorization header to the request, API Gateway rejects it with the following error (from API Gateway logs in CloudWatch):
IncompleteSignatureException
Authorization header requires 'Credential' parameter.
Authorization header requires 'Signature' parameter.
Authorization header requires 'SignedHeaders' parameter.
Authorization header requires existence of either a 'X-Amz-Date' or a 'Date' header.
Authorization=[the header content here]
If I remove the Authorization header, the request is accepted and I get the expected response from my lambda. The method I'm calling has Auth set to NONE.
The strange thing is that if I delete the VPC endpoint and create it manually via the console, it works correctly - the Authorization header is passed through to my lambda, instead of API Gateway inspecting and rejecting it.
I've torn the endpoint down and recreated it multiple times manually and with CloudFormation and the results are consistent. But I've compared them to each other and they look exactly the same: same settings, same subnets, same security groups, same policy. Since I can see no difference between them, I'm at a bit of a loss as to why it doesn't work with the CloudFormation version.
The only difference I've been able to find is in the aws headers for each version (with Authorization header removed, otherwise it doesn't get as far as logging the headers with the CF endpoint). With the CF endpoint, the headers include x-amzn-vpce-config=0 and x-amzn-vpce-policy-url=MQ==. With the manual endpoint I get x-amzn-vpce-config=1, and the policy-url header isn't included.
I've also tried changing the API to both set and remove the VPC endpoint (it can be set on the API in the Settings section), and redeployed it, but in either case it has no effect - requests continue to work/get rejected as before.
Does anyone have any ideas? I've posted this on the AWS forum as well, but just in case anyone here has come across this before...
If it's of any interest, the endpoint is created like so ([] = redacted):
ApiGatewayVPCEndpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
PrivateDnsEnabled: true
PolicyDocument:
Statement:
- Action: '*'
Effect: Allow
Resource: '*'
Principal: '*'
ServiceName: !Sub com.amazonaws.${AWS::Region}.execute-api
SecurityGroupIds:
- !Ref [my sec group]
SubnetIds:
- !Ref [subnet a]
- !Ref [subnet b]
- !Ref [subnet c]
VpcEndpointType: Interface
VpcId: !Ref [my vpc]

Answer 1

I've managed to get it working, and it's the most ridiculous thing.
This is the endpoint policy in CF (including property name to show it in context):
PolicyDocument:
Statement:
- Action: '*'
Effect: Allow
Resource: '*'
Principal: '*'
This is how that policy appears in the console:
{
"Statement": [
{
"Action": "*",
"Effect": "Allow",
"Resource": "*",
"Principal": "*"
}
]
}
This is how the policy appears in describe-vpc-endpoints:
"PolicyDocument": "{\"Statement\":[{\"Action\":\"*\",\"Resource\":\"*\",\"Effect\":\"Allow\",\"Principal\":\"*\"}]}"
Now let's look at the policy of a manually created endpoint.
Console:
{
"Statement": [
{
"Action": "*",
"Effect": "Allow",
"Resource": "*",
"Principal": "*"
}
]
}
describe-vpc-endpoints:
"PolicyDocument": "{\n \"Statement\": [\n {\n \"Action\": \"*\", \n \"Effect\": \"Allow\", \n \"Principal\": \"*\", \n \"Resource\": \"*\"\n }\n ]\n}"
The console shows them exactly the same, and the JSON itself returned in describe-vpc-endpoints is the same except for some "prettifying" newlines and whitespace, surely that could have no effect whatsoever? Wrong! It's those newlines that make the policy actually work!
Anyway, the solution is to supply the policy as JSON, for example:
ApiGatewayVPCEndpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
PrivateDnsEnabled: true
PolicyDocument: '
{
"Statement": [
{
"Action": "*",
"Effect": "Allow",
"Resource": "*",
"Principal": "*"
}
]
}'
ServiceName: !Sub com.amazonaws.${AWS::Region}.execute-api
SecurityGroupIds:
- !Ref [my sec group]
SubnetIds:
- !Ref [subnet a]
- !Ref [subnet b]
- !Ref [subnet c]
VpcEndpointType: Interface
VpcId: !Ref [my vpc]
You can even put all the JSON on a single line, it will get the newline characters put in there by AWS at some point. It's just YAML that gets transformed to JSON without newlines and causes this issue.
With the CF resource like that, API Gateway accepts my Authorization header and passes it through to the Lambda without any issues.