I'm building a data ingestion layer for my company where I have a lot of different integration points (rest apis).
Some of the API's require you to connect from a whitelisted IP.
I'd really like to use google cloud functions / pubsub to build the ingestion logic because of it's scalability and reduced cost.
But the problem is that google cloud functions always connect from random ips and there is nothing we can do about that, as is answered in this question: Possible to get static IP address for Google Cloud Functions?
So my question is: Is there a way to proxy / nat cloud functions so that they come from a set of static ips?
This is now possible via configuring network settings for Cloud Functions particularly Egress Settings.
Taken from the Official Docs:
Via Console:
Open the Functions Overview page in the Cloud Console
Click Create function. Alternatively, click an existing function to go to its details page, and click Edit
Expand the advanced settings by clicking Environment variables, networking, timeouts and more.
In the Networking section, under Egress settings, select a Serverless VPC Access connector.
Select the appropriate egress setting based on how you want to route outbound traffic through the connector.
Via gcloud:
gcloud functions deploy FUNCTION_NAME \
--vpc-connector CONNECTOR_NAME \
--egress-settings EGRESS_SETTINGS \
FLAGS...
where:
FUNCTION_NAME is the name of your function.
CONNECTOR_NAME is the name of the Serverless VPC Access connector to use. See the gcloud documentation for more information.
Note: You can omit the --vpc-connector flag if you are updating egress
settings on an existing function that already has a connector.
EGRESS_SETTINGS is one of the supported values for egress settings: see gcloud documentation.
FLAGS... refers to other flags you pass to the deploy command.
Select the appropriate egress setting based on how you want to route outbound traffic through the connector.
After this, you only need to
Set up Cloud NAT and
Specify a static IP address for NAT.
Create a Cloud NAT:
gcloud compute routers nats create nat-config \
--router=nat-router \
--auto-allocate-nat-external-ips \
--nat-all-subnet-ip-ranges \
--enable-logging
Specify IP addresses:
gcloud compute routers nats create nat-config \
--router=nat-router \
--nat-external-ip-pool=ip-address1,ip-address2
As mentioned by #Murtaza Kanchwala it's not possible to Proxy / NAT Cloud Functions so that they would come from a set of static IPs. However as this would be a good feature, I opened a feature request for this to be implemented. For all further updates refer to the request itself, since all the updates will be posted there.
Related
I am trying to run simple serverless spark(dataproc batch) job which reads object from on-prem ECS with shared VPC. I have open egress firewall in shared vpc to connect on-prem storage but I don't see that firewall rule is getting hit
There are very less resources available at the moment since this is a new product GA recently.
Failed to initialize node
gdpic-srvls-batch-fxxxx7-cxx6-4xxd-bxx6-6xxxxxx4-m: Timed out
waiting for at least 1 worker(s) registered. This is often caused by
firewall rules that prevent Spark workers from communicating with the
master. Please review your network firewall rules and be sure they
allow communication on all ports between all nodes. See
https://cloud.google.com/dataproc-serverless/docs/concepts/network for
instructions. See output in: gs://gcs-object-xxxx
I tried looking into url provided but couldn't find much details.
If I have to setup NAT gateway with shared VPC project with my project how can I do?
Has anyone solved this problem already?
Usually it means that there is no connectivity between internal IP addresses in your VPC. Adding firewall rule as shown in the linked doc, should help:
gcloud compute firewall-rules create allow-internal-ingress \
--network="network-name" \
--source-ranges="subnetwork internal-IP ranges" \
--direction="ingress" \
--action="allow" \
--rules="all"
In CloudRun:
public-service needs to talk to internal-service and the internet
internal-service talks to the internet
Option1: Ideally, internal-service has ingress as internal, but in doing so, public-service requires a vpc-connector for all-traffic, which means it also needs a NAT gateway added.
Option-2: Alternatively, internal-service can have ingress as all and --no-allow-unauthenticated.
Option-1 looks a bit complex. What is recommended? What are the security risks to Option-2?
Your VPC contain a default route that forward the traffic to the internet if no IP match in your VPC
Therefore, you don't need a Cloud NAT. Cloud NAT is useful if you want to go to the internet with a static and your own IP, not with a shared and random IP.
Option 1 is the best, without cloud nat overhead.
EDIT 1
I was sure that the default internet route wasn't delete-able. Thanks to your comment, I checked and.... no, you can delete it. Only the priority 0 rules aren't removable.
But that also means you can recreate it, like that
gcloud beta compute routes create default-to-internet \
--network=default --priority=1000 --destination-range=0.0.0.0/0 \
--next-hop-gateway=default-internet-gateway
Stay on the option 1 ;)
I would like to develop a Google Cloud Function that will subscribe to file changes in a Google Cloud Storage bucket and upload the file to a third party FTP site. This FTP site requires allow-listed IP addresses of clients.
As such, it is possible to get a static IP address for Google Cloud Functions containers?
Update: This feature is now available in GCP https://cloud.google.com/functions/docs/networking/network-settings#associate-static-ip
First of all this is not an unreasonable request, don't get gaslighted. AWS Lambdas already support this feature and have for awhile now. If you're interested in this feature please star this feature request: https://issuetracker.google.com/issues/112629904
Secondly, we arrived at a work-around which I also posted to that issue as well, maybe this will work for you too:
Setup a VPC Connector
Create a Cloud NAT on the VPC
Create a Proxy host which does not have a public IP, so the egress traffic is routed through Cloud NAT
Configure a Cloud Function which uses the VPC Connector, and which is configured to use the Proxy server for all outbound traffic
A caveat to this approach:
We wanted to put the proxy in a Managed Instance Group and behind a GCP Internal LB so that it would dynamically scale, but GCP Support has confirmed this is not possible because the GCP ILB basically allow-lists the subnet, and the Cloud Function CIDR is outside that subnet
I hope this is helpful.
Update: Just the other day, they announced an early-access beta for this exact feature!!
"Cloud Functions PM here. We actually have an early-access preview of this feature if you'd like to test it out.
Please complete this form so we can add you..."
The form can be found in the Issue linked above.
See answer below -- it took a number of years, but this is now supported.
https://cloud.google.com/functions/docs/networking/network-settings#associate-static-ip
For those wanting to associate cloud functions to a static IP address in order to whitelist the IP for an API or something of the sort I recommend checking out this step by step guide which helped me a lot:
https://dev.to/alvardev/gcp-cloud-functions-with-a-static-ip-3fe9 .
I also want to specify that this solution works for Google Cloud Functions and Firebase Functions (as it is based on GCP).
This functionality is now natively part of Google Cloud Functions (see here)
It's a two-step process according to the GCF docs:
Associating function egress with a static IP address In some cases,
you might want traffic originating from your function to be associated
with a static IP address. For example, this is useful if you are
calling an external service that only allows requests from whitelisted
IP addresses.
Route your function's egress through your VPC network. See the
previous section, Routing function egress through your VPC network.
Set up Cloud NAT and specify a static IP address. Follow the guides at
Specify subnet ranges for NAT and Specify IP addresses for NAT to set
up Cloud NAT for the subnet associated with your function's Serverless
VPC Access connector.
Refer to link below:
https://cloud.google.com/functions/docs/networking/network-settings#associate-static-ip
As per Google, the feature has been released check out the whole thread
https://issuetracker.google.com/issues/112629904
It's not possible to assign a static IP for Google Cloud Functions, as it's pretty much orthogonal to the nature of the architecture being 'serverless' i.e. allocate and deallocate servers on demand.
You can, however, leverage a HTTP proxy to achieve a similar effect. Setup a Google Compute Engine instance, assign it a static IP and install a proxy library such as https://www.npmjs.com/package/http-proxy. You can then route all your external API calls etc through this proxy.
However, this probably reduces scale and flexibility, but it might be a workaround.
I would like to develop a Google Cloud Function that will subscribe to file changes in a Google Cloud Storage bucket and upload the file to a third party FTP site. This FTP site requires allow-listed IP addresses of clients.
As such, it is possible to get a static IP address for Google Cloud Functions containers?
Update: This feature is now available in GCP https://cloud.google.com/functions/docs/networking/network-settings#associate-static-ip
First of all this is not an unreasonable request, don't get gaslighted. AWS Lambdas already support this feature and have for awhile now. If you're interested in this feature please star this feature request: https://issuetracker.google.com/issues/112629904
Secondly, we arrived at a work-around which I also posted to that issue as well, maybe this will work for you too:
Setup a VPC Connector
Create a Cloud NAT on the VPC
Create a Proxy host which does not have a public IP, so the egress traffic is routed through Cloud NAT
Configure a Cloud Function which uses the VPC Connector, and which is configured to use the Proxy server for all outbound traffic
A caveat to this approach:
We wanted to put the proxy in a Managed Instance Group and behind a GCP Internal LB so that it would dynamically scale, but GCP Support has confirmed this is not possible because the GCP ILB basically allow-lists the subnet, and the Cloud Function CIDR is outside that subnet
I hope this is helpful.
Update: Just the other day, they announced an early-access beta for this exact feature!!
"Cloud Functions PM here. We actually have an early-access preview of this feature if you'd like to test it out.
Please complete this form so we can add you..."
The form can be found in the Issue linked above.
See answer below -- it took a number of years, but this is now supported.
https://cloud.google.com/functions/docs/networking/network-settings#associate-static-ip
For those wanting to associate cloud functions to a static IP address in order to whitelist the IP for an API or something of the sort I recommend checking out this step by step guide which helped me a lot:
https://dev.to/alvardev/gcp-cloud-functions-with-a-static-ip-3fe9 .
I also want to specify that this solution works for Google Cloud Functions and Firebase Functions (as it is based on GCP).
This functionality is now natively part of Google Cloud Functions (see here)
It's a two-step process according to the GCF docs:
Associating function egress with a static IP address In some cases,
you might want traffic originating from your function to be associated
with a static IP address. For example, this is useful if you are
calling an external service that only allows requests from whitelisted
IP addresses.
Route your function's egress through your VPC network. See the
previous section, Routing function egress through your VPC network.
Set up Cloud NAT and specify a static IP address. Follow the guides at
Specify subnet ranges for NAT and Specify IP addresses for NAT to set
up Cloud NAT for the subnet associated with your function's Serverless
VPC Access connector.
Refer to link below:
https://cloud.google.com/functions/docs/networking/network-settings#associate-static-ip
As per Google, the feature has been released check out the whole thread
https://issuetracker.google.com/issues/112629904
It's not possible to assign a static IP for Google Cloud Functions, as it's pretty much orthogonal to the nature of the architecture being 'serverless' i.e. allocate and deallocate servers on demand.
You can, however, leverage a HTTP proxy to achieve a similar effect. Setup a Google Compute Engine instance, assign it a static IP and install a proxy library such as https://www.npmjs.com/package/http-proxy. You can then route all your external API calls etc through this proxy.
However, this probably reduces scale and flexibility, but it might be a workaround.
I would like to develop a Google Cloud Function that will subscribe to file changes in a Google Cloud Storage bucket and upload the file to a third party FTP site. This FTP site requires allow-listed IP addresses of clients.
As such, it is possible to get a static IP address for Google Cloud Functions containers?
Update: This feature is now available in GCP https://cloud.google.com/functions/docs/networking/network-settings#associate-static-ip
First of all this is not an unreasonable request, don't get gaslighted. AWS Lambdas already support this feature and have for awhile now. If you're interested in this feature please star this feature request: https://issuetracker.google.com/issues/112629904
Secondly, we arrived at a work-around which I also posted to that issue as well, maybe this will work for you too:
Setup a VPC Connector
Create a Cloud NAT on the VPC
Create a Proxy host which does not have a public IP, so the egress traffic is routed through Cloud NAT
Configure a Cloud Function which uses the VPC Connector, and which is configured to use the Proxy server for all outbound traffic
A caveat to this approach:
We wanted to put the proxy in a Managed Instance Group and behind a GCP Internal LB so that it would dynamically scale, but GCP Support has confirmed this is not possible because the GCP ILB basically allow-lists the subnet, and the Cloud Function CIDR is outside that subnet
I hope this is helpful.
Update: Just the other day, they announced an early-access beta for this exact feature!!
"Cloud Functions PM here. We actually have an early-access preview of this feature if you'd like to test it out.
Please complete this form so we can add you..."
The form can be found in the Issue linked above.
See answer below -- it took a number of years, but this is now supported.
https://cloud.google.com/functions/docs/networking/network-settings#associate-static-ip
For those wanting to associate cloud functions to a static IP address in order to whitelist the IP for an API or something of the sort I recommend checking out this step by step guide which helped me a lot:
https://dev.to/alvardev/gcp-cloud-functions-with-a-static-ip-3fe9 .
I also want to specify that this solution works for Google Cloud Functions and Firebase Functions (as it is based on GCP).
This functionality is now natively part of Google Cloud Functions (see here)
It's a two-step process according to the GCF docs:
Associating function egress with a static IP address In some cases,
you might want traffic originating from your function to be associated
with a static IP address. For example, this is useful if you are
calling an external service that only allows requests from whitelisted
IP addresses.
Route your function's egress through your VPC network. See the
previous section, Routing function egress through your VPC network.
Set up Cloud NAT and specify a static IP address. Follow the guides at
Specify subnet ranges for NAT and Specify IP addresses for NAT to set
up Cloud NAT for the subnet associated with your function's Serverless
VPC Access connector.
Refer to link below:
https://cloud.google.com/functions/docs/networking/network-settings#associate-static-ip
As per Google, the feature has been released check out the whole thread
https://issuetracker.google.com/issues/112629904
It's not possible to assign a static IP for Google Cloud Functions, as it's pretty much orthogonal to the nature of the architecture being 'serverless' i.e. allocate and deallocate servers on demand.
You can, however, leverage a HTTP proxy to achieve a similar effect. Setup a Google Compute Engine instance, assign it a static IP and install a proxy library such as https://www.npmjs.com/package/http-proxy. You can then route all your external API calls etc through this proxy.
However, this probably reduces scale and flexibility, but it might be a workaround.