Running the following terraform gcp project i can see the machines do communicate with each other but no internet, machines looks to resolve the domains but not able to ping them. I am adding internal static ip since i need to be static in order for the instances to communicate with each other.
Anything i am missing?
Thank you in advance
provider "google" {
project = "terraform-368808"
region = "us-west1"
}
resource "google_compute_network" "default" {
name = "manager-network"
auto_create_subnetworks = false
mtu = 1460
}
resource "google_compute_subnetwork" "default" {
name = "manager-subnet"
ip_cidr_range = "10.10.10.0/24"
region = "us-west1"
network = google_compute_network.default.id
}
resource "google_compute_address" "manager_ip_one" {
name = "manager-ip-one"
subnetwork = google_compute_subnetwork.default.id
address_type = "INTERNAL"
address = "10.10.10.42"
region = "us-west1"
}
output "manager-ip-one" {
value = google_compute_address.manager_ip_one.address
}
resource "google_compute_address" "manager_ip_two" {
name = "manager-two"
subnetwork = google_compute_subnetwork.default.id
address_type = "INTERNAL"
address = "10.10.10.43"
region = "us-west1"
}
output "manager-ip-two" {
value = google_compute_address.manager_ip_two.address
}
resource "google_compute_instance" "manager1" {
name = "manager-node-1"
machine_type = "e2-medium"
zone = "us-west1-a"
tags = ["ssh"]
boot_disk {
initialize_params {
image = "debian-cloud/debian-10"
}
}
metadata_startup_script = "sudo apt update -y; sudo apt install wget htop -y;"
network_interface {
subnetwork = google_compute_subnetwork.default.id
network_ip = google_compute_address.manager_ip_one.address
}
provisioner "local-exec" {
command = "echo ${google_compute_address.manager_ip_one.address} >> private_ips.txt"
}
}
resource "google_compute_instance" "manager2" {
name = "manager-node-2"
machine_type = "e2-medium"
zone = "us-west1-a"
tags = ["ssh"]
boot_disk {
initialize_params {
image = "debian-cloud/debian-10"
}
}
metadata_startup_script = "sudo apt update -y; sudo apt install wget htop -y;"
network_interface {
subnetwork = google_compute_subnetwork.default.id
network_ip = google_compute_address.manager_ip_two.address
}
provisioner "local-exec" {
command = "echo ${google_compute_address.manager_ip_two.address} >> private_ips.txt"
}
}
resource "google_compute_firewall" "ssh" {
name = "allow-ssh"
allow {
ports = ["22"]
protocol = "tcp"
}
direction = "INGRESS"
network = google_compute_network.default.id
priority = 1000
source_ranges = ["0.0.0.0/0"]
target_tags = ["ssh"]
}
resource "google_compute_firewall" "icmp" {
name = "allow-icmp"
allow {
protocol = "icmp"
}
direction = "INGRESS"
network = google_compute_network.default.id
priority = 1001
source_ranges = ["0.0.0.0/0"]
target_tags = ["icmp"]
}
I think that in order to access an external internet from a compute engine (leaving aside firewall rules), the compute engine should either have an external IP address, or a make a connection using a Cloud NAT.
To communicate with the internet, you can use an external IPv4 or external IPv6 address configured on the instance. If the instance doesn't have an external address, Cloud NAT can be used for IPv4 traffic.
In case of an external IP address, you might like to add a few lines into the terraform script (I use a snippet from your code above) configuring an "access config" section:
network_interface {
subnetwork = google_compute_subnetwork.default.id
network_ip = google_compute_address.manager_ip_one.address
access_config {
// Ephemeral public IP
}
}
Or, presumably, you created/reserved an external IP address in your terraform script as well (let's say with a name manager_ip_ext):
network_interface {
subnetwork = google_compute_subnetwork.default.id
network_ip = google_compute_address.manager_ip_one.address
access_config {
// Explicit public IP
nat_ip = google_compute_address.manager_ip_ext.address
}
}
Another option, as mentioned above, is to organise egress through a Cloud NAT solution. Some details are provided in the documentation - Set up and manage network address translation with Cloud NAT Cloud NAT can be deployed/managed through Terraform as well.
Related
I'm trying to deploy a managed instance group with a load balancer which will be running a web server container.
The container is stored in the google artifcat registry.
If I manually create a VM and define the container usage, it is successfully able to pull and activate the container.
When I try to create the managed instance group via terraform, the VM does not pull nor activate the container.
When I ssh to the VM and try to manually pull the container, I get the following error:
Error response from daemon: Get https://us-docker.pkg.dev/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
The only notable difference between the VM I created manually to the VM created by terraform is that the manual VM has an external IP address. Not sure if this matters and not sure how to add one to the terraform file.
Below is my main.tf file. Can anyone tell me what I'm doing wrong?
terraform {
required_providers {
google = {
source = "hashicorp/google"
version = "3.53.0"
}
google-beta = {
source = "hashicorp/google-beta"
version = "~> 4.0"
}
}
}
provider "google" {
credentials = file("compute_lab2-347808-dab33a244827.json")
project = "lab2-347808"
region = "us-central1"
zone = "us-central1-f"
}
locals {
google_load_balancer_ip_ranges = [
"130.211.0.0/22",
"35.191.0.0/16",
]
}
module "gce-container" {
source = "terraform-google-modules/container-vm/google"
version = "~> 2.0"
cos_image_name = "cos-stable-77-12371-89-0"
container = {
image = "us-docker.pkg.dev/lab2-347808/web-server-repo/web-server-image"
volumeMounts = [
{
mountPath = "/cache"
name = "tempfs-0"
readOnly = false
},
]
}
volumes = [
{
name = "tempfs-0"
emptyDir = {
medium = "Memory"
}
},
]
restart_policy = "Always"
}
resource "google_compute_firewall" "rules" {
project = "lab2-347808"
name = "allow-web-ports"
network = "default"
description = "Opens the relevant ports for the web server"
allow {
protocol = "tcp"
ports = ["80", "8080", "5432", "5000", "443"]
}
source_ranges = ["0.0.0.0/0"]
#source_ranges = local.google_load_balancer_ip_ranges
target_tags = ["web-server-ports"]
}
resource "google_compute_autoscaler" "default" {
name = "web-autoscaler"
zone = "us-central1-f"
target = google_compute_instance_group_manager.default.id
autoscaling_policy {
max_replicas = 10
min_replicas = 1
cooldown_period = 60
cpu_utilization {
target = 0.5
}
}
}
resource "google_compute_instance_template" "default" {
name = "my-web-server-template"
machine_type = "e2-medium"
can_ip_forward = false
tags = ["ssh", "http-server", "https-server", "web-server-ports"]
disk {
#source_image = "cos-cloud/cos-73-11647-217-0"
source_image = module.gce-container.source_image
}
network_interface {
network = "default"
}
service_account {
#scopes = ["userinfo-email", "compute-ro", "storage-ro"]
scopes = ["cloud-platform"]
}
metadata = {
gce-container-declaration = module.gce-container.metadata_value
}
}
resource "google_compute_target_pool" "default" {
name = "web-server-target-pool"
}
resource "google_compute_instance_group_manager" "default" {
name = "web-server-igm"
zone = "us-central1-f"
version {
instance_template = google_compute_instance_template.default.id
name = "primary"
}
target_pools = [google_compute_target_pool.default.id]
base_instance_name = "web-server-instance"
}
Your VM templates haven't public IPs, therefore, you can't reach public IP.
However, you have 3 ways to solve that issue:
Add a public IP on the VM template (bad idea)
Add a Cloud NAT on your VM private IP range to allow outgoing traffic to the internet (good idea)
Activate the Google private access in the subnet that host the VM private iP range. It create a bridge to access to Google services without having a public IP (my prefered idea) -> https://cloud.google.com/vpc/docs/configure-private-google-access
Apparently I was missing the following acecss_config inside network_interface of the google_compute_instance_template as following:
network_interface {
network = "default"
access_config {
network_tier = "PREMIUM"
}
i have this terraform to use to assign a static private ip address to a new machine
i run this terraform but i still see
Error: Error creating instance: googleapi: Error 400: Invalid value
for field 'resource.networkInterfaces[0].networkIP':
'192.168.128.206'. IP address
'projects/prj-xxxxx/regions/europe-xxx/addresses/demo-xxxx-nodoa-c'
(192.168.128.206') is reserved by another project., invalid
this i the code used
resource "google_compute_address" "static-ip" {
for_each = toset(var.zones)
name = "${local.infra_id}-nodoa-${each.key}"
project = var.host_project.project_id
address_type = "INTERNAL"
region = var.region
purpose = "GCE_ENDPOINT"
subnetwork = var.host_project.nodoa_subnet_name
}
resource "google_compute_instance" "nodoa" {
for_each = toset(var.zones)
project = var.service_project.project_id
name = "${local.infra_id}-nodoa-${each.key}"
hostname = "${local.infra_id}-nodoa-${each.key}.${local.subdomain}"
machine_type = "xxx"
zone = "${var.region}-${each.key}"
network_interface {
subnetwork = var.host_project.nodoa_subnet_name
subnetwork_project = var.host_project.project_id
network_ip = google_compute_address.static-ip[each.key].address
}
boot_disk {
initialize_params {
image = var.rhcos_gcp_image
size = var.install_config_params.disk_size
type = "pd-ssd"
}
kms_key_self_link = ecr
}
#cava cavamagie I guess you need to specify 'network_ip' block inside access_config block.
access_config {
nat_ip = google_compute_address.static-ip[each.key].address
}
So in general your resource block for "google_compute_instance" should look like below
resource "google_compute_instance" "nodoa" {
for_each = toset(var.zones)
project = var.service_project.project_id
name = "${local.infra_id}-nodoa-${each.key}"
hostname = "${local.infra_id}-nodoa-${each.key}.${local.subdomain}"
machine_type = "xxx"
zone = "${var.region}-${each.key}"
network_interface {
subnetwork = var.host_project.nodoa_subnet_name
subnetwork_project = var.host_project.project_id
access_config {
nat_ip = google_compute_address.static-ip[each.key].address
}
}
boot_disk {
initialize_params {
image = var.rhcos_gcp_image
size = var.install_config_params.disk_size
type = "pd-ssd"
}
kms_key_self_link = ecr
}
Reference : How to map static IP to terraform google compute engine instance?
How to add the default-allow-http firewall rule in a terraform script to a Google Cloud Compute Instance?
provider "google" {
credentials = file("CREDENTIAL_FILE")
project = "gitlab-project"
region = var.region
}
resource "google_compute_instance" "gitlab" {
name = var.machine_specs.name
machine_type = var.machine_type.emicro
zone = var.zone
boot_disk {
initialize_params {
image = var.machine_specs.os
size = var.machine_specs.size
}
}
network_interface {
# A default network is created for all GCP projects
network = "default"
access_config {
nat_ip = google_compute_address.static.address
}
}
// Add the SSH key
metadata = {
ssh-keys = "martin:${file("~/.ssh/id_rsa.pub")}"
}
}
// A variable for extracting the external ip of the instance
output "ip" {
value = "${google_compute_instance.gitlab.network_interface.0.access_config.0.nat_ip}"
}
resource "google_compute_address" "static" {
name = "ipv4-address"
address_type = "EXTERNAL"
address = "XXX.XXX.XXX.XXX"
}
resource "google_compute_firewall" "allow-http" {
name = "default-allow-http"
network =
allow{
protocol = "tcp"
ports = ["80"]
}
}
You can use tags argument available in google_compute_instance resource.
it would look something like:
resource "google_compute_instance" "gitlab" {
name = var.machine_specs.name
machine_type = var.machine_type.emicro
zone = var.zone
tags = ["http-server"]
http-server tag is for default-allow-http firewall rule.
If you need default-allow-https then simply append https-server to the tag list.
Hope this helps.
You need to add the tags ["http-server", "https-server"] to your resource group google_compute_instance like so:
[...]
resource "google_compute_instance" "gitlab" {
name = var.machine_specs.name
machine_type = var.machine_type.emicro
zone = var.zone
tags = ["http-server", "https-server"]
[...]
Simply add the tags http-server and https-server to your google_cloud_instance resource gruop.
The tags can be found in the Firewall-settings in your GCloud-Console.
I am attempting to write automation to deploy instances in a shared VPC on GCP. I have a host network project and a service project. I can create a static internal IP address resource in the host project (resource "google_compute_address" "internal") in which I specify the VPC host project (NET_HUB_PROJ) but I am unable to use it when creating the instance. I receive the following error:
google_compute_instance.compute: Error creating instance: googleapi:
Error 400: Invalid value for field
'resource.networkInterfaces[0].networkIP': '10.128.0.10'. IP address
'projects/prototype-network-hub/regions/us-central1/addresses/bh-int-
ip' (10.128.0.10) is reserved by another project., invalid
My compute module:
data "google_compute_image" "image" {
name = "${var.IMAGE_NAME}"
project = "${var.IMAGE_PROJECT}"
}
resource "google_compute_address" "internal" {
name = "${var.NAME}-int-ip"
address_type = "INTERNAL"
address = "${var.PRIVATE_IP}"
subnetwork = "${var.NET_HUB_SUBNETWORK}"
region = "${var.NET_HUB_REGION}"
project = "${var.NET_HUB_PROJ}"
}
resource "google_compute_address" "external" {
count = "${var.EXT_IP_CREATE ? 1 : 0}"
name = "${var.NAME}-ext-ip"
address_type = "EXTERNAL"
region = "${var.REGION}"
}
resource "google_compute_instance" "compute" {
depends_on = ["google_compute_address.external"]
name = "${var.NAME}"
machine_type = "${var.MACHINE_TYPE}"
zone = "${var.ZONE}"
can_ip_forward = "${var.CAN_IP_FORWARD}"
deletion_protection ="${var.DELETION_PROTECTION}"
allow_stopping_for_update = "${var.ALLOW_STOPPING_FOR_UPDATE}"
tags = ["allow-ssh"]
metadata = {
"network" = "${var.NETWORK}"
"env" = "${var.ENV}"
"role" = "${var.ROLE}"
"region" = "${var.REGION}"
"zone" = "${var.ZONE}"
}
labels = {
"network" = "${var.NETWORK}"
"env" = "${var.ENV}"
"role" = "${var.ROLE}"
"region" = "${var.REGION}"
"zone" = "${var.ZONE}"
}
boot_disk {
device_name = "${var.NAME}"
auto_delete = "${var.BOOT_DISK_AUTO_DELETE}"
initialize_params {
size = "${var.BOOT_DISK_SIZE}"
type = "${var.BOOT_DISK_TYPE}"
image = "${data.google_compute_image.image.self_link}"
}
}
network_interface {
network_ip = "${google_compute_address.internal.address}"
subnetwork_project = "${var.NET_HUB_PROJ}"
subnetwork = "projects/prototype-network-hub/regions/us-central1/subnetworks/custom"
access_config {
nat_ip = "${element(concat(google_compute_address.external.*.address, list("")), 0)}"
}
}
service_account {
scopes = ["service-control", "service-management", "logging-write", "monitoring-write", "storage-ro", "https://www.googleapis.com/auth/trace.append" ]
}
}
The end goal would be to accomplish the following:
EDIT (new answer):
Per the GCP documentation, the static internal IP must belong to the service project (not the host network project as in your code) if you're looking to reserve internal IP on a shared VPC in a different project. See here:
https://cloud.google.com/vpc/docs/provisioning-shared-vpc#reserve_internal_ip
Seeing as a shared-vpc is unlikely to be found in your TF codebase, you'll have to use data to get the self_link of the subnetwork to use for google_compute_address. Something like the following:
data "google_compute_subnetwork" "subnet" {
name = "${var.NET_HUB_SUBNETWORK}"
project = "${var.NET_HUB_PROJ}"
region = "${var.NET_HUB_REGION}"
}
resource "google_compute_address" "internal" {
name = "${var.NAME}-int-ip"
address_type = "INTERNAL"
address = "${var.PRIVATE_IP}"
subnetwork = "${data.google_compute_subnetwork.subnet.self_link}"
}
This should create the resource under your service project, yet with an address within the designated subnet.
When you deploy your instance you should see it referenced under the internal_ip column on your VM instances tab for the assigned instance.
(old answer for posterity):
Unfortunately, google_compute_address doesn't contain a subnetwork_project like google_compute_instance. A fix around this is to provide a full URL to the subnetwork field in google_compute_address. Something like the following:
resource "google_compute_address" "internal" {
name = "${var.NAME}-int-ip"
address_type = "INTERNAL"
address = "${var.PRIVATE_IP}"
subnetwork = "https://www.googleapis.com/compute/v1/projects/${var.NET_HUB_PROJ}/regions/${var.NET_HUB_REGION}/subnetworks/${var.NET_HUB_SUBNETWORK}"
}
Adding my solution below:-
resource "google_compute_address" "internal_ip" {
count = 1
name = "${local.cluster_name}-int-ip-${count.index}"
project = <service project id>
subnetwork = <host project subnet self_link>
address_type = "INTERNAL"
region = "asia-northeast1"
purpose = "GCE_ENDPOINT"
}
output "internal_ipaddr_info" {
value = google_compute_address.internal_ip
}
resource "google_compute_instance" "test" {
project = module.gcp_service_project.project_id
name = "eps-gce-vm-d-swmgr-ane1-test"
machine_type = "n1-standard-1"
zone = "asia-northeast1-a"
can_ip_forward = true
boot_disk {
initialize_params {
image = "centos7"
}
}
network_interface {
subnetwork = <host project subnet self_link>
network_ip = google_compute_address.internal_ip[0].self_link
}
}
In my terraform configuration file, I define my resource like so:
resource "google_compute_instance" "test" {
...
count = 2
}
What I now want is to create load balancer, that will balance between two instances of my google compute instance. Unfortunatelly, I could not find in documentation anything relative to this task. It seems like google_compute_target_pool or google_compute_lb_ip_ranges have nothing to do with my problem.
You would have to use 'forwarding rules' as indicated on this terraform document. To use load balancing and protocol forwarding, you must create a forwarding rule that directs traffic to specific target instances. The use on Cloud Platform of forwarding rules you can find here.
In common cases you can use something like the following:
resource "google_compute_instance" "test" {
name = "nlb-node${count.index}"
zone = "europe-west3-b"
machine_type = "f1-micro"
count = 2
boot_disk {
auto_delete = true
initialize_params {
image = "ubuntu-os-cloud/ubuntu-1604-lts"
size = 10
type = "pd-ssd"
}
}
network_interface {
subnetwork = "default"
access_config {
nat_ip = ""
}
}
service_account {
scopes = ["userinfo-email", "compute-ro", "storage-ro"]
}
}
resource "google_compute_http_health_check" "nlb-hc" {
name = "nlb-health-checks"
request_path = "/"
port = 80
check_interval_sec = 10
timeout_sec = 3
}
resource "google_compute_target_pool" "nlb-target-pool" {
name = "nlb-target-pool"
session_affinity = "NONE"
region = "europe-west3"
instances = [
"${google_compute_instance.test.*.self_link}"
]
health_checks = [
"${google_compute_http_health_check.nlb-hc.name}"
]
}
resource "google_compute_forwarding_rule" "network-load-balancer" {
name = "nlb-test"
region = "europe-west3"
target = "${google_compute_target_pool.nlb-target-pool.self_link}"
port_range = "80"
ip_protocol = "TCP"
load_balancing_scheme = "EXTERNAL"
}
You can get load balancer external ip via ${google_compute_forwarding_rule.network-load-balancer.ip_address}
// output.tf
output "network_load_balancer_ip" {
value = "${google_compute_forwarding_rule.network-load-balancer.ip_address}"
}