Actually when I create a new machine in GCP, the google compute engine is assigning to me an IP 1xx.
Only I have arround 30 machines, but, when kubernetes generate a new node, the IP is incremented.
How I can reset this IP pool?
For example now I'm trying to generate a marketplace deploy (rabbitmq) and isn't possible to reserve internal IP.
TL;DR: CGE Instances supports two types of internal IP addresses: Static Internal and Ephemeral. For both cases, IPs remain attached to a resource until they are explicitly detached from the resource. So if you can't reserve more IP addresses is because you ran out of IP addresses.
Google released a document named IP addresses where it explains how IP addresses are assigned to resources within GCP.
Based on your question, seems that you have custom VPC. The error you get is because you ran out of IP addresses.
Ephemeral IP addresses are released when the resource is deleted. But Static Internal Addresses are not automatically deleted when the resource is deleted, it only gets detached. Most likely you have a lot of reserved addresses not attached to resources.
Lastly, Stackoverflow may be not the best forum for this question. I highly recommend you to share your question in Serverfault since is a better forum for infrastructure questions.
Related
I have a AWS VPC where the DHCP optionset gives me random addresses in the scope of the CIDR 10.128.1.0/24.
But I want to have the DHCP scope limited, so that I can use static private ip addresses for some of the servers and dynamic for another set of server, like
Static Server scope:
10.128.1.1 .. 10.128.1.99
Dynamic Server scope:
10.128.1.100 .. 10.128.1.200
I have tried using the aws ec2 create-dhcp-options but it doesn't give me the opportunity to limit any ip addresses.
The question is perhaps, Can it done? then of course how to, or should I think complete different when having this perhaps odd need and should do it another AWS way....?
You can't do this on AWS. It is common on standard networking, but not on cloud.
If you really need to use "static" IPs you can reserve the IP creating ENI, Elastic Network Interface, in advance.
When an ENI exist, even if it is available, the IP associated with ENI is already in use, so DHCP will not use it anymore
When you create an EC2 instance you can allow it to use DHCP to get an random IP or you can choose an available ENI.
Just notice some IPs are reserved by AWS, you can't create an ENI for it.
See the reserved IPs on documentation below:
https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html#subnet-sizing
We can reserve a static IP address on Google following the instructions here.
Would it be possible to migrate this IP address to another Cloud, e.g. Amazon, IBM, or elsewhere, and how would that work.
If this is not possible, how can I have one IP address that I can move around?
The use case is for an SFTP setup, which we have now on Google and is already whitelisted by an external party. I would like to change the SFTP server to be deployed on another Cloud Provider, and to still use this IP address so my client does not have to whitelist another IP address.
Reserving addresses per your link, reserves you a Google-owned IP.
You cannot move Google's IP addresses but you can bring-your-own IP address (blocks) to Google and AWS and possibly others.
The IPs can only be lent to one Cloud provider at a time so you'd have some downtime migrating your IPs from one provider to another.
But, it should be possible (with limitations) to achieve your goal.
Ephemeral external IP addresses: these addresses are available to VM
instances and forwarding rules. Ephemeral external IP addresses remain
attached to a VM instance only until the VM is stopped and restarted
or the instance is terminated. If an instance is stopped, any
ephemeral external IP addresses that were assigned to the instance are
released back into the general Compute Engine pool and become
available for use by other projects. When a stopped instance is
started again, a new ephemeral external IP address is assigned to the
instance.
I have thought is instantaneously and we won't be getting back the same ephemeral IP address once the VM is stopped. However, it seems that ephemeral external IP addresses will still be assigned to a project for a while longer. For instance, I deleted a VM and released an IP address two days ago. The same IP address is assigned to a new VM today.
How long does it take to release the ephemeral IP addresses back to the GCE pool?
Are the ephemeral external IP addresses "assigned" to VPC or Project? If one has multiple projects within a VPC, will the ephemeral IP addresses be rotated within the projects until they are released to the GCE pool?
The answer is "it should not matter if ephemeral addresses are reused for your instance or not".
There is no guarantee that your instance will obtain the same address or will not obtain the same address. If you are designing something that depends on a certain ephemeral IP address behavior, your design will fail at some point.
Your question quotes Google's official ephemeral IP address policy. Design to that statement and do not depend on environment level behavior.
If you require a fixed IP address, then assign your instance a static IP address. Otherwise, your instance will have whatever address Google Cloud decides, which may or may not be the same address between restarts or recreates.
If your goal is to randomize public IP addresses, then you cannot count on the ephemeral address behavior to implement that. You can allocate a set of static IP addresses and then change which one is assigned to an instance. Note that unused IP addresses are billed (not free). Another method is to create instances in different regions and zones which will have different public IP addresses. You also could write a script to create VMs until the address is different (not part of a previous set of addresses) and then delete the other VMs (subject to quota restrictions).
From a customer support perspective, it is to Google's benefit to reallocate the same address to you. This minimizes a common problem. Some customers do not understand that the default IP address is ephemeral and what that means. They reboot their instance and the IP address changes. That breaks their SSH scripts, DNS settings, etc. The ephemeral address behavior cannot be consistently deployed but is a positive where possible.
Answers to your questions -
It’s instantaneous.
Ephemeral addresses don't belong to any project and can be assigned to any resources across projects, randomly.
Some users have done some tests like this and concluded that the system tends to assign familiar IP addresses to VMs if it can. However, this is not something confirmed by GCP. Regardless, it’s of no use as there is no guarantee what IP address you’ll be getting.
For our project we need a static IP binding to our Google Cloud VM instance due to IP whitelisting.
Since it's a managed group preemptible, the VM will terminate once in a while.
However, when it terminates I see in the operations log compute.instances.preempted directly followed by compute.instances.repair.recreateInstance with the note:
Instance Group Manager 'xxx' initiated recreateInstance on instance
'xxx'.
Reason: instance's intent is RUNNING but instance's status is
STOPPING.
After that follows a delete and a insert operation in order to restore the instance.
The documentation states:
You can simulate an instance preemption by stopping the instance.
In which case the IP address will stay attached when the VM is started again.
A) So my question, is it possible to have the instance group manager stop and start the VM in the event of preemption, instead of recreating? Since recreating means that the static IP will be detached and needs to be manually attached each time.
B) If option A is not possible, how can I attach the static IP address automatically so that I don't have to attach it manually when the VM is recreated? I'd rather not have an extra NAT VM instance to take care of this problem.
Thanks in advance!
I figured out a workaround to this (specifically, keeping a static IP address assigned to a preemptible VM instance between recreations), with the caveat that your managed instance group has the following properties:
Not autoscaling.
Max group size of 1 (i.e. there is only ever meant to be one VM in this group)
Autohealing is default (i.e. only recreates VMs after they are terminated).
The steps you need to follow are:
Reserve a static IP.
Create an instance template, configured as preemptible.
Create your managed group, assigning your template to the group.
Wait for the group to spin up your VM.
After the VM has spun up, assign the static IP that you reserved in step 1 to the VM.
Create a new instance template derived from the VM instance via gcloud (see https://cloud.google.com/compute/docs/instance-templates/create-instance-templates#gcloud_1).
View the newly create instance template in the Console, and note that you see your External IP assigned to the template.
Update the MiG (Managed Instance Group) to use the new template, created in step 6.
Perform a proactive rolling update on the MiG using the Replace method.
Confirm that your VM was recreated with the same name, the disks were preserved (or not, depending on how you configured the disks in your original template), and the VM has maintained its IP address.
Regards to step 6, my gcloud command looked like this:
gcloud compute instance-templates create vm-template-with-static-ip \
--source-instance=source-vm-id \
--source-instance-zone=us-east4-c
Almost goes without saying, this sort of setup is only useful if you want to:
Minimize your costs by using a single preemptible VM.
Not have to deal with the hassle of turning on a VM again after it's been preempted, ensuring as much uptime as possible.
If you don't mind turning the VM back on manually (and possibly not being aware it's been shutdown for who knows how long) after it has been preempted, then do yourself a favor and don't bother with the MiG and just standup the singular VM.
Answering your questions:
(A) It is not possible at the moment, and I am not sure if it will ever be possible. By design preemptible VMs are deleted to make space for normal VMs (if there are capacity constraints in the given zone) or regularly to differentiate them from normal VMs. In the latter case preemption might seem like a start/stop event, but in the former it may take a substantial amount of time before the VM is recreated.
(B) At the moment there is not good way to achieve it in generality.
If you have a special case where your group has only one instance you can hardcode the IP address in the Instance Template
Otherwise at the moment the only solution I can think of (other than using a Load Balancer) is to write a startup script that would attach the NAT IP.
I've found one way that ensures that all VM's in your network have the same outgoing IP address. Using Cloud NAT you can assign a static IP which all VM's will use, there is a downside though:
GCP forwards traffic using Cloud NAT only when there are no other
matching routes or paths for the traffic. Cloud NAT is not used in the
following cases, even if it is configured:
You configure an external IP on a VM's interface.
If you configure an external IP on a VM's interface, IP packets with the VM's internal IP as the source IP will use the VM's
external IP to reach the Internet. NAT will not be performed on
such packets. However, alias IP ranges assigned to the interface
can still use NAT because they cannot use the external IP to reach
the Internet. With this configuration, you can connect directly to
a GKE VM via SSH, and yet have the GKE pods/containers use Cloud
NAT to reach the Internet.
Note that making a VM accessible via a load balancer external IP does not prevent a VM from using NAT, as long as the VM network
interface itself does not have an external IP address.
Removing the VM's external IP also prevents you from direct SSH access to the VM, even SSH access from the gcloud console itself. The quote above shows an alternative with a load balancer, another way is a bastion, but doesn't directly solve access from for example Kubernetes/kubectl.
If that's no problem for you, this is the way to go.
One solution is to let the instances have dynamically chosen ephemeral IPs, but set the group as the target of a Load Balancer with a static IP. This way even when instances are created or destroyed, the LB acts as a frontend keeping the IP continious over time.
When I allocate an Elastic IP, it stays with me forever (until I remove it).
This also means, no one can ever use this IP address in the entire world. And you can see there is not DNS or stuff. Its is a straight forward IP address reachable from browser.
How does Amazon block the IP to itself? In other words, how does Amazon hold an IP address so that no one can use it.
Same way anybody else can:
https://www.apnic.net/get-ip/faqs/obtaining-resources/
Can I buy IP addresses?
IP addresses are not for sale, instead, they are public resources
administered by the Internet Numbers Registry System for the entire
Internet community. IP addresses delegated by the Regional registries,
such as APNIC are not "owned" by the Members. IP addresses will be
reclaimed by APNIC when the Membership is terminated.
Why does APNIC require so much detailed information about my network?
APNIC must ensure that address space, which is a shared public
resource, is managed responsibly in the region. APNIC needs detailed
information so that it can properly evaluate whether your plans
conform to agreed addressing policy and are consistent with the goals
of global Internet management.