TLDR: I could use some advice on how to setup Redis for production use on GPC, it just switched IP addresses on us randomly, and there is nothing in the documentation about that / I have no idea how to build a stable solution with that possibility.
Background:
We've been using google cloud for a few years and had a stable Redis Memorystore instance on the 'Standard' Tier.
In the past few days, our web servers started slowly crashing every so often. After investigating, something was locking up when connecting to celery / Redis, and we found that all our config files had 10.0.0.3 as the Redis instance, and the IP address for the server was listed as 10.0.0.4. This hasn't changed ever, and our configs are in git so we're sure they were unchanged.
Since Celery won't boot up with a bad connection we know it was correct on Tuesday when we pushed up new code. It seems like the server failed over and somehow issued an IP address change on us. As evidence,
Our graphical usage bizarrely change color at a specific point
Which matches our error logs "[2020-06-16 03:09:21,873: ERROR/MainProcess] Error in timer: ReadOnlyError("You can't write against a read-only slave.",)"
All the documentation we have found says the IP address would stay the same, but given that didn't happen, I'm hoping for some feedback on how one would work around a non-static IP in this case on GPC
Memorystore does not support static IP address. Some scenarios where IP address change can occur are restarts or when connection modes are changed.
From review of the Memorystore for Redis networking page, when using direct access connection via IP address your project will set up a VPC network peering connection with Google's internal project, where the instance is managed. This will create an allocated IP range for Memorystore to use for the instances, this can either be provided by you or picked from the available space (will be a /29 block by default).
On the other hand, Memorystore for Redis exposes the uptime as a metric that is available through Cloud Monitoring (formally Stackdriver). This can be used as a health check for the instance as you will be able to determine if there has been a restart or points of unavailability.
Following the point above, you are able to set up an alert on the uptime metric directly in Cloud Monitoring. Unfortunately there is nothing specific to IP address changes though.
Related
I am using express-sessions and Redis(Memorystore) 5.0 to save sessions and it appears Redis is clearing out all the keys(randomly and not at intervals) way before the TTL on a key runs out. Leaving only a couple of backup entries
This entry should be valid for another week judging from the TTL.
I have never configured a Redis instance before and it is likely I misconfigured this one, some insight would be appreciated.
Also, this is what I get when I run monitor
Avoid exposing your Memorystore instance through the external IP of your Compute Engine. Combined with the fact that
Memorystore instances currently requires no authentication, this will lead to a vulnerability that
will allow anyone to read, write and execute scripts on your instance. On the worst case scenario, it will allow someone to mine cryptocurrency using your instance, hence causing a suspension on your project or account unfortunately.
There are multiple guides online to connect to Memorystore remotely. I suggest connecting through SSH by following this thread: Accessing GCP Memorystore from local machines
For reference, I will paste the answer here (credits to the person who posted it):
"If your Redis machine has internal IP address 10.0.0.3 you'd do:
gcloud compute instances create redis-forwarder --machine-type=f1-micro
gcloud compute ssh redis-forwarder -- -N -L 6379:10.0.0.3:6379
As long as you keep the ssh tunnel open you can connect to localhost:6379"
We deployed the Gitlab server on a Compute Engine instance with an attached static external IP address. From time to time, the server is unavailable to us upon requests from Russia. When using the "tracert" command, packets do not go beyond the address 209.85.243.152 (Google LLC). However, the Gitlab site is fully accessible upon requests from other regions at any time.
The problem was preinstalled sshguard, which mistakenly added my IP to the list of blocked ones.
There is a solution instruction in this answer.
I recently start using GCP but i have one thing i can't solve.
I have: 1 VM + 1 DB Instance + 1 LB. DB instance allow only conections from the VM IP. bUT THE VM IP allow traffic from all ip (if i configure the firewall to only allow CloudFlare and LB IP's the website crash and refuse conections).
Recently i was under attack, i activate the Cloudflare ddos mode, restart all and in like 6 h the attack come back with the Cloudflare activate. Wen i see mysql conections bump from 20-30 to 254 and all conections are from the IP of the VM so i think the problem are the public accesibility of the VM but i don't know how to solved it...
If i activate my firewall rules to only allow traffic from LB and Cloudflare the web refuses all conections..
Any idea what i can do?
Thanks.
Cloud Support here, unfortunately, we do not have visibility into what is installed on your instance or what software caused the issue.
Generally speaking you're responsible for investigating the source of the vulnerability and taking steps to mitigate it.
I'm writing here some hints that will help you:
Make sure you keep your firewall rules in a sensible manner, e.g. is not a good practice to have a firewall rule to allow all ingress connections on port 22 from all source IPs for obvious reasons.
Since you've already been rooted, change all your passwords: within the Cloud SQL instance, within the GCE instance, even within the GCP project.
It's also a good idea to check who has access to your service accounts, just in case people that aren't currently working for you or your company still have access to them.
If you're using certificates revoke them, generate new ones and share them in a secure way and with the minimum required number of users.
Securing GCE instances is a shared responsability, in general, OWASP hardening guides are really good.
I'm quoting some info here from another StackOverflow thread that might be useful in your case:
General security advice for Google Cloud Platform instances:
Set user permissions at project level.
Connect securely to your instance.
Ensure the project firewall is not open to everyone on the internet.
Use a strong password and store passwords securely.
Ensure that all software is up to date.
Monitor project usage closely via the monitoring API to identify abnormal project usage.
To diagnose trouble with GCE instances, serial port output from the instance can be useful.
You can check the serial port output by clicking on the instance name
and then on "Serial port 1 (console)". Note that this logs are wipped
when instances are shutdown & rebooted, and the log is not visible
when the instance is not started.
Stackdriver monitoring is also helpful to provide an audit trail to
diagnose problems.
You can use the Stackdriver Monitoring Console to set up alerting policies matching given conditions (under which a service is considered unhealthy) that can be set up to trigger email/SMS notifications.
This quickstart for Google Compute Engine instances can be completed in ~10 minutes and shows the convenience of monitoring instances.
Here are some hints you can check on keeping GCP projects secure.
I currently have a google cloud compute instance set up to be the back-end for a multiplayer game. Certain publishers and app stores that I'm trying to publish the game on require that the server can be reached via a client using an IPv6 address, which makes perfect sense. So the question is, how do I go about making it that the compute instance can be connected to via IPv6?
It's worth noting that the connection between the client and server is done via UDP, so using load balancing doesn't appear to work (since load balancers in google cloud can only be done over TCP, from what I can tell).
Has anyone else had this issue, and if so how did you solve it?
Many thanks in advance.
IPv6 Termination for HTTP(S), SSL Proxy, and TCP Proxy Load Balancing is currently in Beta.
https://cloud.google.com/compute/docs/load-balancing/ipv6
Configuring IPv6 termination for your load balancers lets your backend instances appear as IPv6 applications to your IPv6 clients.
Note: The documentation says this feature is not covered by any SLA or deprecation policy and may be subject to backward-incompatible changes.
The definition of Beta from their documentation: Beta is the point at which we are ready to open a release for any customer to use. There are no SLA or technical support obligations in a Beta release, and charges may be waived in some cases. Products will be complete from a feature perspective, but may have some open outstanding issues. Beta releases are suitable for limited production use cases.
https://cloud.google.com/terms/launch-stages
IPv6 Termination for HTTP(S), SSL Proxy, and TCP Proxy Load Balancing became GA on September 20, 2017.
Source: https://cloudplatform.googleblog.com/2017/09/announcing-ipv6-global-load-balancing-ga.html.
See the documentation at https://cloud.google.com/compute/docs/load-balancing/ipv6
Keep in mind that inside the GCP network, all is still on IPv4, https://issuetracker.google.com/issues/35904387
Google cloud now supports external ipv6 on VM instances. Each instance can get a /96 external ip range and it can be used to access internet (without NAT) or be used for VM to VM traffic.
At this moment (July 2021) it's only supported limited regions:
asia-east1
asia-south1
europe-west2
us-west2
See more detailed in
https://cloud.google.com/compute/docs/ip-addresses/configure-ipv6-address https://cloud.google.com/vpc/docs/vpc#ipv6-addresses
If your instance happened to be one of the 4 regions above then you should be able to use the VM instance IPv6 feature.
May 2022 update.
Per https://cloud.google.com/vpc/docs/subnets#limitations
Internal and external IPv6 subnets are available in all but asia-southeast2 and asia-northeast3 regions.
I have a load balancer and EC2 instance with AWS. I had problems with e-mail restrictions and was recommended to use an elastic IP. I then read somethere that you can't use elastic IP and a load balancer so I removed the elastic IP. I can no longer access my instance even when I've rebooted it and waiting 2 hours later. I can ping it (after enabling ICMP with network security) but I can't SSH or go to the web server. All the network settings remain, which included allowing TCP ports for HTTP and SSH. Does anyone know what has happened to make port 80 and 23 no longer accessible? This is a real nightmare for me because I did a bit of a marketing campaign, got increased traffic, noticed emails weren't getting sent, then in an attempt to fix that I've screwed the server completely so the website is down at the worst possible time :(
I fixed it all up. This isn't a direct solution to the problem, more like a workaround. I couldn't connect to that server no matter what, so I created a new instance and that worked. It was as if the Linux server itself was corrupt, not the AWS settings. I detached the volume from the old instance and attached it as a secondary volume on the new instance. When I logged into the new instance I was able to mount the secondary volume as a new drive and I just copied the files over that way. I don't have a bloated server so this wasn't really a big deal to pull off. Anyway, if you can't log in to a server anymore, you can always mount it to a new instance and access it via the file-system