Kubernetes does not seem to update internal IP table after node restart - google-cloud-platform

Question

We're currently experiencing an issue with our GCP Kubernetes which is forwarding client requests to pods that have been assigned IPs that other pods within the cluster have /previously/ gotten. The way we can see this is by using the following query in Logs Explorer:
resource.type="http_load_balancer"
httpRequest.requestMethod="GET"
httpRequest.status=404
Snippet from one of the logs:
httpRequest: {
latency: "0.017669s"
referer: "https://asdf.com/"
remoteIp: "5.57.50.217"
requestMethod: "GET"
requestSize: "34"
requestUrl: "https://[asdf.com]/api/service2/[...]"
responseSize: "13"
serverIp: "10.19.160.16"
status: 404
userAgent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36"
}
...where the requestUrl property indicates the incoming URL to the load balancer.
Then I search for the IP 10.19.160.16 to find out which pod the IP is assigned to:
c:\>kubectl get pods -o wide | findstr 10.19.160.16
service1-675bfc4f97-slq6g 1/1 Terminated 0 40h 10.19.160.16 gke-namespace-te-namespace-te-153a9649-p2mg
service2-574d69cf69-c7knp 0/1 Error 0 3d16h 10.19.160.16 gke-namespace-te-namespace-te-153a9649-p2mg
service3-6db4c97784-428pq 1/1 Running 0 16h 10.19.160.16 gke-namespace-te-namespace-te-153a9649-p2mg
So based on requestUrl the request should have been sent to service2. Instead, what we see is that it gets sent to service3 because it's gotten the IP that service2 once used to have, in other words it seems that the cluster still thinks that service2 is holding on to the IP 10.19.160.16. The effect is that service3 returns status code 404 because it doesn't recognize the endpoint.
This behavior only stops if we manually delete the pods in failed state (eg Error or Terminated) by using the kubectl delete pod ... command.
We suspect that this behavior started since we upgraded our cluster to v1.23 which required us to migrate away from extensions/v1beta1 to networking.k8s.io/v1 as described in https://cloud.google.com/kubernetes-engine/docs/deprecations/apis-1-22.
Our test environment is using pre-emptible VM and whilst we're not 100% (but pretty close) sure it seems like the pods end in Error state after a node is pre-empted.
Why does the cluster still think that a dead pod still has the IP that it used to have? Why is the problem gone after deleting failed pods? Shouldn't they have been cleaned up after a node pre-emption?

Answer 1

Gari Singh provided the answer in the comment.