Slow Performance with libcurl when changing Ip Address - c++

Question

First time developing with libcurl and got some unexpected results:
I'm trying to reuse the same CURL handle in order to reuse it's connections associated with it. However, I noticed a huge performance drop when I constantly change the Ip address of my request.
When I keep the IP address unchanged everything is fine, I'm pretty sure there's something I'm missing, but I don't know what and where to search for.
Here's my code:
// curl init
curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &RestClientPool::WriteCallBack); // custom write function
curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE, 1L);
curl_easy_setopt(curl, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT_MS, 200L);
// perform request, curl handle is stored and reused
curl_easy_setopt(curl, CURLOPT_HTTPGET, 1L);
curl_easy_setopt(curl, CURLOPT_URL, req_url.c_str()); // here req_url is something like http://123.123.123.123:8080/api/api
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &response);
CURLcode res = curl_easy_perform(curl);
Problem:
say i have 3 hosts with ip: a, b and c, if i send requests to a,b,c,a,b,c,a... 100 times V.S. a,a,a,a,a,a... 100 times with the same handle, performance is around 9s to 1s (service on all hosts are identical, no host issues needs to be considered)
PS: I can tell that libcurl handle is reusing the connections by monitoring（and by verbose）, because i monitored the servers. So, new connections when ip change shouldn't be the problem here.
Environment:
libcurl:
curl 7.86.0 (x86_64-pc-linux-gnu) libcurl/7.86.0 OpenSSL/1.0.2k-fips zlib/1.2.7
Release-Date: 2022-10-26
Protocols: dict file ftp ftps gophergophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS HSTS HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL UnixSockets
os:
CentOS 7.6.1810
What I tried:
1、I trie using CURLOPT_RESOLVE(cause i thought it might have something do with DNS cache), but nothing changed.
2、I made three handle and fired off 3 threads, where each handle would request to same ip(host A for example) repeatedly(no handle were shared across threads). And it worked fine.
However, when I change the ip for each handle(thread 1 would now repeatedly make request to host A, thread 2 to B and thread 3 to C), the performance drop appears again.