From remote machine getting connection refused for GCP Bigquery - google-cloud-platform

Question

I am running a sample code from google to get a simple select query. Which is working fine in my local but from my k8s environment I am getting the below error
Exception in thread "main" com.google.cloud.bigquery.BigQueryException: Error getting access token for service account: connect timed out
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.create(HttpBigQueryRpc.java:220)
at com.google.cloud.bigquery.BigQueryImpl$5.call(BigQueryImpl.java:369)
at com.google.cloud.bigquery.BigQueryImpl$5.call(BigQueryImpl.java:366)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:365)
at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:340)
at com.rakuten.dps.dataplatform.ingest.utility.BQ_test.main(BQ_test.java:67)
Caused by: java.io.IOException: Error getting access token for service account: connect timed out
at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:444)
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:157)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:145)
at com.google.auth.oauth2.ServiceAccountCredentials.getRequestMetadata(ServiceAccountCredentials.java:603)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:91)
at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159)
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:88)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:422)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.create(HttpBigQueryRpc.java:218)
... 8 more
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:284)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:113)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:441)
... 19 more
Below is the sample Code:
public class BQ_test {
private static final Logger logger = LoggerFactory.getLogger(BQ_test.class);
public static void main(String[] args) {
Job queryJob = null;
String actualValue = "";
NetHttpTransport transport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
String query = "SELECT * FROM `iconic-parsec-315409.bookmark_BQ.sbm_item_tbl``";
String projectId = "iconic-parsec-315409";
File credentialsPath = new File("/tmp/iconic-parsec-315409-823ef1c38a9d.json");
GoogleCredentials credentials;
try {
FileInputStream serviceAccountStream = new FileInputStream(credentialsPath);
credentials = ServiceAccountCredentials.fromStream(serviceAccountStream);
if (credentials.createScopedRequired()) {
Collection<String> bigqueryScopes = BigqueryScopes.all();
credentials = credentials.createScoped(bigqueryScopes);
}
BigQuery bigquery = BigQueryOptions
.newBuilder()
.setCredentials(credentials)
.setProjectId(projectId)
.build()
.getService();
QueryJobConfiguration queryConfig =
QueryJobConfiguration.newBuilder(query)
.setUseLegacySql(false)
.setJobTimeoutMs(180000L)
.build();
// Create a job ID so that we can safely retry.
JobId jobId = JobId.of(UUID.randomUUID().toString());
queryJob = bigquery.create(JobInfo.newBuilder(queryConfig).setJobId(jobId).build());
// Wait for the query to complete.
queryJob = queryJob.waitFor();
} catch (IOException | InterruptedException e) {
e.printStackTrace();
}
// Check for errors
if (queryJob == null) {
throw new RuntimeException("Job no longer exists");
} else if (queryJob.getStatus().getError() != null) {
// You can also look at queryJob.getStatus().getExecutionErrors() for all
// errors, not just the latest one.
throw new RuntimeException(queryJob.getStatus().getError().toString());
}
// Get the results.
TableResult result = null;
try {
result = queryJob.getQueryResults();
// Print all pages of the results.
// writeFvLToOrcFile(result,"/Users/susanta.a.adhikary/Downloads/test.orc");
for (FieldValueList row : result.iterateAll()) {
// String type
actualValue = row.get("sbm_item_id").getStringValue();
System.out.println(actualValue);
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
I tried curl -I "https://oauth2.googleapis.com/token" from my remote k8s pod and getting
HTTP/2 404
content-type: text/html
date: Sun, 04 Jul 2021 05:54:09 GMT
server: scaffolding on HTTPServer2
So I dont think its a EGRESS issue.
The Data location is US-east-1 for GCP and the pod local timezone is UTC, I am not sure if its a NTP sync issue. Need Advice. Same Code runs fine from my local with the same serviceaccount key. ( Just to mention I have done a kubectl cp to move the serviceaccount.json to the pod for testing later I'll create a configmap or something)