AmazonS3 error while downloading the file using java - amazon-web-services

I am getting the following exception while trying to download a file from AmazonS3 bucket. This is working fine from Java IDE but when I run the same code from other java based product it is failing. I have loaded all the client needed jar files,
VENDOR_JAR=/u01/app/sterling/jar/amazons3/1_0/aws-java-sdk-core-1.11.250.jar
VENDOR_JAR=/u01/app/sterling/jar/amazons3/1_0/aws-java-sdk-s3-1.11.250.jar
VENDOR_JAR=/u01/app/sterling/jar/amazons3/1_0/httpclient-4.5.2.jar
VENDOR_JAR=/u01/app/sterling/jar/amazons3/1_0/httpcore-4.4.4.jar
VENDOR_JAR=/u01/app/sterling/jar/amazons3/1_0/jackson-annotations-2.6.0.jar
VENDOR_JAR=/u01/app/sterling/jar/amazons3/1_0/jackson-core-2.6.7.jar
VENDOR_JAR=/u01/app/sterling/jar/amazons3/1_0/jackson-databind-2.6.7.1.jar
VENDOR_JAR=/u01/app/sterling/jar/amazons3/1_0/joda-time-2.8.1.jar
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1138)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1165)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1149)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
at com.amazonaws.http.conn.$Proxy15.connect(Unknown Source)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1236)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
... 24 more
Java Code:
AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
AmazonS3 s3Client = new AmazonS3Client(credentials);
System.out.println("Initialized Client Finally!!");
S3Object object = s3Client.getObject(new GetObjectRequest(bucketName, fileName));
displayTextInputStream(object.getObjectContent());
public void displayTextInputStream(InputStream input)throws IOException {
System.out.println("Inside displayTextInputStream()..........");
// Read one text line at a time and display.
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
while (true) {
String line = reader.readLine();
if (line == null) break;
System.out.println(" " + line);
}
System.out.println();
}

Related

From remote machine getting connection refused for GCP Bigquery

I am running a sample code from google to get a simple select query. Which is working fine in my local but from my k8s environment I am getting the below error
Exception in thread "main" com.google.cloud.bigquery.BigQueryException: Error getting access token for service account: connect timed out
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.create(HttpBigQueryRpc.java:220)
at com.google.cloud.bigquery.BigQueryImpl$5.call(BigQueryImpl.java:369)
at com.google.cloud.bigquery.BigQueryImpl$5.call(BigQueryImpl.java:366)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:365)
at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:340)
at com.rakuten.dps.dataplatform.ingest.utility.BQ_test.main(BQ_test.java:67)
Caused by: java.io.IOException: Error getting access token for service account: connect timed out
at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:444)
at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:157)
at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:145)
at com.google.auth.oauth2.ServiceAccountCredentials.getRequestMetadata(ServiceAccountCredentials.java:603)
at com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:91)
at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:159)
at com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:88)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:422)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.create(HttpBigQueryRpc.java:218)
... 8 more
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:284)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1340)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1315)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:264)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:113)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
at com.google.auth.oauth2.ServiceAccountCredentials.refreshAccessToken(ServiceAccountCredentials.java:441)
... 19 more
Below is the sample Code:
public class BQ_test {
private static final Logger logger = LoggerFactory.getLogger(BQ_test.class);
public static void main(String[] args) {
Job queryJob = null;
String actualValue = "";
NetHttpTransport transport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
String query = "SELECT * FROM `iconic-parsec-315409.bookmark_BQ.sbm_item_tbl``";
String projectId = "iconic-parsec-315409";
File credentialsPath = new File("/tmp/iconic-parsec-315409-823ef1c38a9d.json");
GoogleCredentials credentials;
try {
FileInputStream serviceAccountStream = new FileInputStream(credentialsPath);
credentials = ServiceAccountCredentials.fromStream(serviceAccountStream);
if (credentials.createScopedRequired()) {
Collection<String> bigqueryScopes = BigqueryScopes.all();
credentials = credentials.createScoped(bigqueryScopes);
}
BigQuery bigquery = BigQueryOptions
.newBuilder()
.setCredentials(credentials)
.setProjectId(projectId)
.build()
.getService();
QueryJobConfiguration queryConfig =
QueryJobConfiguration.newBuilder(query)
.setUseLegacySql(false)
.setJobTimeoutMs(180000L)
.build();
// Create a job ID so that we can safely retry.
JobId jobId = JobId.of(UUID.randomUUID().toString());
queryJob = bigquery.create(JobInfo.newBuilder(queryConfig).setJobId(jobId).build());
// Wait for the query to complete.
queryJob = queryJob.waitFor();
} catch (IOException | InterruptedException e) {
e.printStackTrace();
}
// Check for errors
if (queryJob == null) {
throw new RuntimeException("Job no longer exists");
} else if (queryJob.getStatus().getError() != null) {
// You can also look at queryJob.getStatus().getExecutionErrors() for all
// errors, not just the latest one.
throw new RuntimeException(queryJob.getStatus().getError().toString());
}
// Get the results.
TableResult result = null;
try {
result = queryJob.getQueryResults();
// Print all pages of the results.
// writeFvLToOrcFile(result,"/Users/susanta.a.adhikary/Downloads/test.orc");
for (FieldValueList row : result.iterateAll()) {
// String type
actualValue = row.get("sbm_item_id").getStringValue();
System.out.println(actualValue);
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
I tried curl -I "https://oauth2.googleapis.com/token" from my remote k8s pod and getting
HTTP/2 404
content-type: text/html
date: Sun, 04 Jul 2021 05:54:09 GMT
server: scaffolding on HTTPServer2
So I dont think its a EGRESS issue.
The Data location is US-east-1 for GCP and the pod local timezone is UTC, I am not sure if its a NTP sync issue. Need Advice. Same Code runs fine from my local with the same serviceaccount key. ( Just to mention I have done a kubectl cp to move the serviceaccount.json to the pod for testing later I'll create a configmap or something)

Zuul not throwing expected exception

I am throwing a ZuulRuntimeException and written a test case to validate it but it is throwing a different one.
Code:
public String extractCdsid(String accessToken) {
ObjectMapper objectMapper = new ObjectMapper();
String cdsid = "";
try {
String payload = JwtHelper.decode(accessToken).getClaims();
cdsid = objectMapper.readTree(payload).get("upn").textValue();
} catch (NullPointerException e) {
ZuulException ex = new ZuulException(
"Token does not contain upn. Please send a valid that contains upn(cdsid)", 401, "");
throw new ZuulRuntimeException(ex);
} }
Testcase:
#Test(expected = ZuulRuntimeException.class)
public void getPrincipalHeader_withAuthorizationHeaderNotContainUpn_returnsException(){
// mock
MockHttpServletRequest req = new MockHttpServletRequest();
req.addHeader("Authorization", "sendingtokenthatdoesnotcontainupn");
// run
tokenParser.getPrincipalHeader(req);
}
I am getting
java.lang.Exception: Unexpected exception, expected<com.netflix.zuul.exception.ZuulException> but was<java.lang.IllegalStateException>
at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:28)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:678)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: java.lang.IllegalStateException: CounterFactory not initialized
at com.netflix.zuul.monitoring.CounterFactory.instance(CounterFactory.java:42)
at com.netflix.zuul.exception.ZuulException.incrementCounter(ZuulException.java:73)
at com.netflix.zuul.exception.ZuulException.<init>(ZuulException.java:54)
at com.cv.vadr.vadrgateway.util.TokenParser.extractCdsid(TokenParser.java:51)
at com.cv.vadr.vadrgateway.util.TokenParser.getPrincipalHeader(TokenParser.java:32)
at com.cv.vadr.vadrgateway.util.TokenParserTest.getPrincipalHeader_withAuthorizationHeaderNotContainUpn_returnsException(TokenParserTest.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19)
... 16 more
I see the correct exception being thrown with message when I hit the service through postman
I had the same problem and for me adding
#Before
public void setUp() {
CounterFactory.initialize(new EmptyCounterFactory());
}
worked. Thanks to that I get rid of the java.lang.IllegalStateException: CounterFactory not initialized

I am new to JUnit Testing.I am unable to write successful test cases and the test Case I create is returning a nullpointer exception

I am trying to write test cases for an endpoint controller for the following controllers
#RestController
#RequestMapping("/agent/")
public class AgentRestController extends BaseRestController {
#RequestMapping(value = "/log/{revisionId}/", method = RequestMethod.PUT)
public ResponseEntity<?> log(#PathVariable("revisionId") String revisionId, #RequestBody LogMessage message) {
ConfigurationRevision cr = configurationRevisionRepository.findOne(revisionId);
BackupLog log = new BackupLog(message);
log.setRevision(cr);
backupLogRepository.save(log);
return new ResponseEntity<>(HttpStatus.CREATED);
}
#RequestMapping("/config")
#ResponseBody
public AgentConfig getConfiguration(HttpServletRequest request) {
AgentConfig cfg = getAgentConfigByIP(request.getRemoteAddr());
LogMessage log = new LogMessage();
log.level = LogLevel.INFO;
log.date = System.currentTimeMillis();
log.msg = "SYSTEM: Configuration retrieved by client";
// log the configuration retrieval
log(cfg.id, log);
return cfg;
}
}
I am trying to write Junit Test cases for these controllers
#RunWith(SpringJUnit4ClassRunner.class)
#SpringApplicationConfiguration(classes = { Application.class })
#WebAppConfiguration
public class ApiControllerTest {
private MockMvc mockMvc;
#Autowired
private WebApplicationContext webApplicationContext;
#Before
public void setup() throws Exception {
this.mockMvc = webAppContextSetup(webApplicationContext).build();
}
#Test
public void getConfigurationSuccess() throws Exception {
mockMvc.perform(get("/agent/config").accept(MediaType.APPLICATION_JSON)).andExpect(status().is2xxSuccessful());
}
}
This is the stacktrace
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.NullPointerException
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:980)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:859)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:844)
at org.springframework.test.web.servlet.TestDispatcherServlet.service(TestDispatcherServlet.java:65)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.springframework.mock.web.MockFilterChain$ServletFilterProxy.doFilter(MockFilterChain.java:167)
at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:134)
at org.springframework.test.web.servlet.MockMvc.perform(MockMvc.java:155)
at backup.itests.ServiceTest.getConfigurationSuccess(ServiceTest.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:254)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:89)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:193)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:678)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: java.lang.NullPointerException
at backup.restservice.BaseRestController.getAgentConfigByIP(BaseRestController.java:25)
at backup.restservice.AgentRestController.getConfiguration(AgentRestController.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:817)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:731)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:968)
... 39 more
Can Someone help me in writing JUnit or JBehave testcases for the the given two controller and Expanding the test case for the controllers in AgenRestController. This would be a Great and a constructive help.
private WebApplicationContext webApplicationContext;
#Before
public void setup() throws Exception {
this.mockMvc = webAppContextSetup(webApplicationContext).build();
}
Just a shot in the dark here, but webApplicationContext is null, and not initialized. Perhaps before the this.mockMcx = ... you might insert the line:
webApplicationContext = new WebApplicationContext();

Amazon S3 Error Code: 400 while running mr-job on EMR

Got this error running a custom jar on EMR.
Exception in thread "main" com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: B042BB0B40A75966), S3 Extended Request ID: vr/DUr8HD3xjomauyzqvVdGuW3fHBP8PDUmTIAoVLUxrmsxh9H+OS9+cgo4OmHxaz/b8CSPGmuc=
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1389)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:902)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:607)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3826)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1071)
at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1029)
at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.ensureBucketExists(Jets3tNativeFileSystemStore.java:138)
at com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.initialize(Jets3tNativeFileSystemStore.java:116)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy23.initialize(Unknown Source)
at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.initialize(S3NativeFileSystem.java:461)
at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize(EmrFileSystem.java:110)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2737)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2719)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:375)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:485)
at SentimentsDriver.run(SentimentsDriver.java:21)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at SentimentsDriver.main(SentimentsDriver.java:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
x
arguments:
s3://sentimentproj/input/tweet.txt
s3://sentimentproj/output
SentimentsDriver:
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
/*This class is responsible for running map reduce job*/
public class SentimentsDriver extends Configured implements Tool{
public int run(String[] args) throws Exception{
if(args.length !=2) {
System.err.println("Usage: MaxTemperatureDriver <input path> <outputpath>");
System.exit(-1);
}
#SuppressWarnings("deprecation")
Job job = new Job();
job.setJarByClass(SentimentsDriver.class);
job.setJobName("SentimentAnalysis");
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job,new Path(args[1]));
job.setMapperClass(SentimentsMapper.class);
job.setReducerClass(SentimentsReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
System.exit(job.waitForCompletion(true) ? 0:1);
boolean success = job.waitForCompletion(true);
return success ? 0 : 1;
}
public static void main(String[] args) throws Exception {
SentimentsDriver driver = new SentimentsDriver();
int exitCode = ToolRunner.run(driver, args);
System.exit(exitCode);
}
}
The issue may be with the S3 bucket. Make sure to keep the S3 bucket region in same Region of your cluster while creating bucket. If it will be different then it usually throw the Bad Request 400 error.

Upgrade from HTTP/1.0 to HTTP/1.1 at server side, SOAP messages >4kb throws '400: Bad Request' Error

I'm consuming a webservice through a WSDL. I generated Java classes using Apache CXF wsdl2java.
For my soap messages >4kb, I get the below error. Messages <4kb has no problem getting response whatsoever. I tried disabling CHUNKING as follows:
soapMessageContext.put(HTTPConstants.CHUNKED, Boolean.FALSE);
I tried adding Content-Length to the HTTP header explicitly so that it disables chunking if its so.
Map<String, List<String>> httpHeaders = new HashMap<String,List<String>>();
httpHeaders.put("Content-Length", Collections.singletonList(filelength));
Map<String, Object> reqContext = ((BindingProvider) port).getRequestContext();
reqContext.put(MessageContext.HTTP_REQUEST_HEADERS, httpHeaders);
repairOrderResponse = port.submitRepairOrder(security,
repairorders);
Please help me through this. I cannot figure out why.
Stacktrace:
javax.xml.ws.WebServiceException: Could not send Message.
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:146)
at com.sun.proxy.$Proxy80.submitRepairOrder(Unknown Source)
at com.ssss.ssjdecommonws.webservices.SSJDECommonService.submitOrder(SSJDECommonService.java:640)
at com.ssss.ssjdecommonws.webservices.SSJDECommonService.getRepairOrderResult(SSJDECommonService.java:529)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167)
at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:269)
at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:227)
at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:216)
at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:542)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:524)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:126)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:559)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:340)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:353)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:911)
at org.apache.tomcat.util.net.NioEndpoint$ChannelProcessor.run(NioEndpoint.java:920)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.cxf.transport.http.HTTPException: HTTP response '400: Bad Request' when communicating with <MY_ENDPOINT_COMES_HERE>
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1549)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1504)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1310)
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:628)
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:565)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:474)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:377)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:330)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:135)
... 34 more
This is how I call the service in submitOrder()
SubmitRepairOrderService sService = new SubmitRepairOrderService(u, SERVICE_NAME);
SubmitRepairOrderType port = sService.getSubmitRepairOrderPort();
List<Handler> new_handlerChain = new ArrayList<Handler>();
fdda = new MessageHandler(props);
new_handlerChain.add(fdda);
((BindingProvider) port).getBinding().setHandlerChain(new_handlerChain);
security = null;
repairOrderResponse = port.submitRepairOrder(security,repairorders);
My handler method is given below:
public boolean handleMessage(SOAPMessageContext messageContext) {
try {
SOAPHeader header = null;
boolean outMessageIndicator = (Boolean) messageContext
.get(MessageContext.MESSAGE_OUTBOUND_PROPERTY);
if (outMessageIndicator) {
SOAPEnvelope envelope = messageContext.getMessage().getSOAPPart().getEnvelope();
if (envelope.getHeader() != null) {
header = envelope.getHeader();
}
SOAPElement security = header.addChildElement("Security","oas","http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd");
SOAPElement usernameToken = security.addChildElement("UsernameToken", "oas");
usernameToken.addAttribute(new QName("xmlns:wsu"),
"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd");
SOAPElement username = usernameToken.addChildElement("Username", "oas");
username.addTextNode(userId);
SOAPElement password = usernameToken.addChildElement("Password", "oas");
password.setAttribute("Type",
"http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText");
password.addTextNode(passcode);
}
} catch (Exception ex) {
throw new WebServiceException(ex);
}
return true;
}
Could you please let me know what other code you would want to see to understand it better?
You can disable chunking on JBoss using below code.
Add this code right after creating port object.
SubmitRepairOrderType port = sService.getSubmitRepairOrderPort();
HTTPClientPolicy hTTPClientPolicy = new HTTPClientPolicy();
hTTPClientPolicy.setAllowChunking(false);
   
Client client =  org.apache.cxf.jaxws.JaxWsClientProxy.getClient( port );
HTTPConduit http = ( HTTPConduit ) client.getConduit();
 
http.setClient(hTTPClientPolicy);
You may need the following jar files in your classpath: