Spring WebFlux project that connects to ElasticSearch Unit Testing - unit-testing

I am currently working on a Spring WebFlux project that connects to ElasticSearch. I have a Rest Service that in turn calls a method in the Service Layer that connects to the ES. I am having trouble writing UnitTests for my Service Layer. Any help would be appreciated as this is the first time I am working with Reactive Programming. Below are the code snippets for my Controller and Service methods.
Controller Code :
#GetMapping(path = "/api/apis/services/{id}", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
Flux<ClassA> serviceApis(#PathVariable final String serviceKey) {
return apiService.getDataForService(serviceKey);
}
Service Layer :
#PreAuthorize("isFullyAuthenticated()")
public Flux<ClassA> getDataForService(
final String id) {
IdentityToken token = GSLSecurityContext.getCurrentUserIdentityToken();
if (token == null) {
return Flux.error(new Exception("No token found"));
}
String securityQueryJson = getSecurityShould(token);
String queryToRun = QUERY
.replace("XXX_SIZE_XXX", config.getValueAsString("scroll.size"))
.replace("XXX_SECURITY_SHOULD_XXX", securityQueryJson)
.replace("XXX_SERVICE_KEY_XXX", id);
WebClient client = ClientUtil.getDataLakeWebClient(config);
Flux<ClassA> response = getData(client, queryToRun);
return response;
}
The getData code is as below :
protected Flux<ClassA> getData(
final WebClient client,
final String queryToRun) {
String scrollTimeoutQuery = "?scroll=" + config.getValueAsString("scroll.timeout");
long timeout = config.getValueAsLong("query.timout");
return Flux.generate(
() -> null,
(scrollId, sink) -> {
ClassAWrapper lastWrapper = null;
if (scrollId == null) {
Mono<ClassAWrapper> wrapper = client.post()
.uri(getSearchURI() + scrollTimeoutQuery)
.body(BodyInserters.fromObject(queryToRun)).retrieve()
.bodyToMono(ClassAWrapper.class)
.onErrorMap(original -> new Exception("Unable to retrieve from elastic search for query " + queryToRun, original))
.log();
try {
lastWrapper = wrapper.block(Duration.ofSeconds(timeout));
} catch (IllegalStateException ex) {
LOG.error("Timedout after " + timeout + " seconds while getting data from elastic search for query " + queryToRun);
lastWrapper = null;
} catch (Exception ex) {
LOG.error("Error in getting message details",ex);
lastWrapper = null;
}
} else {
String scrollQuery = "{\"scroll\" : \"" + config.getValueAsString("scroll.timeout") + "\", \"scroll_id\" : \"" + scrollId + "\"}";
Mono<ClassAWrapper> wrapper = client.post()
.uri("_search/scroll")
.body(BodyInserters.fromObject(scrollQuery)).retrieve()
.bodyToMono(ClassAWrapper.class)
.onErrorMap(original -> new Exception("Unable to retrieve next page of data from elastic search", original))
.log();
try {
lastWrapper = wrapper.block(Duration.ofSeconds(timeout));
} catch (IllegalStateException ex) {
LOG.error("Timeout after " + timeout + " seconds while getting data from elastic search for query " + queryToRun);
lastWrapper = null;
} catch (Exception ex) {
LOG.error("Error in getting message details",ex);
lastWrapper = null;
}
}
if (lastWrapper == null || lastWrapper.getResult() == null || lastWrapper.getResult().getDetails().isEmpty()) {
sink.next(new ClassA());
sink.complete();
return null;
}
sink.next(lastWrapper.getResult());
return lastWrapper.getScrollId();
}
);
}
Here, queryToRun is the ES query to be executed. config is the configuration. I need to test the method "getDataForService()".

Related

S3 bukcet getobject throws error Method not found: 'Amazon.Runtime.Internal.Transform.UnmarshallerContext

I am tring to read my s3 bucket object from EKS cluster application method as listed below.
[HttpGet("ReadFromS3")]
[ProducesResponseType(typeof(ApiResponse<string>), 200)]
[ProducesResponseType(500)]
public async Task<String> ReadFromS3()
{
AmazonS3Config config = new AmazonS3Config { RegionEndpoint = RegionEndpoint.EUWest2 };
string aws_tokenFile = Environment.GetEnvironmentVariable("AWS_WEB_IDENTITY_TOKEN_FILE");
string aws_rolw_arn = Environment.GetEnvironmentVariable("AWS_ROLE_ARN");
string logs = string.Empty;
using (var stsClient = new AmazonSecurityTokenServiceClient(new AnonymousAWSCredentials()))
{
logs += "AmazonSecurityTokenServiceClient";
var assumeRoleResult = await stsClient.AssumeRoleWithWebIdentityAsync(new AssumeRoleWithWebIdentityRequest
{
WebIdentityToken = System.IO.File.ReadAllText(Environment.GetEnvironmentVariable("AWS_WEB_IDENTITY_TOKEN_FILE")),
RoleArn = Environment.GetEnvironmentVariable("AWS_ROLE_ARN"),
RoleSessionName = "S3Session",
DurationSeconds = 900
});
if (assumeRoleResult != null)
{
logs += " -> assumeRoleResult";
var credentials = assumeRoleResult.Credentials;
var basicCredentials = new BasicAWSCredentials(credentials.AccessKeyId,
credentials.SecretAccessKey);
logs += " -> BasicAWSCredentials";
using (client = new AmazonS3Client(basicCredentials, RegionEndpoint.EUWest2) )
{
logs += " -> AmazonS3Client";
try
{
GetObjectRequest request = new GetObjectRequest
{
BucketName = bucketName,
Key = keyName
};
GetObjectResponse response = await client.GetObjectAsync(request);
logs += " -> GetObjectAsync";
Stream responseStream = response.ResponseStream;
StreamReader reader = new StreamReader(responseStream);
logs += " -> ResponseStream";
return await reader.ReadToEndAsync();
}
catch (Exception ex)
{
return "Exception in ReadFromS3 :" + ex.Message + logs;
}
}
}
else
{
return "Unable to Connect";
}
}
}
I am getting error at line GetObjectResponse response = await client.GetObjectAsync(request);
Method not found: 'Amazon.Runtime.Internal.Transform.UnmarshallerContext Amazon.Runtime.Internal.Transform.ResponseUnmarshaller.CreateContext

Multipart upload using AWS Java SDK hangs at 99%

We are trying to upload a file of size 46GB to AWS S3. However the file upload hangs once it reaches 99%. We are using AWS java SDK for the mulitpart upload.
Following is the code which is used for multipart upload:
void startMultipartUpload(ObjectStoreAccess creds, List<Path> files) {
List<File> filenames = files.stream().map(Path::toString).map(File::new).collect(Collectors.toList());
TransferManager transferManager = transferManagerFactory.createTransferManager(creds);
List<File> filesNotUploaded = new ArrayList<>();
boolean isFileUploadSuccessful;
Integer timeElapsed;
for (File file : filenames) {
isFileUploadSuccessful = false;
timeElapsed = 0;
try {
String keyName = creds.getAwsS3TemporaryUploadCredentials().getKeyPrefix() + file.getName();
PutObjectRequest request = new PutObjectRequest(creds.getAwsS3TemporaryUploadCredentials().getBucketName(), keyName, new File(file.getPath()));
Upload upload = transferManager.upload(request);
logger.info(String.format("Starting upload for : %s ", file.getName()));
while (!upload.getState().equals(Transfer.TransferState.Completed)) {
Thread.sleep(1000);
timeElapsed++;
progressLogger.writeProgress(timeElapsed, upload.getProgress().getPercentTransferred());
}
upload.waitForCompletion();
isFileUploadSuccessful = true;
progressLogger.writeProgressStatus("...upload complete!\n");
} catch (AmazonServiceException e) {
String message = "AmazonServiceException: " + e.getMessage();
logger.error(message);
} catch (SdkClientException e) {
String message = "SdkClientException: " + e.getMessage();
logger.error(message);
} catch (InterruptedException e) {
String message = "InterruptedException: " + e.getMessage();
logger.error(message);
Thread.currentThread().interrupt();
} finally {
if (!isFileUploadSuccessful) {
String message = this.appMessages.getMessageByKey("FAIL_TO_UPLOAD_FILE") + " " + file.getPath();
logger.error(message);
filesNotUploaded.add(file);
}
}
}
}
Try using the AWS SDK for Java V2 and following this example that shows how to upload an object in parts. See:
https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/javav2/example_code/s3/src/main/java/com/example/s3/S3ObjectOperations.java

I am getting a 401 error when I am sending a soap request to a nav web service

I am trying to send an XML soap request to a dynamics nav web service. This is the XML from the WSDL. I have created a web access key and its the one in the key parameter of the XML.
<s11:Envelope xmlns:s11='http://schemas.xmlsoap.org/soap/envelope/'>
<s11:Body>
<ns1:Create xmlns:ns1='urn:microsoft-dynamics-schemas/page/customerws'>
<ns1:CustomerWS>
<ns1:Key>+gn8Nu4i7iW7D/g9vCaI8HZE5IEi1NBkTBqDp5QfXe4=</ns1:Key>
<ns1:Shipping_Advice></ns1:Shipping_Advice>
<ns1:Shipment_Method_Code></ns1:Shipment_Method_Code>
<ns1:Shipping_Agent_Code></ns1:Shipping_Agent_Code>
<ns1:Shipping_Agent_Service_Code></ns1:Shipping_Agent_Service_Code>
<ns1:Shipping_Time></ns1:Shipping_Time>
<ns1:Base_Calendar_Code></ns1:Base_Calendar_Code>
<ns1:Customized_Calendar></ns1:Customized_Calendar>
<ns1:Currency_Code></ns1:Currency_Code>
<ns1:Language_Code></ns1:Language_Code>
<ns1:VAT_Registration_No></ns1:VAT_Registration_No>
</ns1:CustomerWS>
</ns1:Create>
</s11:Body>
</s11:Envelope>
And this is the code that am using to send this request:
Console.WriteLine("We have started");
string pageName = "http://hrp-dmu.uganda.hrpsolutions.co.ug:9047/DynamicsNAV80/WS/Uganda%20Management%20Institute/Page/CustomerWS";
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(pageName);
req.Method = "POST";
req.ContentType = "text/xml;charset=UTF-8";
req.ProtocolVersion = new Version(1, 1);
req.Headers.Add("SOAPAction", #"urn:microsoftdynamicsschemas/page/customerws:Create");
Console.WriteLine("After preparing request object");
string xmlRequest = GetTextFromXMLFile("E:\\tst3.xml");
Console.WriteLine("xml request : "+xmlRequest);
byte[] reqBytes = new UTF8Encoding().GetBytes(xmlRequest);
req.ContentLength = reqBytes.Length;
try
{
using (Stream reqStream = req.GetRequestStream())
{
reqStream.Write(reqBytes, 0, reqBytes.Length);
}
}
catch (Exception ex)
{
Console.WriteLine("GetRequestStreamException : " + ex.Message);
}
HttpWebResponse resp = null;
try
{
resp = (HttpWebResponse)req.GetResponse();
}
catch (Exception exc)
{
Console.WriteLine("GetResponseException : " + exc.Message);
}
string xmlResponse = null;
if (resp == null)
{
Console.WriteLine("Null response");
}
else
{
using (StreamReader sr = new StreamReader(resp.GetResponseStream()))
{
xmlResponse = sr.ReadToEnd();
}
Console.WriteLine("The response");
Console.WriteLine(xmlResponse);
}
Console.ReadKey();
when using NavUserPassword Authentication you'll need a certificate.
See here on MSDN
Cheers!

Cannot genetrate java client for file upload webservice

I have a simple file upload web service as a small part of my project.
This is what I have done so far on the server side :
#POST
#Path("/file")
#Consumes(MediaType.MULTIPART_FORM_DATA)
public Response uploadFile(List<Attachment> attachments,#Context HttpServletRequest request) {
System.out.println("Got an attachment!");
for(Attachment attr : attachments) {
DataHandler handler = attr.getDataHandler();
try {
InputStream stream = handler.getInputStream();
MultivaluedMap map = attr.getHeaders();
OutputStream out = new FileOutputStream(new File("/home/yashdosi/s/" + getFileName(map))); //getFileName is a seperate private function..
int read = 0;
byte[] bytes = new byte[1024];
while ((read = stream.read(bytes)) != -1) {
out.write(bytes, 0, read);
}
stream.close();
out.flush();
out.close();
} catch(Exception e) {
e.printStackTrace();
}
}
return Response.ok("file uploaded").build();
}
It works perfectly well when requests come from html forms...when I try to send a request from a java client it simply doesnt work..!!
Any ideas about on creating a java client for this code..
Here is the code I tried with...maybe there is a simple error in this code but..I dont see it...also as I said this code simple wont work...no errors or anything else....when I tried printing something on the server console to see if the service is invoked...it did NOT print anything..so I think I am unable to contact the service for some reason...
public static void uploadPhoto()
{
String url = "http://localhost:8080/fileupload-ws/services/postdata";
String output = null;
PostMethod mPost = new PostMethod(url);
HttpClient client = new HttpClient();
try
{
File imageFile = new File("/home/yashdosi/1.jpg");
BufferedImage image = ImageIO.read(imageFile);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ImageIO.write(image, "jpg", baos);
byte[] encodedImage = Base64.encodeBase64(baos.toByteArray());
String data = " " + " " + "" + "image/jpeg" + " " + "" + new String(encodedImage) + " " + "";
mPost.setRequestBody(data);
mPost.setRequestHeader("Content-Type", "text/xml");
client.executeMethod( mPost );
output = mPost.getResponseBodyAsString( );
mPost.releaseConnection( );
} catch (HttpException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(output);
}
Finally got a client working!!
HttpClient httpclient = new DefaultHttpClient();
try {
HttpPost httppost = new HttpPost("http://localhost:8080/fileupload-ws/services/postdata");
FileBody img = new FileBody(new File("/home/yashdosi/1.jpg"));
FileBody html = new FileBody(new File("/home/yashdosi/hotmail.html"));
MultipartEntity reqEntity = new MultipartEntity();
reqEntity.addPart("image", img);
reqEntity.addPart("html", html);
httppost.setEntity(reqEntity);
httppost.setHeader("Content-Type", "multipart/form-data");
System.out.println("executing request " + httppost.getRequestLine());
HttpResponse response = httpclient.execute(httppost);
HttpEntity resEntity = response.getEntity();
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
if (resEntity != null) {
System.out.println("Response content length: " + resEntity.getContentLength());
}
EntityUtils.consume(resEntity);
}
catch(Exception e)
{
e.printStackTrace();
}
finally {
try { httpclient.getConnectionManager().shutdown(); } catch (Exception ignore) {}
}

Salesforce Webservice Error

I am receiving the following error from a class that invokes a webservice.
"You have uncommitte​d work pending. Please commit or rollback before calling out"
Here is the class that is calling the webservice:
global class myWS
{
WebService static string invokeExternalWs(string childValue, string parentValue)
{
HttpRequest req = new HttpRequest();
req.setEndpoint('https://externalURL/Services');
req.setMethod('POST');
req.setHeader('Content-Type', 'text/xml; charset=utf-8');
req.setHeader('SOAPAction', 'http://externalService/externalMethod');
string b = '--soap request goes here--';
req.setBody(b);
Http http = new Http();
try {
//Execute web service call here
String xObjectID ='';
HTTPResponse res = http.send(req);
Dom.Document doc = res.getBodyDocument();
String soapNS = 'http://schemas.xmlsoap.org/soap/envelope/';
Dom.XmlNode root = doc.getRootElement();
for(dom.XmlNode node1 : root.getChildElements()) {
for(dom.XmlNode node2 : node1.getChildElements()) {
for(dom.XmlNode node3 : node2.getChildElements()) {
for(dom.XmlNode node4 : node3.getChildElements()) {
xObjectID = node4.getText();
}
}
}
}
return xObjectID;
} catch(System.CalloutException e){
return 'ERROR:' + e;
}
}
}
UPDATE: Here is my class that is executing myWS
public void applyURLString(ID ArgBuildID) {
Builder__c current_build = [SELECT id, name, LLURL__c, column1, column2, Opportunity__c
FROM Builder__c
WHERE id = :ArgBuildID];
if(current_build.LLURL__c == null || current_build.LLURL__c.trim().length() == 0)
{
String tmpFolderName = current_build.column1 + ' - ' + current_build.column2;
String LLWSResultPattern = '[0-9]{2,}';
String myWSXMLResult = myWS.invokeExternalWs(tmpFolderName,'test');
Boolean LLWSPatternMatched = pattern.matches(LLWSResultPattern,myWSXMLResult);
if(LLWSPatternMatched)
{
Opportunity oppt = [SELECT Id,Name
FROM Opportunity
WHERE Id = :current_build.Opportunity__c
LIMIT 1];
oppt.LLURL__c = 'https://someService/' + myWSXMLResult;
update oppt;
}
}
}
UPDATE #2 - Here is where applyURLString() is executed. This is the only place that DML is executed prior to my HTTP request. Yet I need the ID of the new Builder record.
Builder__c insertBuild = new Builder__c();
insertBuild.Opportunity__c = opportunityId;
insertBuild.Product_Group__c = selectedBuild.Product_Group__c;
insertBuild.Manual_Build_Product__c = selectedBuild.Manual_Build_Product__c;
insert insertBuild;
applyURLString(insertBuild.Id);
Any ideas why this error would occur?
#JCD's suggestion to use an #future annotation solved my problem. Thanks again!