We are trying to upload a file of size 46GB to AWS S3. However the file upload hangs once it reaches 99%. We are using AWS java SDK for the mulitpart upload.
Following is the code which is used for multipart upload:
void startMultipartUpload(ObjectStoreAccess creds, List<Path> files) {
List<File> filenames = files.stream().map(Path::toString).map(File::new).collect(Collectors.toList());
TransferManager transferManager = transferManagerFactory.createTransferManager(creds);
List<File> filesNotUploaded = new ArrayList<>();
boolean isFileUploadSuccessful;
Integer timeElapsed;
for (File file : filenames) {
isFileUploadSuccessful = false;
timeElapsed = 0;
try {
String keyName = creds.getAwsS3TemporaryUploadCredentials().getKeyPrefix() + file.getName();
PutObjectRequest request = new PutObjectRequest(creds.getAwsS3TemporaryUploadCredentials().getBucketName(), keyName, new File(file.getPath()));
Upload upload = transferManager.upload(request);
logger.info(String.format("Starting upload for : %s ", file.getName()));
while (!upload.getState().equals(Transfer.TransferState.Completed)) {
Thread.sleep(1000);
timeElapsed++;
progressLogger.writeProgress(timeElapsed, upload.getProgress().getPercentTransferred());
}
upload.waitForCompletion();
isFileUploadSuccessful = true;
progressLogger.writeProgressStatus("...upload complete!\n");
} catch (AmazonServiceException e) {
String message = "AmazonServiceException: " + e.getMessage();
logger.error(message);
} catch (SdkClientException e) {
String message = "SdkClientException: " + e.getMessage();
logger.error(message);
} catch (InterruptedException e) {
String message = "InterruptedException: " + e.getMessage();
logger.error(message);
Thread.currentThread().interrupt();
} finally {
if (!isFileUploadSuccessful) {
String message = this.appMessages.getMessageByKey("FAIL_TO_UPLOAD_FILE") + " " + file.getPath();
logger.error(message);
filesNotUploaded.add(file);
}
}
}
}
Try using the AWS SDK for Java V2 and following this example that shows how to upload an object in parts. See:
https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/javav2/example_code/s3/src/main/java/com/example/s3/S3ObjectOperations.java
Related
I'm downloading a file (zip) from a bucket in my aws s3 using a java code & aws sdk.
However, the downloaded file is corrupted.
Downloading the file manually works.
I compared the content of the files and notices that the corrupted file contains kind of uncoded chars
see this:
The code that is used to download is the following:
public boolean downloadFile(String bucketName, String fileNameOnServer, String localFileName )
{
S3Object object =null;
InputStream objectData =null;
try
{
object = s3.getObject(new GetObjectRequest(bucketName, fileNameOnServer));
objectData = object.getObjectContent();
}
catch(Exception e)
{
LoggingService.writeToLog("Error001 downloading file "+bucketName+"/"+fileNameOnServer+" to "+localFileName, e, LogModule.CommonUtils,LogLevel.Error);
return false;
}
try
{
FileWriter fw = new FileWriter(localFileName, true);
BufferedWriter bw = new BufferedWriter(fw);
try
{
BufferedReader reader = new BufferedReader(new InputStreamReader(object.getObjectContent()));
String line;
while( (line = reader.readLine() ) !=null)
{
bw.write(line);
bw.newLine();
}
LoggingService.writeToLog("file from "+bucketName+"/"+fileNameOnServer+" "+" downloaded to "+bucketName + " successfully", LogModule.CommonUtils,LogLevel.Info);
return true;
}
catch(IOException e)
{
LoggingService.writeToLog("Error downloading file "+bucketName+"/"+fileNameOnServer+" to "+localFileName, e, LogModule.CommonUtils,LogLevel.Error);
return false;
}
finally
{
objectData.close();
bw.close();
object.close();
}
}
catch(IOException e)
{
LoggingService.writeToLog("Error opening local file "+localFileName+" for writing ", e, LogModule.CommonUtils,LogLevel.Error);
return false;
}
}
The documentation for InputStreamReader includes this:
An InputStreamReader is a bridge from byte streams to character streams: It reads bytes and decodes them into characters using a specified charset. The charset that it uses may be specified by name or may be given explicitly, or the platform's default charset may be accepted.
In other words, it attempts to treat the data as text data. For binary data like in a zip file, this will almost assuredly corrupt the data. Instead, if you read and write the bytes directly, you will pass them along without changing them:
package com.exampleapp.app;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.s3.model.S3ObjectInputStream;
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.IOException;
import java.io.OutputStream;
public class App
{
public static void main(String[] args)
{
downloadFile("example-bucket", "example-key.zip", "local-file.zip");
}
static AmazonS3 s3 = AmazonS3ClientBuilder.standard().withRegion(Regions.DEFAULT_REGION).build();
public static boolean downloadFile(String bucketName, String fileNameOnServer, String localFileName )
{
S3Object object = null;
InputStream objectData = null;
InputStream reader = null;
OutputStream writer = null;
try
{
object = s3.getObject(new GetObjectRequest(bucketName, fileNameOnServer));
objectData = object.getObjectContent();
}
catch(Exception e)
{
System.out.println("Error001 downloading file "+bucketName+"/"+fileNameOnServer+" to "+localFileName);
return false;
}
try
{
File file = new File(localFileName);
try
{
reader = new BufferedInputStream(object.getObjectContent());
writer = new BufferedOutputStream(new FileOutputStream(file));
int read = -1;
while ( ( read = reader.read() ) != -1 )
{
writer.write(read);
}
System.out.println("file from "+bucketName+"/"+fileNameOnServer+" "+" downloaded to "+bucketName + " successfully");
return true;
}
catch(IOException e)
{
System.out.println("Error downloading file "+bucketName+"/"+fileNameOnServer+" to "+localFileName);
return false;
}
finally
{
object.close();
writer.flush();
writer.close();
reader.close();
}
}
catch(IOException e)
{
System.out.println("Error opening local file "+localFileName+" for writing ");
return false;
}
}
}
The Photo Spring BOOT app now supports downloading an image. It works perfectly. You should use the AWS SDK for Java V2. Here is the example app:
When i open the downloaded image - it is valid and not corrupt:
The code to download this image from an Amazon S3 bucket is located in a Spring Controller:
#RequestMapping(value = "/downloadphoto", method = RequestMethod.GET)
void buildDynamicReportDownload(HttpServletRequest request, HttpServletResponse response) {
try {
//Get the photo object name
String photoKey = request.getParameter("photoKey");
byte[] photoBytes = s3Client.getObjectBytes("myphotobucket", photoKey) ;
InputStream is = new ByteArrayInputStream(photoBytes);
//define the required information to download the image
response.setContentType("image/png");
response.setHeader("Content-disposition", "attachment; filename="+photoKey);
org.apache.commons.io.IOUtils.copy(is, response.getOutputStream());
response.flushBuffer();
} catch (Exception e) {
e.printStackTrace();
}
}
The V2 S3 code is here. Notice it reads an object and returns a byte[].
public byte[] getObjectBytes (String bucketName, String keyName) {
s3 = getClient();
try {
// create a GetObjectRequest instance
GetObjectRequest objectRequest = GetObjectRequest
.builder()
.key(keyName)
.bucket(bucketName)
.build();
// get the byte[] from this AWS S3 object
ResponseBytes<GetObjectResponse> objectBytes = s3.getObjectAsBytes(objectRequest);
byte[] data = objectBytes.asByteArray();
return data;
} catch (S3Exception e) {
System.err.println(e.awsErrorDetails().errorMessage());
System.exit(1);
}
return null;
}
Finally the Javascript call
function DownloadImage(){
//Post the values to the controller
var photo = $('#photo').val();
window.location="../downloadphoto?photoKey=" + photo ;
}
This example will be updated so all of this code is in the example doc.
https://github.com/awsdocs/aws-doc-sdk-examples/tree/master/javav2/usecases/creating_photo_analyzer_app
Unable to Download a file from AWS S3 to the Downloads folder. However file is getting downloaded from AWS S3 server to the intermediate local server (E:\New**\E***\wwwroot\DownloadFiles). Now the issue is getting this file from (\DownloadFiles) this folder to the Downloads folder. Getting a error pop-up on attempting so.
return File(memory, tmpConType, Path.GetFileName(filePath));
Error Popup
Error-Popup-fromBrowser
Here is my full code for Download:
[HttpPost]
public async Task<ActionResult> DownloadFile(string fileName, string originalFilename, int clientId, int taskId)
{
using (LogContext.PushProperty("UserId", userId))
{
try
{
if (fileName != null && fileName != "")
{
var fName = fileName.Split("_")[0];
var s3Client = new AmazonS3Client(accesskey, secretkey, bucketRegion);
var s3companyFolder = "Company-" + companyId + "/" + "Client-" + clientId + "/" + "Job-" + taskId;
GetObjectRequest request = new GetObjectRequest
{
BucketName = bucketName + "/" + s3companyFolder,
Key = fName
};
using (GetObjectResponse response = await s3Client.GetObjectAsync(request))
{
using Stream responseStream = response.ResponseStream;
string tmpName = response.Metadata["x-amz-meta-filename"];
var tmpConType = response.Headers["Content-Type"];
string filePath = Path.Combine(hostingEnvironment.WebRootPath, "DownloadFiles");
filePath = Path.Combine(filePath, tmpName);
using FileStream outFile = System.IO.File.Create(filePath);
responseStream.CopyTo(outFile);
outFile.Dispose();
responseStream.Dispose();
var memory = new MemoryStream();
using (var stream = new FileStream(filePath, FileMode.Open))
{
await stream.CopyToAsync(memory);
}
if (System.IO.File.Exists(filePath))
System.IO.File.Delete(filePath);
memory.Position = 0;
return File(memory, tmpConType, Path.GetFileName(filePath));
}
}
return Json(new { success = true });
}
catch (Exception ex)
{
throw;
}
}
}
Thanks in Advance
I am uploading files to Amazon S3 bucket and that is working perfectly fine. Now I want to show the progress while uploading them, I have research everywhere but none of them is working. Below is the code for uploading file to Amazon S3 bucket-
On Spring boot controller-
private static final String UPLOAD_FILE_URL = "/uploadImages.htm";
#RequestMapping(value = UPLOAD_FILE_URL, method = RequestMethod.POST)
public ModelAndView uploadImagesRequest(#RequestParam("file") MultipartFile[] files, #RequestParam("filename") String title, HttpServletRequest request) {
ModelAndView model = new ModelAndView("index");
if(awsUploadImageInterface.uploadImage(filess, title, request)) {
model.addObject("successMsg", "Banner Image Is Added Successfully");
}
return "index";
}
Upload Image Functionality-
#Override
public boolean uploadImage(MultipartFile[] file, String title, HttpServletRequest request) {
boolean result = false;
S3BucketUtility s3client = new S3BucketUtility();
InputStream Is;
String key;
Properties prop = new Properties();
InputStream propstream = getClass().getClassLoader().getResourceAsStream("s3.properties");
try {
prop.load(propstream);
} catch (IOException e) {
System.out.println("Properties File Exception in AWS Connection Class: " + e);
}
try {
for (int j = 0; j < file.length; j++) {
if (file[j].getSize() != 0) {
Is = file[j].getInputStream();
String fileext = FilenameUtils.getExtension(file[j].getOriginalFilename());
AWSCredentials credentials = new BasicAWSCredentials(prop.getProperty("acceskey"), prop.getProperty("scretkey"));
String BucketName = prop.getProperty("bucketName");
key = title.concat(".").concat(fileext);
ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(Long.valueOf(Is.available()));
metadata.setContentType("image".concat("/").concat(fileext));
s3client.uploadfile(credentials, BucketName, key, Is, metadata);
}
}
}catch (AmazonClientException e) {
return result;
} catch (IOException ex) {
ex.printStackTrace();
}
return result;
}
Hope anyone can help me how to show the progress as I am unable to do. Thanks in advance.
I am currently working on a Spring WebFlux project that connects to ElasticSearch. I have a Rest Service that in turn calls a method in the Service Layer that connects to the ES. I am having trouble writing UnitTests for my Service Layer. Any help would be appreciated as this is the first time I am working with Reactive Programming. Below are the code snippets for my Controller and Service methods.
Controller Code :
#GetMapping(path = "/api/apis/services/{id}", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
Flux<ClassA> serviceApis(#PathVariable final String serviceKey) {
return apiService.getDataForService(serviceKey);
}
Service Layer :
#PreAuthorize("isFullyAuthenticated()")
public Flux<ClassA> getDataForService(
final String id) {
IdentityToken token = GSLSecurityContext.getCurrentUserIdentityToken();
if (token == null) {
return Flux.error(new Exception("No token found"));
}
String securityQueryJson = getSecurityShould(token);
String queryToRun = QUERY
.replace("XXX_SIZE_XXX", config.getValueAsString("scroll.size"))
.replace("XXX_SECURITY_SHOULD_XXX", securityQueryJson)
.replace("XXX_SERVICE_KEY_XXX", id);
WebClient client = ClientUtil.getDataLakeWebClient(config);
Flux<ClassA> response = getData(client, queryToRun);
return response;
}
The getData code is as below :
protected Flux<ClassA> getData(
final WebClient client,
final String queryToRun) {
String scrollTimeoutQuery = "?scroll=" + config.getValueAsString("scroll.timeout");
long timeout = config.getValueAsLong("query.timout");
return Flux.generate(
() -> null,
(scrollId, sink) -> {
ClassAWrapper lastWrapper = null;
if (scrollId == null) {
Mono<ClassAWrapper> wrapper = client.post()
.uri(getSearchURI() + scrollTimeoutQuery)
.body(BodyInserters.fromObject(queryToRun)).retrieve()
.bodyToMono(ClassAWrapper.class)
.onErrorMap(original -> new Exception("Unable to retrieve from elastic search for query " + queryToRun, original))
.log();
try {
lastWrapper = wrapper.block(Duration.ofSeconds(timeout));
} catch (IllegalStateException ex) {
LOG.error("Timedout after " + timeout + " seconds while getting data from elastic search for query " + queryToRun);
lastWrapper = null;
} catch (Exception ex) {
LOG.error("Error in getting message details",ex);
lastWrapper = null;
}
} else {
String scrollQuery = "{\"scroll\" : \"" + config.getValueAsString("scroll.timeout") + "\", \"scroll_id\" : \"" + scrollId + "\"}";
Mono<ClassAWrapper> wrapper = client.post()
.uri("_search/scroll")
.body(BodyInserters.fromObject(scrollQuery)).retrieve()
.bodyToMono(ClassAWrapper.class)
.onErrorMap(original -> new Exception("Unable to retrieve next page of data from elastic search", original))
.log();
try {
lastWrapper = wrapper.block(Duration.ofSeconds(timeout));
} catch (IllegalStateException ex) {
LOG.error("Timeout after " + timeout + " seconds while getting data from elastic search for query " + queryToRun);
lastWrapper = null;
} catch (Exception ex) {
LOG.error("Error in getting message details",ex);
lastWrapper = null;
}
}
if (lastWrapper == null || lastWrapper.getResult() == null || lastWrapper.getResult().getDetails().isEmpty()) {
sink.next(new ClassA());
sink.complete();
return null;
}
sink.next(lastWrapper.getResult());
return lastWrapper.getScrollId();
}
);
}
Here, queryToRun is the ES query to be executed. config is the configuration. I need to test the method "getDataForService()".
I have a simple file upload web service as a small part of my project.
This is what I have done so far on the server side :
#POST
#Path("/file")
#Consumes(MediaType.MULTIPART_FORM_DATA)
public Response uploadFile(List<Attachment> attachments,#Context HttpServletRequest request) {
System.out.println("Got an attachment!");
for(Attachment attr : attachments) {
DataHandler handler = attr.getDataHandler();
try {
InputStream stream = handler.getInputStream();
MultivaluedMap map = attr.getHeaders();
OutputStream out = new FileOutputStream(new File("/home/yashdosi/s/" + getFileName(map))); //getFileName is a seperate private function..
int read = 0;
byte[] bytes = new byte[1024];
while ((read = stream.read(bytes)) != -1) {
out.write(bytes, 0, read);
}
stream.close();
out.flush();
out.close();
} catch(Exception e) {
e.printStackTrace();
}
}
return Response.ok("file uploaded").build();
}
It works perfectly well when requests come from html forms...when I try to send a request from a java client it simply doesnt work..!!
Any ideas about on creating a java client for this code..
Here is the code I tried with...maybe there is a simple error in this code but..I dont see it...also as I said this code simple wont work...no errors or anything else....when I tried printing something on the server console to see if the service is invoked...it did NOT print anything..so I think I am unable to contact the service for some reason...
public static void uploadPhoto()
{
String url = "http://localhost:8080/fileupload-ws/services/postdata";
String output = null;
PostMethod mPost = new PostMethod(url);
HttpClient client = new HttpClient();
try
{
File imageFile = new File("/home/yashdosi/1.jpg");
BufferedImage image = ImageIO.read(imageFile);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ImageIO.write(image, "jpg", baos);
byte[] encodedImage = Base64.encodeBase64(baos.toByteArray());
String data = " " + " " + "" + "image/jpeg" + " " + "" + new String(encodedImage) + " " + "";
mPost.setRequestBody(data);
mPost.setRequestHeader("Content-Type", "text/xml");
client.executeMethod( mPost );
output = mPost.getResponseBodyAsString( );
mPost.releaseConnection( );
} catch (HttpException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(output);
}
Finally got a client working!!
HttpClient httpclient = new DefaultHttpClient();
try {
HttpPost httppost = new HttpPost("http://localhost:8080/fileupload-ws/services/postdata");
FileBody img = new FileBody(new File("/home/yashdosi/1.jpg"));
FileBody html = new FileBody(new File("/home/yashdosi/hotmail.html"));
MultipartEntity reqEntity = new MultipartEntity();
reqEntity.addPart("image", img);
reqEntity.addPart("html", html);
httppost.setEntity(reqEntity);
httppost.setHeader("Content-Type", "multipart/form-data");
System.out.println("executing request " + httppost.getRequestLine());
HttpResponse response = httpclient.execute(httppost);
HttpEntity resEntity = response.getEntity();
System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
if (resEntity != null) {
System.out.println("Response content length: " + resEntity.getContentLength());
}
EntityUtils.consume(resEntity);
}
catch(Exception e)
{
e.printStackTrace();
}
finally {
try { httpclient.getConnectionManager().shutdown(); } catch (Exception ignore) {}
}