aws s3 - browser upload using post - gwt & java - amazon-web-services

I am working on a wgt webapp and would like to upload files to s3 from the browser.
Since my credentials are on the server, I need to create the signature on the server and send it to the client to be able to ulpoad.
here is the code I am using:
-dateStamp is in the right format - yyyyMMdd
-policy is base64 encoded - I double checked that
public static String getSignatureForS3Upload(final String dateStamp, final String policy) {
byte[] signingKey = null;
byte[] signature = null;
String strSignature = null;
try {
signingKey = AwsUtill.getSignatureKey(AppConfig.getS3SecretKey(), dateStamp,
AppConfigShared.getMyAwsS3RegionName(), "s3");
signature = HmacSHA256(policy, signingKey);
strSignature = bytesToHex(signature);
}
catch (Exception e) {
// log
}
ServerDBLogger.log(Level.INFO, byteArrayToHex(signature));
ServerDBLogger.log(Level.INFO, bytesToHex(signature));
return strSignature;
private static byte[] HmacSHA256(final String data, final byte[] key) throws Exception {
String algorithm = "HmacSHA256";
Mac mac = Mac.getInstance(algorithm);
mac.init(new SecretKeySpec(key, algorithm));
return mac.doFinal(data.getBytes("UTF-8"));
}
private static byte[] getSignatureKey(final String key, final String dateStamp, final String regionName,
final String serviceName) throws Exception {
byte[] kSecret = ("AWS4" + key).getBytes("UTF-8");
byte[] kDate = HmacSHA256(dateStamp, kSecret);
byte[] kRegion = HmacSHA256(regionName, kDate);
byte[] kService = HmacSHA256(serviceName, kRegion);
byte[] kSigning = HmacSHA256("aws4_request", kService);
return kSigning;
}
final protected static char[] hexArray = "0123456789ABCDEF".toCharArray();
public static String bytesToHex(final byte[] bytes) {
char[] hexChars = new char[bytes.length * 2];
for (int j = 0; j < bytes.length; j++) {
int v = bytes[j] & 0xFF;
hexChars[j * 2] = hexArray[v >>> 4];
hexChars[j * 2 + 1] = hexArray[v & 0x0F];
}
return new String(hexChars);
}
With the genarated signature I get an error message:
The request signature we calculated does not match the signature you provided. Check your key and signing method.
What am I doing wrong?
Is there any good tutorial or sample code to do this?
Thank you!

There have been changes to the AWS signature process - the version at this time (May 2016) is AWS Signature version 4 (http://docs.aws.amazon.com/general/latest/gr/signature-version-4.html). You need to be careful when looking at examples and Stackoverflow etc that the information relates to the same signature version you are using as they don't mix well.
I started using the AWS SDK for an angular/node file upload but eventually found it easier to generate the policy on the server (node.js) side without the SDK. There is a good example (albeit node based which may not be what you are looking for) here: https://github.com/danialfarid/ng-file-upload/wiki/Direct-S3-upload-and-Node-signing-example (but note the issue with the S3 bucket name here: AngularJs Image upload to S3 ).
One key thing to watch is that you correctly include the file content type in the policy generation and that this content type properly matches the content type of the file you are actually uploading.

Related

how to set hash in Postman Pre-Request Script for Marvel API

I have a pre-request script that I gathered from another post on StackOverflow, but I'm still getting invalid credentials.
Attempted to do this just with str_1 but it's not working. Not sure what request.data is supposed to do as it keeps returning NaN. I think that the problem might be there, but still at a loss. I've attempted converting all variables to a string, but that still returned the same error.
URL = https://gateway.marvel.com/v1/public/characters?ts={{timeStamp}}&apikey={{apiKey}}&hash={{hash}}
// Access your env variables like this
var ts = new Date();
ts = ts.getUTCMilliseconds();
var str_1 = ts + environment.apiKey + environment.privateKey;
// Or get your request parameters
var str_2 = request.data["timeStamp"] + request.data["apiKey"];
console.log('str_2 = ' + str_2);
// Use the CryptoJS
var hash = CryptoJS.MD5(str_1).toString();
// Set the new environment variable
pm.environment.set('timeStamp', ts);
pm.environment.set('hash', hash);
{
"code": "InvalidCredentials",
"message": "That hash, timestamp and key combination is invalid."
}
If someone can comment on why this is the solution, I would appreciate it. Here is what the issue was. The order of the hash actually matters. So had to flip the order of pvtkey + pubkey to pubkey + pvtkey. Why is this?
INCORRECT
var message = ts+pubkey+pvtkey;
var a = CryptoJS.MD5(message);
pm.environment.set("hash", a.toString());
CORRECT
var message = ts+pvtkey+pubkey;
var a = CryptoJS.MD5(message);
pm.environment.set("hash", a.toString());
I created in Android Studio, a new java class named MD5Hash, following the steps of https://javarevisited.blogspot.com/2013/03/generate-md5-hash-in-java-string-byte-array-example-tutorial.html
I just simplified his (her) code, only to use it with Java utility MessageDigest
public class MD5Hash {
public static void main(String args[]) {
String publickey = "abcdef"; //your api key
String privatekey = "123456"; //your private key
Calendar calendar=Calendar.getInstance();
String stringToHash = calendar
.getTimeInMillis()+ privatekey + publickey;
System.out.println("hash : " + md5Java(stringToHash));
System.out.println("ts : "+ calendar.getTimeInMillis());
}
public static String md5Java(String message){
String digest = null;
try {
MessageDigest md = MessageDigest.getInstance("MD5");
byte[] hash = md.digest(message.getBytes("UTF-8"));
//converting byte array to Hexadecimal String
StringBuilder sb = new StringBuilder(2*hash.length);
for(byte b : hash){
sb.append(String.format("%02x", b&0xff));
}
digest = sb.toString();
} catch (UnsupportedEncodingException ex) {
} catch (NoSuchAlgorithmException ex) {
}
return digest;
}
}
As you can see, if you copy paste this code, it has a green arrow on the left side of the class declaration, clicking it you can run MD5Hash.main() and you'll have printed in your Run Screen the values for the time (ts) and for the hash.
Then go to verify directly into the internet :
https://gateway.marvel.com/v1/public/characters?limit=20&ts=1574945782067&apikey=abcdef&hash=4bbb5dtf899th5132hjj66

AWS Signature Creation. Confused on how to covert the SigningKey and SingingString into Signature. AWS Example Seems To Not Yield Expected Result

I'm trying to write my own AWS4 signer, and I've gotten about 2/3 of the way there. Source code here :
public class Test
{
private static String region = "us-east-1";
static byte[] HmacSHA256(String data, byte[] key) throws Exception {
String algorithm="HmacSHA256";
Mac mac = Mac.getInstance(algorithm);
mac.init(new SecretKeySpec(key, algorithm));
return mac.doFinal(data.getBytes("UTF8"));
}
public static byte[] justSha256(String data) throws NoSuchAlgorithmException, UnsupportedEncodingException {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
byte[] hash = digest.digest(data.getBytes("UTF8"));
return hash;
}
static byte[] getSigningKey(String key, String dateStamp, String regionName, String serviceName) throws Exception {
byte[] kSecret = ("AWS4" + key).getBytes("UTF8");
byte[] kDate = HmacSHA256(dateStamp, kSecret);
byte[] kRegion = HmacSHA256(regionName, kDate);
byte[] kService = HmacSHA256(serviceName, kRegion);
byte[] kSigning = HmacSHA256("aws4_request", kService);
return kSigning;
}
public static String getSimpleDate()
{
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("YYYYMMdd");
return LocalDate.now().format(formatter);
}
public static String getAMZDate()
{
/*DateTimeFormatter formatter = DateTimeFormatter.ofPattern("YYYYMMDDHHMMSS");
String timeStamp = new SimpleDateFormat("YYMMDD'T'HHMMSS'Z'").format(Calendar.getInstance().getTime());*/
TimeZone tz = TimeZone.getTimeZone("UTC");
DateFormat df = new SimpleDateFormat("yyyyMMdd'T'HHmmss'Z'"); // Quoted "Z" to indicate UTC, no timezone offset
df.setTimeZone(tz);
String timeStamp = df.format(new Date());
return timeStamp;
}
public static String createSigningString(String timeStamp, String simpleDate,String serviceName) throws UnsupportedEncodingException, NoSuchAlgorithmException {
/*AWS4-HMAC-SHA256
20150830T123600Z
20150830/us-east-1/iam/aws4_request
f536975d06c0309214f805bb90ccff089219ecd68b2577efef23edd43b7e1a59*/
String algorithm = "AWS4-HMAC-SHA256\n";
String amzDate = timeStamp+"\n";
String simpleDateRegionServiceRequest = simpleDate+"/"+region+"/"+serviceName+"/"+"aws4_request\n";
String canonicalHash = getCanonicalHash(getCanonicalString("GET","/","Action=ListUsers&Version=2010-05-08","20150830T123600Z",""));
String signingString = algorithm+amzDate+simpleDateRegionServiceRequest+canonicalHash;
return signingString;
}
public static String getCanonicalString(String method, String absolutePath, String queryString, String timeStamp, String payload) throws UnsupportedEncodingException, NoSuchAlgorithmException {
String contentType = "Content-Type:application/x-www-form-urlencoded; charset=utf-8\n".toLowerCase();
String hostUrl = "host:iam.amazonaws.com\n";
String date = "x-amz-date:"+timeStamp+"\n";
String signedHeader = "content-type;host;x-amz-date\n";
String hashedPayload = Hex.encodeHexString(justSha256(payload)).toLowerCase();
String canonicalString = method+"\n"+absolutePath+"\n"+queryString+"\n"+contentType+hostUrl+date+"\n"+signedHeader+hashedPayload;
return canonicalString;
}
public static String getCanonicalHash(String canonicalString) throws UnsupportedEncodingException, NoSuchAlgorithmException {
return Hex.encodeHexString(justSha256(canonicalString)).toLowerCase();
}
public static void main(String[] args) throws Exception {
DateTimeFormatter formatter = DateTimeFormatter.ofPattern("YYYYMMDDHHMMSS");
String timeStamp = new SimpleDateFormat("YYYYMMDD'T'HHMMSS'Z'").format(Calendar.getInstance().getTime());
String canonString;
System.out.println( canonString = getCanonicalHash(getCanonicalString("GET","/","Action=ListUsers&Version=2010-05-08","20150830T123600Z","")));
String signingString = createSigningString("20150830T123600Z","20150830","iam");
String key = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY";
String dateStamp = "20120215";
String regionName = "us-east-1";
String serviceName = "iam";
String signingKey = Hex.encodeHexString(getSigningKey(key,dateStamp,regionName,serviceName));
SoftAssertions softly = new SoftAssertions();
softly.assertThat(canonString).isEqualToIgnoringCase("f536975d06c0309214f805bb90ccff089219ecd68b2577efef23edd43b7e1a59");
String copiedSigningString = "AWS4-HMAC-SHA256\n" +
"20150830T123600Z\n" +
"20150830/us-east-1/iam/aws4_request\n" +
"f536975d06c0309214f805bb90ccff089219ecd68b2577efef23edd43b7e1a59";
softly.assertThat(signingString).isEqualTo(copiedSigningString);
softly.assertThat(signingKey).isEqualToIgnoringCase("f4780e2d9f65fa895f9c67b32ce1baf0b0d8a43505a000a1a9e090d414db404d");
dateStamp ="20150830";
signingKey = Hex.encodeHexString(getSigningKey(key,dateStamp,regionName,serviceName));
softly.assertThat(signingKey).isEqualToIgnoringCase("c4afb1cc5771d871763a393e44b703571b55cc28424d1a5e86da6ed3c154a4b9");
System.out.println("COPIED STRING : "+copiedSigningString);
System.out.println("SIGNING KEY : "+signingKey);
String signature = Hex.encodeHexString(HmacSHA256(signingKey.trim(),justSha256(copiedSigningString)));
System.out.println("Signature : "+signature);
softly.assertThat(signature).isEqualToIgnoringCase("5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924a6f2b5d7");
softly.assertAll();
}
}
For some reason I'm failing to create the correct Signature. Which is created by using the SigningKey and SingingString as input into an HMACSha256 function, specifically the function highlighted on this page https://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-java.
But for some reason, I cannot produce the signature that AWS says will be created here on this page.: https://docs.aws.amazon.com/general/latest/gr/sigv4-calculate-signature.html
(5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924a6f2b5d7)
Despite using their function, their Strings as input, and double checking my work. What am I missing?
Are you supposed to use a different function for these Strings or byte[]?
I feel like I'm missing something, and I don't know where to reconcle this confusion because from my code and what I see on these pages I seem to be doing the right thing.
I'm especially confused because when I pasted the String from their site into my code to see if it would convert properly it did not, despite using their same Hmac function (which did work for creating every other hash string posted on their site).
The hex representation of the date, region, service, and signing keys is shown for illustration, because the keys contain bytes that do not represent printable characters.
But you appear to be hex-encoding your signing key before using it to sign the request. Don't do that. You will want to hex-encode it only for viewing/debugging. The actual signing key should be retained and used in its original binary/byte form.
Use the digest (binary format) for the key derivation. Most languages have functions to compute either a binary format hash, commonly called a digest, or a hex-encoded hash, called a hexdigest. The key derivation requires that you use a binary-formatted digest.
https://docs.aws.amazon.com/general/latest/gr/sigv4-calculate-signature.html
The canonical request hash in the string-to-sign is used in hex-encoded form, as is the final signature. The key derivation is all binary.

Sending Multiple images from web service SPRINGBOOT+REST+MAVEN

I am writing a web service in spring boot restful web App using which i am sending a image to anyone who wants to consume it below is a code snippet which worked for me
#RequestMapping(value = "/photo_1",method = RequestMethod.GET )
public ResponseEntity<byte[]> greeting_image_1(#RequestParam(value="name", defaultValue="World") String name) throws IOException{
InputStream in = getClass().getResourceAsStream("/images/someimage.jpg");
byte[] a = IOUtils.toByteArray(in);
final HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.IMAGE_JPEG);
return new ResponseEntity<byte[]>(a,headers,HttpStatus.CREATED);
}
This web service works perfectly fine in case i want to return a single image from a web service
But what if in case i want to return array of images(i.e. more than 1 image)
Any help is highly appreciated.
Regards,
Here's the sample code that I wrote to generate a multipart response with multiple images in it. It's properly consumed by Firefox and it prompts to save each image in the response.
public ResponseEntity<byte[]> showImages () throws IOException {
String boundary="---------THIS_IS_THE_BOUNDARY";
List<String> imageNames = Arrays.asList(new String[]{"1.jpg","2.jpg"});
List<String> contentTypes = Arrays.asList(new String[]{MediaType.IMAGE_JPEG_VALUE,MediaType.IMAGE_JPEG_VALUE});
List<Byte[]> imagesData = new ArrayList<Byte[]>();
imagesData.add(ArrayUtils.toObject(IOUtils.toByteArray(getClass().getResourceAsStream("/images/1.jpg"))));
imagesData.add(ArrayUtils.toObject(IOUtils.toByteArray(getClass().getResourceAsStream("/images/2.jpg"))));
byte[] allImages = getMultipleImageResponse(boundary, imageNames,contentTypes, imagesData);
final HttpHeaders headers = new HttpHeaders();
headers.set("Content-Type","multipart/x-mixed-replace; boundary=" + boundary);
return new ResponseEntity<byte[]>(allImages,headers,HttpStatus.CREATED);
}
private static byte[] getMultipleImageResponse(String boundary, List<String> imageNames, List<String> contentTypes, List<Byte[]> imagesData){
byte[] finalByteArray = new byte[0];
Integer imagesCounter = -1;
for(String imageName : imageNames){
imagesCounter++;
String header="--" + boundary
+ "\r\nContent-Disposition: form-data; name=\"" + imageName
+ "\"; filename=\"" + imageName + "\"\r\n"
+ "Content-type: " + contentTypes.get(imagesCounter) + "\r\n\r\n";
byte[] currentImageByteArray=ArrayUtils.addAll(header.getBytes(), ArrayUtils.toPrimitive(imagesData.get(imagesCounter)));
finalByteArray = ArrayUtils.addAll(finalByteArray,ArrayUtils.addAll(currentImageByteArray, "\r\n\r\n".getBytes()));
if (imagesCounter == imageNames.size() - 1) {
String end = "--" + boundary + "--";
finalByteArray = ArrayUtils.addAll(finalByteArray, end.getBytes());
}
}
return finalByteArray;
}
You should implement this depending on the capability of the consumer. If the consumer can parse multipart response, please go ahead with this approach, else consider other options like
Sending a zipped file of all images
Returning a json/xml of image names along with URLs to download them
Returning a json/xml with all images in Base64 encoded string
You may also send a html response with all images embedded in it using the below code. This should work fine in all browsers as it is pure html.
public ResponseEntity<byte[]> getAllImages() throws IOException {
List<String> imageNames = Arrays.asList(new String[]{"1.jpg","2.jpg"});
List<String> contentTypes = Arrays.asList(new String[]{MediaType.IMAGE_JPEG_VALUE,MediaType.IMAGE_JPEG_VALUE});
List<Byte[]> imagesData = new ArrayList<Byte[]>();
imagesData.add(ArrayUtils.toObject(IOUtils.toByteArray(getClass().getResourceAsStream("/images/1.jpg"))));
imagesData.add(ArrayUtils.toObject(IOUtils.toByteArray(getClass().getResourceAsStream("/images/2.jpg"))));
byte[] htmlData=getHtmlData(imageNames,contentTypes, imagesData);
final HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.TEXT_HTML);
return new ResponseEntity<byte[]>(htmlData,headers,HttpStatus.OK);
}
private static byte[] getHtmlData(List<String> imageNames, List<String> contentTypes, List<Byte[]> imagesData){
String htmlContent="<!DOCTYPE html><html><head><title>Images</title></head><body>";
Integer imagesCounter = -1;
for(String imageName : imageNames){
imagesCounter++;
htmlContent = htmlContent + "<br/><br/><b>" + imageName + "</b><br/></br/><img src='data:" + contentTypes.get(imagesCounter) + ";base64, " + org.apache.commons.codec.binary.StringUtils.newStringUtf8(Base64
.encodeBase64(ArrayUtils.toPrimitive(imagesData.get(imagesCounter)))) + "'/>";
}
htmlContent = htmlContent + "</body></html>";
return htmlContent.getBytes();
}
You should take a look to "Uploading Files" Spring Boot guide : https://spring.io/guides/gs/uploading-files/
There's a an example of upload and download.

How to get value of 'CARBON_HOME' in java code

I was trying to implement an Axis2 service that receives user requests and publishes them as events to a CEP using carbon databridge thrift (via 'org.wso2.carbon.databridge.agent.thrift.DataPublisher')
I followed the code sample provided in wso2cep-3.1.0/samples/producers/activity-monitor
please see the following code snippet
public class GatewayServiceSkeleton{
private static Logger logger = Logger.getLogger(GatewayServiceSkeleton.class);
public RequestResponse request(Request request)throws AgentException,
MalformedStreamDefinitionException,StreamDefinitionException,
DifferentStreamDefinitionAlreadyDefinedException,
MalformedURLException,AuthenticationException,DataBridgeException,
NoStreamDefinitionExistException,TransportException, SocketException,
org.wso2.carbon.databridge.commons.exception.AuthenticationException
{
final String GATEWAY_SERVICE_STREAM = "gateway.cep";
final String VERSION = "1.0.0";
final String PROTOCOL = "tcp://";
final String CEPHOST = "cep.gubnoi.com";
final String CEPPORT = "7611";
final String CEPUSERNAME = "admin";
final String CEPPASSWORD = "admin";
Object[] metadata = { request.getDeviceID(), request.getViewID()};
Object[] correlationdata = { request.getSessionID()};
Object[] payloaddata = {request.getBucket()};
KeyStoreUtil.setTrustStoreParams();
KeyStoreUtil.setKeyStoreParams();
DataPublisher dataPublisher = new DataPublisher(PROTOCOL + CEPHOST + ":" + CEPPORT, CEPUSERNAME, CEPPASSWORD);
//create event
Event event = new Event (GATEWAY_SERVICE_STREAM + ":" + VERSION, System.currentTimeMillis(), metadata, correlationdata, payloaddata);
//Publish event for a valid stream
dataPublisher.publish(event);
//stop
dataPublisher.stop();
RequestResponse response = new RequestResponse();
response.setSessionID(request.getSessionID());
response.setDeviceID(request.getDeviceID());
response.setViewID(request.getViewID());
response.setBucket(request.getBucket());
return response;
}
there is also a utility class that set the key store parameters as following
public class KeyStoreUtil {
static File filePath = new File("../../../repository/resources/security");
public static void setTrustStoreParams() {
String trustStore = filePath.getAbsolutePath();
System.setProperty("javax.net.ssl.trustStore", trustStore + "/client-truststore.jks");
System.setProperty("javax.net.ssl.trustStorePassword", "wso2carbon");
}
public static void setKeyStoreParams() {
String keyStore = filePath.getAbsolutePath();
System.setProperty("Security.KeyStore.Location", keyStore + "/wso2carbon.jks");
System.setProperty("Security.KeyStore.Password", "wso2carbon");
}
}
I uploaded the service into a wso2as-5.2.1, and called the service using SOAPUI
the request returned an error message "cannot borrow client for TCP"
I debug, and found out the problem might lies with the class 'KeyStoreUtil',
where the 'filePath' somehow retuned a 'null',
static File filePath = new File("../../../repository/resources/security");
and caused the failure on this line
DataPublisher dataPublisher = new DataPublisher(PROTOCOL + CEPHOST + ":" + CEPPORT, CEPUSERNAME, CEPPASSWORD);
I guess it could be a better idea if I use the value of "CARBON_HOME" to figure out the location of Key Store
so my question is :
How may I be able to get the value of 'CARBON_HOME' in the Java code?
that said. If you think a bit more:
the service will be called numerous time; whileas the 'setTrustStoreParams' and the 'setKeyStoreParams' will only be needed to executed once at the server/service initiate.
So, are there any even better ways to remove 'setTrustStoreParams' and 'setKeyStoreParams' out of the service code, or implement as configurable items?
Please advise
thanks
so my question is :
How may I be able to get the value of 'CARBON_HOME' in the Java code?
You should use the property carbon.home like following which will retrieve the WSO2 product's home directory.
System.getProperty("carbon.home");

Unable to pass username & password in jaxws

I'm trying to call a webservice using username/pwd using the below client but I don't see the username/password being set in the headers
Client code
AttachmentWSImplService service = new AttachmentWSImplService();
AttachmentWS aws = service.getAttachmentWS();
BindingProvider bindingProvider = (BindingProvider) aws;
SOAPBinding sopadBinding = (SOAPBinding) bindingProvider.getBinding();
sopadBinding.setMTOMEnabled(true);
bindingProvider.getRequestContext().put(bindingProvider.USERNAME_PROPERTY,"p3xferdt");
bindingProvider.getRequestContext().put(bindingProvider.PASSWORD_PROPERTY,"92mnGg1Cb14D9hVhG1W5fZra4UI=");
Server code
SOAPMessageContext ctx = (SOAPMessageContext) wsCtx
.getMessageContext();
java.util.Map<java.lang.String, java.util.List<java.lang.String>> headers = (Map<String, List<String>>) ctx
.get(MessageContext.HTTP_REQUEST_HEADERS);
if (headers.keySet() != null && !headers.keySet().isEmpty()) {
Iterator<String> keys = headers.keySet().iterator();
while (keys.hasNext()) {
String key = (String) keys.next();
logger.info("HeaderKey->" + key);
logger.info("Header values->" + headers.get(key));
// getting Basic Authentication
String tmpusername = getUsernameFromAuthentication(key,
headers.get(key).toString());
Code looks ok to me, should work fine.
Anyways try accessing Request Header by
Headers headers = ex.getRequestHeaders();
List<String> ulist = headers.get(BindingProvider.USERNAME_PROPERTY);
List<String> plist = headers.get(BindingProvider.PASSWORD_PROPERTY);
PS: Remember USERNAME_PROPERTY is static string from BindingProvider interface, can be accessed in Static way. (Coding standards :))