I want to do two things in my code
I want to move the array of images from cloudinary to s3 bucket
Replace the url with the s3 bucket url
I was able to replace the url but I am not able to move those images from cloudinary to s3
Here is that API
const userSchemaPortfolio = async (req, res) => {
let userPortfolio = (await userSchema.find({}, { portfolio: 1 })).forEach(async
function (docs) {
let updatedPortfolio = []
if (docs.portfolio.length > 0) {
// console.log(docs.portfolio.length, typeof (docs.portfolio), Object.keys(docs.portfolio).length, docs._id)
for (i = 0; i < docs.portfolio.length; i++) {
if (docs.portfolio != "") {
// console.log(docs)
let endPoint = new Date().getTime() + '_' + docs.portfolio[i].toString().split('/').pop();
// console.log(endPoint, docs.portfolio[i])
let imageURL = docs.portfolio[i]
// console.log(imageURL)
let resImg = await fetch(imageURL);
console.log(resImg.url)
let blob = await resImg.buffer()
// console.log(blob)
var uploadedImage = s3.upload({
Bucket: process.env.AWS_BUCKET,
Key: endPoint,
Body: blob,
}).promise()
// console.log(uploadedImage)
var portfolio = "https://**************.amazonaws.com/" + endPoint;
// console.log("....>>>>>",portfolio)
updatedPortfolio.push(portfolio);
// console.log(updatedPortfolio)
// return;
}
}
// console.log(updatedPortfolio)
return;
let updateData = { portfolio: updatedPortfolio }
await userSchema.updateOne({ _id: docs._id }, { $set: updateData });
// console.log(updatedPortfolio, docs._id, docs.portfolio)
}
});
let user = await userSchema.find({}, { portfolio: 1 })
// console.log(chat)
res.send(user)
}
Related
I have s3 bucket as below,
myBucket
a/
b/
c/
where myBucket is a s3 bucket and a, b, c are the key folders inside that bucket.
I will upload images into a/. The s3 event notification will trigger SQS which will then trigger lambda function which does the process of removing image background and uploads into b/ folder.
The problem here is for example, if I upload a folder which has around 26 images into s3 only 23 or 22 images are getting triggered by lambda and only those images are getting are processed.
For some reason s3 is not triggering all the images or is that something I should configure in my lambda function?
Here is my function code
exports.handler = async(event, context, callback) => {
try {
console.log(event.Records[0]);
var json = JSON.parse(event.Records[0]['body']);
console.log('json: '+json);
json = JSON.parse(json['Message']);
json = json['Records'][0]['s3'];
console.log(json);
var srcBucket = json['bucket']['name'];
console.log('srcBucket: ' + srcBucket);
var srcKey = decodeURIComponent(json['object']['key'].replace(/\+/g, ' '));
console.log('srcKey: ' + srcKey);
var str = (srcKey.split('/').pop()).split('.')[0];
console.log('str: ' + str);
if (str != '') {
var folderPath = srcKey.substr(srcKey.indexOf('/') + 1).split('.')[0];
folderPath = folderPath.substring(0, folderPath.lastIndexOf('/'));
console.log('folderPath: ' + folderPath);
const params1 = { Bucket: srcBucket, Key: srcKey };
var origimage = await s3.getObject(params1).promise();
var destObject = await origimage.Body;
var destKey = 'removebg/' + folderPath + '/' + str + '.jpg';
var options = {
'method': 'POST',
'url': 'https://api.remove.bg/v1.0/removebg',
'headers': {
'X-Api-Key': 'xxxxxxxxxxx'
},
formData: {
'image_file': destObject,
'size': 'auto'
},
encoding: null
};
request(options, function(error, response, body) {
if (error) {
console.log(error);
sendmessage(error, 'Error removing image background', arn, srcBucket + '/' + srcKey, destBucket + destKey);
}
var params = { Bucket: destBucket, Key: destKey, Body: body };
s3.upload(params, function(err, data) {
if (err) {
console.log('Error uploading data: ', err);
sendmessage(err, 'Error uploading transparent image to s3', arn, srcBucket + '/' + srcKey, destBucket + destKey);
}
else { console.log('Successfully uploaded data to ' + destBucket); }
});
});
}
}
catch (e) {
console.log(e);
}
callback(null, 'All done!');
};
Please let me know. Thanks in advance.
I think your problem lies on these lines:
console.log(event.Records[0]);
var json = JSON.parse(event.Records[0]['body']);
console.log('json: '+json);
json = JSON.parse(json['Message']);
json = json['Records'][0]['s3'];
Your function is only looking at the first record that is provided to the function. Multiple events can be given the the Lambda function, so your function should loop through the Records entries and process all of the events that are provided.
It should do something like:
for record in event.Records:
console.log(record);
var json = JSON.parse(record['body']);
console.log('json: '+json);
json = JSON.parse(json['Message']);
json = record['s3'];
hello i wonder to upload images in flutter
i try to use http.MultipartRequest
like this
request.fields["name"] = "$RegisterName";
request.fields["description"] = "$RegisterDescription";
request.fields["caution"] = "$RegisterCaution";
request.fields["price"] = "$RegisterPrice";
request.fields["price_prop"] = "$RegisterPriceProp";
request.fields["user.id"] = "1";
request.fields["lend"] = "$RegisterCategory";
request.fields["category"] = "Digital";
request.fields["place_option"] = "true";
var multipartFile = http.MultipartFile.fromBytes(
'file',
(await rootBundle.load('assets/images/main_1.jpg')).buffer.asUint8List(),
filename: 'test01.jpg',
contentType: MediaType('image', 'jpg'),
);
request.files.add(multipartFile);
var response = await request.send();
if (response.statusCode == 200) print('Upload');
}
but this code is not working
if i use this code, upload only another data
upload things
then json type is this
json type image
i want upload images files ...:(
i use this to send picture with formData
var head = Api().bearerHeader; ////just bearerToken
var request = http.MultipartRequest(
'POST',
Uri.parse(
'https://c.....'));
request.files
.add(await http.MultipartFile.fromPath('TITLEOFFORMDATA', imageFile.path));
request.headers.addAll(head);
http.StreamedResponse response = await request.send();
if (response.statusCode == 200) {
String varo = await response.stream.bytesToString();
}
This is how you can send image to your server with MultipartRequest with http package
try {
final uri = Uri.parse(your_url);
final request = http.MultipartRequest('POST', uri);
final multipartFile =
await http.MultipartFile.fromPath('Image', 'your_path_of_image'); // Image is the parameter name
request.files.add(multipartFile);
request.fields['userId_if_required'] = value;
final response = await request.send();
if (response.statusCode == 200) {
print('success');
} else {
print('Something went wrong');
}
} catch (e) {
print('Something went wrong');
}
How to upload your image to a Django rest API server
this will work for sure, let me know if you have any issues.
Please be sure to add the necessary packages to your pubspec.yaml file
image_picker
http
if there is some I missed please ask me or add it and add as a reply
import 'dart:convert';
import 'package:http/http.dart' as http;
import 'dart:io';
import 'package:get/get.dart';
import 'package:image_picker/image_picker.dart';
final _picker = ImagePicker();
File? _image;
// use this to send your image
Future<void>uploadImage(filePath) async {
// your token if needed
try{
var headers = {
'Authorization':
'Bearer ' + "token",
};
// your endpoint and request method
var request = http.MultipartRequest(
'POST',
Uri.parse("https://api.imgur.com/3/image"));
request.fields
.addAll({'yourFieldNameKey1': 'yourFieldNameValue1', 'yourFieldNameKey2': 'yourFieldNameValue2'});
request.files.add(await http.MultipartFile.fromPath(
'yourPictureKey', filePath));
request.headers.addAll(headers);
http.StreamedResponse response = await request.send();
if (response.statusCode == 200) {
print(await response.stream.bytesToString());
} else {
print(response.reasonPhrase);
}
}catch(e){
print(e);
}
}
// Use this to pick your image
Future<void> _openImagePicker() async {
try {
var pickedImage = await _picker.pickImage(source: ImageSource.gallery);
if (pickedImage != null) {
setState(() {
_image = File(pickedImage.path);
});
uploadImage(pickedImage.path);
}
} catch (e) {
//print(e);
}
}
I am trying to write a script that will loop thru an array of items for a DynamoDB table and run a batch write command. My functionality is good, but I am having trouble with DynamoDB. Would be great if I could point my AWS.DynamoDB.DocumentClient() to my localhost running DynamoDB. Any tips?
Would also consider a way to just run the commands via the aws cli but I am not sure how to do that. I am running Node.js so it maybe possible?
Here is my code:
var AWS = require('aws-sdk');
AWS.config.update({ region: 'eu-central-1' });
var DynamoDB = new AWS.DynamoDB.DocumentClient()
DynamoDB.endpoint = 'http://localhost:8000';
const allItems = require('./resource.json');
const tableName = 'some-table-name';
console.log({ tableName, allItems });
var batches = [];
var currentBatch = [];
var count = 0;
for (let i = 0; i < allItems.length; i++) {
//push item to the current batch
count++;
currentBatch.push(allItems[i]);
if (count % 25 === 0) {
batches.push(currentBatch);
currentBatch = [];
}
}
//if there are still items left in the curr batch, add to the collection of batches
if (currentBatch.length > 0 && currentBatch.length !== 25) {
batches.push(currentBatch);
}
var completedRequests = 0;
var errors = false;
//request handler for DynamoDB
function requestHandler(err, data) {
console.log('In the request handler...');
return function (err, data) {
completedRequests++;
errors = errors ? true : err;
//log error
if (errors) {
console.error(JSON.stringify(err, null, 2));
console.error('Request caused a DB error.');
console.error('ERROR: ' + err);
console.error(JSON.stringify(err, null, 2));
} else {
var res = {
statusCode: 200,
headers: {
'Content-Type': 'application/json',
'Access-Control-Allow-Methods': 'GET,POST,OPTIONS',
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Credentials': true,
},
body: JSON.stringify(data),
isBase64Encoded: false,
};
console.log(`Success: returned ${data}`);
return res;
}
if (completedRequests == batches.length) {
return errors;
}
};
}
//Make request
var params;
for (let j = 0; j < batches.length; j++) {
//items go in params.RequestedItems.id array
//format for the items is {PutRequest : {Item: ITEM_OBJECT}}
params = '{"RequestItems": {"' + tableName + '": []}}';
params = JSON.parse(params);
params.RequestItems[tableName] = batches[j];
console.log('before db.batchWriteItem: ', params);
//send to db
DynamoDB.batchWrite(params, requestHandler(params));
}
I figured it out and will leave this here for anyone that may need it.
var { DynamoDB } = require('aws-sdk');
var db = new DynamoDB.DocumentClient({
region: 'localhost',
endpoint: 'http://localhost:8000',
});
Following this Streaming CloudWatch Logs Data to Amazon Elasticsearch Service, it's working fine to stream cloud watch log to ELK having one log group and one Lambda function.
But now I want to change target lambda function for my other logs group, but I am not able to do that as there is no option in AWS console.
Any Help will be appreciated.
Thanks
I was streaming to ELK using the AWS console option which is Start Streaming to Amazon Elasticsearch Service, But I failed to change or choose different lambda function as there is only lambda function can be selected for any log group using this option.
So, I create new lambda function and set stream target to AWS lambda function,
Here is the code that all you need, Node version for lambda function is 4.* as it was some issue with the new version but the pulse point is it does not require any extra NPM packages.
// v1.1.2
var https = require('https');
var zlib = require('zlib');
var crypto = require('crypto');
var endpoint = 'search-my-test.us-west-2.es.amazonaws.com';
exports.handler = function(input, context) {
// decode input from base64
var zippedInput = new Buffer(input.awslogs.data, 'base64');
// decompress the input
zlib.gunzip(zippedInput, function(error, buffer) {
if (error) { context.fail(error); return; }
// parse the input from JSON
var awslogsData = JSON.parse(buffer.toString('utf8'));
// transform the input to Elasticsearch documents
var elasticsearchBulkData = transform(awslogsData);
// skip control messages
if (!elasticsearchBulkData) {
console.log('Received a control message');
context.succeed('Control message handled successfully');
return;
}
// post documents to the Amazon Elasticsearch Service
post(elasticsearchBulkData, function(error, success, statusCode, failedItems) {
console.log('Response: ' + JSON.stringify({
"statusCode": statusCode
}));
if (error) {
console.log('Error: ' + JSON.stringify(error, null, 2));
if (failedItems && failedItems.length > 0) {
console.log("Failed Items: " +
JSON.stringify(failedItems, null, 2));
}
context.fail(JSON.stringify(error));
} else {
console.log('Success: ' + JSON.stringify(success));
context.succeed('Success');
}
});
});
};
function transform(payload) {
if (payload.messageType === 'CONTROL_MESSAGE') {
return null;
}
var bulkRequestBody = '';
payload.logEvents.forEach(function(logEvent) {
var timestamp = new Date(1 * logEvent.timestamp);
// index name format: cwl-YYYY.MM.DD
var indexName = [
'prod-background-wo-' + timestamp.getUTCFullYear(), // year
('0' + (timestamp.getUTCMonth() + 1)).slice(-2), // month
('0' + timestamp.getUTCDate()).slice(-2) // day
].join('.');
var source = buildSource(logEvent.message, logEvent.extractedFields);
source['response_time'] = source["end"] - source["start"];
source['#id'] = logEvent.id;
source['#timestamp'] = new Date(1 * logEvent.timestamp).toISOString();
source['#message'] = logEvent.message;
source['#owner'] = payload.owner;
source['#log_group'] = payload.logGroup;
source['#log_stream'] = payload.logStream;
var action = { "index": {} };
action.index._index = indexName;
action.index._type = payload.logGroup;
action.index._id = logEvent.id;
bulkRequestBody += [
JSON.stringify(action),
JSON.stringify(source),
].join('\n') + '\n';
});
return bulkRequestBody;
}
function buildSource(message, extractedFields) {
if (extractedFields) {
var source = {};
for (var key in extractedFields) {
if (extractedFields.hasOwnProperty(key) && extractedFields[key]) {
var value = extractedFields[key];
if (isNumeric(value)) {
source[key] = 1 * value;
continue;
}
jsonSubString = extractJson(value);
if (jsonSubString !== null) {
source['$' + key] = JSON.parse(jsonSubString);
}
source[key] = value;
}
}
return source;
}
jsonSubString = extractJson(message);
if (jsonSubString !== null) {
return JSON.parse(jsonSubString);
}
return {};
}
function extractJson(message) {
var jsonStart = message.indexOf('{');
if (jsonStart < 0) return null;
var jsonSubString = message.substring(jsonStart);
return isValidJson(jsonSubString) ? jsonSubString : null;
}
function isValidJson(message) {
try {
JSON.parse(message);
} catch (e) { return false; }
return true;
}
function isNumeric(n) {
return !isNaN(parseFloat(n)) && isFinite(n);
}
function post(body, callback) {
var requestParams = buildRequest(endpoint, body);
var request = https.request(requestParams, function(response) {
var responseBody = '';
response.on('data', function(chunk) {
responseBody += chunk;
});
response.on('end', function() {
var info = JSON.parse(responseBody);
var failedItems;
var success;
if (response.statusCode >= 200 && response.statusCode < 299) {
failedItems = info.items.filter(function(x) {
return x.index.status >= 300;
});
success = {
"attemptedItems": info.items.length,
"successfulItems": info.items.length - failedItems.length,
"failedItems": failedItems.length
};
}
var error = response.statusCode !== 200 || info.errors === true ? {
"statusCode": response.statusCode,
"responseBody": responseBody
} : null;
callback(error, success, response.statusCode, failedItems);
});
}).on('error', function(e) {
callback(e);
});
request.end(requestParams.body);
}
function buildRequest(endpoint, body) {
var endpointParts = endpoint.match(/^([^\.]+)\.?([^\.]*)\.?([^\.]*)\.amazonaws\.com$/);
var region = endpointParts[2];
var service = endpointParts[3];
var datetime = (new Date()).toISOString().replace(/[:\-]|\.\d{3}/g, '');
var date = datetime.substr(0, 8);
var kDate = hmac('AWS4' + process.env.AWS_SECRET_ACCESS_KEY, date);
var kRegion = hmac(kDate, region);
var kService = hmac(kRegion, service);
var kSigning = hmac(kService, 'aws4_request');
var request = {
host: endpoint,
method: 'POST',
path: '/_bulk',
body: body,
headers: {
'Content-Type': 'application/json',
'Host': endpoint,
'Content-Length': Buffer.byteLength(body),
'X-Amz-Security-Token': process.env.AWS_SESSION_TOKEN,
'X-Amz-Date': datetime
}
};
var canonicalHeaders = Object.keys(request.headers)
.sort(function(a, b) { return a.toLowerCase() < b.toLowerCase() ? -1 : 1; })
.map(function(k) { return k.toLowerCase() + ':' + request.headers[k]; })
.join('\n');
var signedHeaders = Object.keys(request.headers)
.map(function(k) { return k.toLowerCase(); })
.sort()
.join(';');
var canonicalString = [
request.method,
request.path, '',
canonicalHeaders, '',
signedHeaders,
hash(request.body, 'hex'),
].join('\n');
var credentialString = [ date, region, service, 'aws4_request' ].join('/');
var stringToSign = [
'AWS4-HMAC-SHA256',
datetime,
credentialString,
hash(canonicalString, 'hex')
] .join('\n');
request.headers.Authorization = [
'AWS4-HMAC-SHA256 Credential=' + process.env.AWS_ACCESS_KEY_ID + '/' + credentialString,
'SignedHeaders=' + signedHeaders,
'Signature=' + hmac(kSigning, stringToSign, 'hex')
].join(', ');
return request;
}
function hmac(key, str, encoding) {
return crypto.createHmac('sha256', key).update(str, 'utf8').digest(encoding);
}
function hash(str, encoding) {
return crypto.createHash('sha256').update(str, 'utf8').digest(encoding);
}
i stumbled across this function on the internet.
private createFileName() {
var d = new Date(),
n = d.getTime(),
newFileName = n + ".jpg";
return newFileName;
}
It's purpose is to rename an image file. The questions i have are
Does this restrict image upload to jpg only or does it convert it to
jpg.
If it's the former how can i get the image extension before upload
and modify private createFileName(ext) function like so, so as to upload
every image type.
If it would help here is the full code
lastImage: string = null;
loading: Loading;
public takePicture(sourceType) {
// Create options for the Camera Dialog
var options = {
quality: 100,
sourceType: sourceType,
saveToPhotoAlbum: false,
correctOrientation: true
};
// Get the data of an image
this.camera.getPicture(options).then((imagePath) => {
// Special handling for Android library
if (this.platform.is('android') && sourceType === this.camera.PictureSourceType.PHOTOLIBRARY) {
this.filePath.resolveNativePath(imagePath)
.then(filePath => {
let correctPath = filePath.substr(0, filePath.lastIndexOf('/') + 1);
let currentName = imagePath.substring(imagePath.lastIndexOf('/') + 1, imagePath.lastIndexOf('?'));
this.copyFileToLocalDir(correctPath, currentName, this.createFileName());
});
} else {
var currentName = imagePath.substr(imagePath.lastIndexOf('/') + 1);
var correctPath = imagePath.substr(0, imagePath.lastIndexOf('/') + 1);
this.copyFileToLocalDir(correctPath, currentName, this.createFileName());
}
}, (err) => {
this.presentToast('Error while selecting image.');
});
}
private copyFileToLocalDir(namePath, currentName, newFileName) {
this.file.copyFile(namePath, currentName, cordova.file.dataDirectory, newFileName).then(success => {
this.lastImage = newFileName;
}, error => {
this.presentToast('Error while storing file.');
});
}
private presentToast(text) {
let toast = this.toastCtrl.create({
message: text,
duration: 3000,
position: 'top'
});
toast.present();
}
// Always get the accurate path to your apps folder
public pathForImage(img) {
if (img === null) {
return '';
} else {
return cordova.file.dataDirectory + img;
}
}
public uploadImage() {
// Destination URL
var url = "http://my_url/uploads.php";
// File for Upload
var targetPath = this.pathForImage(this.lastImage);
// File name only
var filename = this.lastImage;
var options = {
fileKey: "file",
fileName: filename,
chunkedMode: false,
mimeType: "multipart/form-data",
params : {'fileName': filename}
};
const fileTransfer: TransferObject = this.transfer.create();
this.loading = this.loadingCtrl.create({
content: 'Uploading...',
});
this.loading.present();
// Use the FileTransfer to upload the image
fileTransfer.upload(targetPath, url, options).then(data => {
this.loading.dismissAll()
this.presentToast('Image succesfully uploaded.');
}, err => {
this.loading.dismissAll()
this.presentToast('Error while uploading file.');
});
}
1- It does not restrict to "jpg", and it does not convert to "jpg", as the method name says: create Filename, it creates a string, with the filename, just that.
2- The image extension will be jpg, as it is the default camera encodingType, and it seems like you are taking pictures using the camera. You can change the encodingtype in the camera options.
You can find all the camera options in https://ionicframework.com/docs/native/camera/