I have to read and process a file in an AWS Lambda function from an SFTP server that is not on AWS.
Some external source is putting the file in the SFTP server which is not in AWS, and whenever the file is uploaded completely, we have to check it via AWS CloudWatch and then trigger an AWS Lambda for processing this file.
Is this approach right? Is this even possible?
If this is possible, please suggest some steps. I checked in AWS CloudWatch but I was not able to found any trigger which checks the file outside the AWS.
You need to create some sort of job that will monitor your SFTP directory (e.g., using inotify) and then invoke your AWS Lambda function by using AWS access keys of an IAM user created with programmatic access enabled and sufficient permissions to invoke that AWS Lambda function.
You can also create an AWS CloudWatch event that will be triggered on scheduled basis like every 5 mins that will trigger the AWS Lambda function to check for any news file by maintaining a history somewhere for example on AWS DynamoDB but I would rather trigger the AWS Lambda from SFTP server as using AWS based file-upload on SFTP detection sounds better if AWS Transfer for SFTP is used instead of on-premises SFTP server because it uses AWS S3 as an SFTP store and AWS S3 as the feature of creating an event on files/objects upload and trigger an AWS Lambda function.
Can you modify the external source script ?
If yes, you can send a SNS notification to specific topic using the aws cli or a specific language sdk.
Then you can have a lambda to process your file, triggered by the SNS topic.
Related
is there a way to push newly uploaded file from S3 to an FTP or SFTP server within AWS Services?
my s3 looks something like this:
s3-bucket/some_path/yyyymm/yyyymmdd/file_yyymmdd.csv.gz
and everytime we generate a new file based on date, we need to upload or transfer to FTP server
You can have S3 send event notifications to other AWS services when a new object is uploaded to a bucket.
You could have that trigger a Lambda function each time a new object is uploaded. The Lambda function would receive an event object with information about the S3 bucket and the path of the object in the bucket. It can use that info to download the file from S3, and upload it to an FTP server.
I would recommend having S3 send the events to an SQS queue, and having your Lambda function pull events from the queue, that way you have both built-in error handling, and concurrency throttling of your Lambda function invocations.
If you don't want to use a Lambda function for this, you could have S3 send the events to SQS, and then run some code that polls SQS anywhere, such as on an EC2 server, or in an ECS task.
How I can send or export logs from my AWS Lambda function to S3 without using by CloudWatch?
Is there another option?
Yes, you can stop your cloudwatch log ingesting from function and send the log to another destination.
Please check- Disabling logging to CloudWatch Logs
This AWS Sample shows you have to use a Lambda extension to send logs to S3. But there is currently no way to stop logs from going to CloudWatch.
I am looking for a way to trigger my gitlab ci pipeline whenever there is a object (with specific name) is added in S3 bucket.
I am new with AWS lambda, can someone please help
You can use s3 event notifications and deliver these events to other AWS services, including AWS Lambda or SQS queues.
Leveraging these event notifications, you could send the event directly to a lambda function that subscribes to putobject events and parses the event content structure to determine if an object of the specific name was created and use the create pipeline API to trigger pipelines on GitLab.
Architecture overview:
I created a custom app that automatically uploads logs to s3.
Is there a way to push those logs to cloudwatch from s3 for analysis and alerting?
I'm aware that I can use a cloudwatch agent to push directly to cloudwatch from the app but there are complications involved in that option.
Thank you!
You could probably use Cloudwatch Events to listen to S3 changes. Not sure about if you can get the data from the S3 file, or just a trigger saying that a new log has been added.
You could also use S3 event notifications (https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) connected either to a lambda or SQS, and from there create the logs to Cloudwatch. (similar to what was suggested by #marcin)
A better solution, but one that is a bit beyond the scope of the question, would be to send your logs through Kinesis Firehose and from there add the Cloudwatch and S3 logs.
I'm not aware of any out-of-the-box mechanism for that provided by AWS. But I think it could be relatively easy to develop.
Namely, you can create S3 notification for a PUT of a new log file from your app to S3. The event would trigger a lambda function. The function would get the file and using AWS SDK, e.g. boto3's put_log_events, it would send the log events to CloudWatch logs.
I have an AWS Lambda function that takes in and processes logs from CloudWatch Logs that are sent to specific log groups. The thing is, I may need to add more triggers as more log groups are created. The only way I have found to create a trigger for a specific log group is to use the AWS Lambda console and the AWS CloudFront console. Is it possible to create a trigger for an AWS Lambda function programmatically? For instance, in some Java code?
Yes, one of the common ways of triggering server-less functions is using endpoints. I believe you can expose an API endpoint from the Function's console using a an API Gateway, and call this endpoint URL from your java code or whatever programmatic entity you wish.