AWS Glue Job - Unable to pass the transformed glue data to a lambda function - amazon-web-services

Question

I'm a beginner at the AWS-Glue service. I'm having an issue with Glue's ETL-job.
My use case:
1) Processing the S3 JSON files using Glue-Crawler to catalog
Note: The S3 files will be created frequently.
2) Created Glue-Job to process the catalog data & that will be transformed and stored in the S3 target location.
In addition to that, I'm calling a Lambda function to notify it to slack channel which is working. But I couldn't send the transformed data to the lambda function.
Glue-Job(Python) Script:-
The value block used in the code [bucket_name] need to be updated
import sys
import boto3
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "[bucket_name]", table_name = "data_source", transformation_ctx = "datasource0")
applymapping1 = ApplyMapping.apply(frame = datasource0, mappings = [("doc_id", "string", "doc_id", "string"), ("skill_campaign", "string", "skill_campaign", "string"), ("disposition", "string", "disposition", "string"), ("customer_num", "string", "customer_num", "string"), ("call_date", "string", "call_date", "string"), ("segstart", "string", "segstart", "string"), ("segstop", "string", "segstop", "string"), ("duration", "string", "duration", "string"), ("vendor", "string", "vendor", "string"), ("user_id", "string", "user_id", "string"), ("supervisor_user_id", "string", "supervisor_user_id", "string"), ("evaluation_status_id", "string", "evaluation_status_id", "string"), ("assigned_to", "string", "assigned_to", "string"), ("evaluation_form_id", "string", "evaluation_form_id", "string")], transformation_ctx = "applymapping1")
datasink2 = glueContext.write_dynamic_frame.from_options(frame = applymapping1, connection_type = "s3", connection_options = {"path": "s3://[bucket_name]/data-target"}, format = "json", transformation_ctx = "datasink2")
job.commit()
##Call Lambda function: How to send data that above ETL job processed to this lambda function?
client = boto3.client('lambda' , region_name='ap-south-1')
r_lambda = client.invoke(FunctionName='lambda-slack-notification', InvocationType='Event')
Lambda Function(NodeJS) Script:-
The value blocks used in the code [token], [channel], [username] & [text] need to be updated
const https = require('https');
exports.handler = (event, context, callback) => {
const options = {
hostname: "slack.com",
method: "POST",
path: "/api/chat.postMessage?token=[token]&channel=[channel]&username=[username]&text=[text]",
};
const req = https.request(options, (res) => {
res.setEncoding('utf8');
res.on('data', (chunk) => {
// code to execute
});
res.on('end', () => {
// code to execute
});
});
req.on('error', (e) => {
callback(null, "Error has occured");
});
req.end();
}
Thanks in advance...