AWS Lambda with Apache Kafka Trigger returns "PROBLEM: Lambda internal error. Please contact Lambda customer support." - spring-cloud-function

Question

I am using Spring Cloud Functions with Kafka Binder. My application SpringcloudfuncApplication.class
is pretty straightforward as can be seen below:
#SpringBootApplication
#Log4j2
public class SpringcloudfuncApplication {
public static void main(String[] args) {
SpringApplication.run(SpringcloudfuncApplication.class, args);
}
#Bean
public Function<Message<String>, String> greeter() {
return (input) -> {
log.info("Hello {}", input.getPayload());
return "Hello " + input.getPayload();
};
}
}
I am expecting a incoming StringInput then will return a greeting -> Hello {StringInput}
I have setup a self-managed Apache Kafka/Zookeeper on AWS EC2 instance with public access for testing purposes then updated my application.yml file:
spring:
cloud:
function:
definition: greeter
stream:
kafka:
default:
consumer:
startOffset: earliest
binder:
brokers: ec2-13-212-236-60.ap-southeast-1.compute.amazonaws.com:9092
bindings:
greeter-in-0:
destination: topic-names
greeter-out-0:
destination: topic-greetings
Everything is working fine on my local setup pointing to this AWS EC2 with Kafka/Zookeeper. However, I am not able to make this work when I attempt to create a function in AWS Lambda with my shaded jar then added Apache Kafka for trigger. I have followed this AWS Blog in setting up AWS Lambda with self-hosted Kafka as an Event source.
Take note that doing a AWS Lambda test by sending a string (using hello-world template) my function works. But adding a Kafka trigger gives me below error:
Last processing result: PROBLEM: Lambda internal error. Please contact Lambda customer support.
Maven dependencies:
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-function-context</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-function-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-function-adapter-aws</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
Maven build plugin for shaded JAR:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-deploy-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<excludes>
<exclude>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</exclude>
</excludes>
</configuration>
<dependencies>
<dependency>
<groupId>org.springframework.boot.experimental</groupId>
<artifactId>spring-boot-thin-layout</artifactId>
<version>${spring-boot-thin-layout.version}</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>aws</shadedClassifierName>
</configuration>
</plugin>
</plugins>
</build>
Thank you in advance! Cheers!

Answer 1

It looks like the main issue is I have used public subnet on my EC2 instance with Apache Kafka/Zookeeper thus I have declared a public DNS on the AWS Lambda trigger - Apache Kafka Bootsrap servers. Now, I got it working by going thru again this reference but instead of using a NAT gateway I have setup a NAT Instance for better costing. I am now getting OK on the processing results.
Apache Kafka: Endpoints: [ip-10-0-1-199.ap-southeast-1.compute.internal:9092] (Enabled)
Details
Batch size: 100
Last processing result: OK
selfManagedEventSource:
Looking into the AWS CloudWatch logs, I am now seeing these logs:
2021-03-02 12:00:45.912 INFO 8 --- [ main] o.s.c.f.a.aws.CustomRuntimeEventLoop : Located function greeter
2021-03-02 12:00:45.918 INFO 8 --- [ main] c.j.s.s.SpringcloudfuncApplication : Hello {
"eventSource": "SelfManagedKafka",
"bootstrapServers": "ip-10-0-1-199.ap-southeast-1.compute.internal:9092",
"records": {
"topic-names-0": [
{
"topic": "topic-names",
"partition": 0,
"offset": 13,
"timestamp": 1614686444377,
"timestampType": "CREATE_TIME",
"value": "SnVuIEtpbmc="
}
]
}
}
And this goes to my next question, my String message produced on the topic is "Jun King" but the message returned to me is a JSON payload with my expected value encoded into Base64. I have tried to decode "SnVuIEtpbmc=" using this Online Base64Decoder which indeed return my actual message "Jun King".
I can process this within the code but would that make my function AWS-aware? and would have to traverse it. Thank you again in advance. Cheers!