c++: AWS S3 Object Lambda - c++

AWS recently introduced S3 Object Lambda, however looking at the online documentation:
Writing and debugging Lambda functions for S3 Object Lambda Access Points
Introducing Amazon S3 Object Lambda – Use Your Code to Process Data as It Is Being Retrieved from S3
How to use Amazon S3 Object Lambda to generate thumbnails
I can only find example for Java, Python and NodeJS.
Is there an example out there for c++ that I missed ? In particular I fail to understand what is the equivalent in c++ for getObjectContext (python/nodejs) / S3ObjectLambdaEvent (java) ? How should I retrieve outputRoute, outputToken, inputS3Url ? The integration test does not make it particularly clear either:
https://github.com/aws/aws-sdk-cpp/blob/main/aws-cpp-sdk-s3-integration-tests/BucketAndObjectOperationTest.cpp#L580-L581

S3 Object Lambda is using the AWS Lambda service. AWS Lambda does only support the following runtimes "natively":
Go
.NET Core
Ruby
Java
Python
NodeJS
C++ is support through "custom" runtimes or through Docker containers. Usually, the AWS documentation only covers the "natively" supported runtimes and even then not all (as you noticed). Mostly they have examples for the most popular ones.
So what you need to look for are C++ Lambda examples, examples using the C++ AWS SDK and reference documentation.
Using the reference documentation and the Java/Python/NodeJS examples, it should be easy to write a C++ version.
For example:
Introducing the C++ Lambda Runtime (Blog Post)
C++ AWS SDK Page
API Reference for WriteGetObjectResponseRequest

Answering my own post after a couple of weeks of struggle.
Basic point is summarized at Introduction to Amazon S3 Object Lambda
The JSON payload of interest is simply:
{ "xAmzRequestId": "1a5ed718-5f53-471d-b6fe-5cf62d88d02a",
"getObjectContext": {
"inputS3Url": "https://transform-424432388155.s3-accesspoint.us-east-1.amazonaws.com/title.txt?X-Amz-Security-Token=...",
"outputRoute": "io-iad-cell001",
"outputToken": "..." },
You can simply open it using:
static invocation_response my_handler(invocation_request const& req, Aws::S3::S3Client const& client)
{
using namespace Aws::Utils::Json;
JsonValue json(req.payload);
if (!json.WasParseSuccessful()) {
return invocation_response::failure("Failed to parse input JSON", "InvalidJSON");
}
auto view = json.View();
auto s3url = view.GetObject("getObjectContext").GetString("inputS3Url");
auto route = view.GetObject("getObjectContext").GetString("outputRoute");
auto token = view.GetObject("getObjectContext").GetString("outputToken");
Which will later be used in WriteGetObjectResponseRequest as:
S3::Model::WriteGetObjectResponseRequest request;
request.WithRequestRoute(route);
request.WithRequestToken(token);
request.SetBody(objectStream);
auto outcome = client.WriteGetObjectResponse(request);
It is then up to the application to decide on how to construct the objectStream, but a simple pass-through example would be:
std::shared_ptr<Aws::Http::HttpRequest> getRequest =
CreateHttpRequest(s3url, Http::HttpMethod::HTTP_GET,
Aws::Utils::Stream::DefaultResponseStreamFactoryMethod);
std::shared_ptr<Aws::Http::HttpClient> httpClient =
Aws::Http::CreateHttpClient(Aws::Client::ClientConfiguration());
std::shared_ptr<Aws::Http::HttpResponse> getResponse =
httpClient->MakeRequest(getRequest);
std::shared_ptr<Aws::IOStream> objectStream =
Aws::MakeShared<Aws::StringStream>("SO-WriteGetObjectResponse");
const Aws::IOStream& responseBody = getResponse->GetResponseBody();
*objectStream << responseBody.rdbuf();
objectStream->flush();
Examples which help me are:
https://github.com/awslabs/aws-lambda-cpp/blob/master/examples/s3/main.cpp
Working with WriteGetObjectResponse (Example #2/Python)

Related

Scala: Get URL Parameters

I'm writing my first Scala lambda, and I have run into a bit of an issue, which i think should be straightforward, but I'm having trouble finding the answer.
So I have the following code that will allow me to accept json and use it in the lambda.
--eventTest.scala
val stream : InputStream = getClass.getResourceAsStream("/test_data/body.json")
--request handler
def handleRequest(input: InputStream): Unit = {
val name = scalaMapper.readValue(input, classOf[NameInfo])
val result = s"Hello there, ${name.firstName} ${name.lastName}."
println(result)
}
This works just fine, but I'm having problems figuring out how to be able to get URL parameters. Does it automatically use the same input Stream? There is seems to be very little documentation on this in Scala.
Thanks
A Lambda function's event is a JSON object. The Lambda runtime will introspect the handler function and attempt to extract or convert that object based on the function signature. I believe the easiest representation is a java.util.Map[String,String] (iirc, Lambda doesn't have a Scala runtime, so you'll have to use Java classes and convert them).
An example event from the API Gateway proxy integration: https://github.com/awsdocs/aws-lambda-developer-guide/blob/master/sample-apps/nodejs-apig/event.json
For more information about the Java runtime: https://docs.aws.amazon.com/lambda/latest/dg/java-handler.html

file payload in google cloud function bucket triggered

I have a question about a Google Cloud functions triggered by an event on a storage bucket (I’m developing it in Python).
I have to read the data of the file just finalized (a PDF file) on the bucket that is triggering the event and I was looking for the file payload on the event object passed to my function (data, context) but it seems there is not payload on that object.
Do I have to use the cloud storage library to get the file from the bucket ? Is there a way to get the payload directly from the context of the triggered function ?
Enrico
From checking the more complete examplein the Firebase documentation, it indeed seems that the payload of the file is not included in the parameters. That make sense, since there's no telling how big the file is that was just finalized, and if that will even fit in the memory of your Functions runtime.
So you'll have to indeed grab the file from the bucket with a separate call, based on the information in the metadata. The full Firebase example grabs the filename and other info from its context/data with:
exports.generateThumbnail = functions.storage.object().onFinalize(async (object) => {
const fileBucket = object.bucket; // The Storage bucket that contains the file.
const filePath = object.name; // File path in the bucket.
const contentType = object.contentType; // File content type.
const metageneration = object.metageneration; // Number of times metadata has been generated. New objects have a value of 1.
...
I'll see if I can find a more complete example. But I'd expect it to work similarly on raw Google Cloud Functions, which Firebase wraps, even when using Python.
Update: from looking at this Storage/Function/PubSub documentation that the Python binding is apparently based on, it looks like the the path should be available as data['resource'] or as data['name'].

Google Speech-to-text on Amazon Kinesis stream: io.grpc.StatusRuntimeException: CANCELLED: The operation was cancelled

I want to use Google speech-to-text using Kinesis stream as part of voicebot service using Amazon Connect, Amazon Lex and Amazon SQS (I used code from https://cloud.google.com/speech-to-text/docs/streaming-recognize#speech-streaming-mic-recognize-java and I changes reference type from AudioInputStream to InputStream).
I use Amazon Transcribe speech-to-text service but I want to replace it with Google as Google supports more languages. However, Google Speech can't accept InputStream object created by Amazon SDK.
I use the code below. Instead of changing AudioInputStream to InputStream, I also tried getAudioInputStream() method (also with creating BufferedInputStream).
String streamName = streamARN.substring(streamARN.indexOf("/") + 1, streamARN.lastIndexOf("/"));
InputStream kvsInputStream = KVSUtils.getInputStreamFromKVS(streamName, REGION, startFragmentNum, getAWSCredentials());
//InputStream bufferedIn = new BufferedInputStream(kvsInputStream); //to solve 'no reset/mark support in stream' error
//AudioInputStream audioStream = AudioSystem.getAudioInputStream(bufferedIn); //to solve 'no reset/mark support in stream' error
streamingMicRecognize(kvsInputStream);
In current state I get the error
com.google.api.gax.rpc.CancelledException: io.grpc.StatusRuntimeException: CANCELLED: The operation was cancelled.
When I used two commented lines (I found this solution on SO), the error was
java.lang.ClassCastException: com.amazonaws.util.ServiceClientHolderInputStream cannot be cast to javax.sound.sampled.AudioInputStream
Can you please suggest any solution? For English voicebot Connect offers a special block which lets me connect phone call voice with Lex, but Lex supports only US English and I need other languages as well. I know Google Dialogflow ("Google's Lex") can process many languages and offers integration with phone gateway, but the phone gateway supports English only (which is ridiculous).
Thanks in advance.
UPDATE
I solved this problem with the following code:
InputStream kvsInputStream = KVSUtils.getInputStreamFromKVS(input.getStreamName(), Regions.US_EAST_1, input.getStartFragmentNum(), new SystemPropertiesCredentialsProvider());
StreamingMkvReader streamingMkvReader = StreamingMkvReader.createDefault(new InputStreamParserByteSource(kvsInputStream));
FragmentMetadataVisitor.BasicMkvTagProcessor tagProcessor = new FragmentMetadataVisitor.BasicMkvTagProcessor();
FragmentMetadataVisitor fragmentVisitor = FragmentMetadataVisitor.create(Optional.of(tagProcessor));
ByteBuffer audioBuffer = KVSUtils.getByteBufferFromStream(streamingMkvReader, fragmentVisitor, tagProcessor, input.getConnectContactId());
SpeechClient client = SpeechClient.create();
clientStream = client.streamingRecognizeCallable().splitCall(responseObserver); //responseObserver is an instance of ResponseObserver<StreamingRecognizeResponse>() with onXXX methods defined by user -- see Google Speech-To-Text examples
byte[] audioBytes;
int counter = 0;
do {
audioBytes = new byte[audioBuffer.remaining()];
audioBuffer.get(audioBytes);
request = StreamingRecognizeRequest.newBuilder()
.setAudioContent(ByteString.copyFrom(audioBytes))
.build();
clientStream.send(request);
audioBuffer = KVSUtils.getByteBufferFromStream(streamingMkvReader, fragmentVisitor, tagProcessor, input.getConnectContactId());
counter++;
} while (audioBytes.length > 0);
The solution is to go to the lowest possible level in stream handling, i.e. to get simply a byte array instead of stream objects

Using AWS Lambda to trigger C++ file - Hello World

I have been stuck on this for a while.
I want to be able to trigger lambda functions to run my .cpp files via lambda function.
For simplicity, I want to run a hello.cpp file that just runs hello world. I want to know how this is possible, what is the file architecture I need? What goes inside my handler function?
I know this is very simple to do in node.js, but how would I replicate the node.js hello world example to run the c++ file?
The AWS website does say I'm allowed to use custom runtime, so it should be possible.
Any insight will help.
AWS Lambda is now supporting C++ natively.
Announcement post here.
If you still want to use NodeJS, compile your program using an Amazon Linux image, and run this index.js, adding the compiled file to the folder node_modules/.bin/ and the dynamic libraries required to the folder node_modules/.bin/lib in the uploaded .zip file:
exports.handler = (event, context, callback) => {
var result = require('child_process').execSync("/lib64/ld-linux-x86-64.so.2 --library-path "+process.cwd() +"/node_modules/.bin/lib "+process.cwd() +"/node_modules/.bin/helloworld ").toString();
result = JSON.parse(result);
const response = {
statusCode: 200,
body: result
};
callback(null, response);
};

Ways to implement a JSON RESTful service in C/C++

I am trying to do a JSON Restful web service in C/C++.
I have tried Axis2/C and Staff, which work great for XML serialization/deserialization but not for JSON.
You might want to take a look at Casablanca introduced in Herb Sutter's blog.
there are a small number of libraries that support creating rest services with c, e.g. restinio:
#include <restinio/all.hpp>
int main()
{
restinio::run(
restinio::on_this_thread()
.port(8080)
.address("localhost")
.request_handler([](auto req) {
return req->create_response().set_body("Hello, World!").done();
}));
return 0;
}
try https://github.com/babelouest/ulfius great library to build C/C++ Restful APIs. can support all platforms: Linux, FreeBSD, Windows and others
Try ngrest. It's a simple but fast C++ RESTful JSON Web Services framework. It can be deployed on top of Apache2, Nginx or own simple http server.
Regarding Axis2/C with JSON. It's seems that official Axis2/C no longer maintained. So Axis2/C become obsolete (but still works).
JSON support for Axis2/C is available in axis2c-unofficial project.
There are an installation manuals on how to install Axis2/C with JSON support under Linux, Windows using binary package, Windows from source code.
You can try it with WSF Staff using Customers (REST) example in JSON mode (which is available from staff/samples/rest/webclient directory of staff source code).
You could look at ffead-cpp. Apart from providing support for json and restfull web services it also includes more features. This framework may be too heavy weight for your situation though.
For C++ web service, I am using the following stack:
ipkn/crow C++ micro web framework
nlohmann/json for json serialization/deserialization.
Take a look at Oat++
It has:
URL routing with URL-parameters mapping
Support for Swagger-UI endpoint annotations.
Object-Mapping with JSON support.
Example endpoint:
ENDPOINT("GET", "users/{name}", getUserByName, PATH(String, name)) {
auto userDto = UserDto::createShared();
userDto->name = name;
return createDtoResponse(Status::CODE_200, userDto);
}
Curl:
$ curl http://localhost:8000/users/john
{"name":"john"}
You may want to take a look at webcc.
It's a lightweight C++ HTTP client and server library for embedding purpose based on Boost.Asio (1.66+).
It's quite promising and actively being developed.
It includes a lot of examples to demonstrate how to create a server and client.
There is a JIRA project resolved the support of JSON in AXIS2/C .
I implemented in my project and I managed with the writer (Badgerfish convention) but still I am trying to manage with the reader.It seems more complicated managing with the stack in the memory.
JSON and JSONPath are supported for both C and C++ in gsoap with a new code generator and a new JSON API to get you started quickly.
Several JSON, JSON-RPC and REST examples are included. Memory management is automatic.
The code generator can be useful. Take for example the json.org menu.json snippet:
{ "menu": {
"id": "file",
"value": "File",
"popup": {
"menuitem": [
{"value": "New", "onclick": "CreateNewDoc()"},
{"value": "Open", "onclick": "OpenDoc()"},
{"value": "Close", "onclick": "CloseDoc()"}
]
}
}
}
The gsoap command jsoncpp -M menu.json generates this code to populate a JSON value:
value x(ctx);
x["menu"]["id"] = "file";
x["menu"]["value"] = "File";
x["menu"]["popup"]["menuitem"][0]["value"] = "New";
x["menu"]["popup"]["menuitem"][0]["onclick"] = "CreateNewDoc()";
x["menu"]["popup"]["menuitem"][1]["value"] = "Open";
x["menu"]["popup"]["menuitem"][1]["onclick"] = "OpenDoc()";
x["menu"]["popup"]["menuitem"][2]["value"] = "Close";
x["menu"]["popup"]["menuitem"][2]["onclick"] = "CloseDoc()";
Also reading parsed JSON values and JSONPath code can be generated by this tool.
EDIT
To clarify, the jsoncpp command-line code generator shows the API code to read and write JSON data by using a .json file as a template, which I found is useful to save time to write the API code to populate and extract JSON data. JSONPath query code can also be generated with this tool.
For web service in C, you can leverage library like ulfius, civetweb:
https://github.com/babelouest/ulfius
https://github.com/civetweb/civetweb/blob/master/docs/Embedding.md
For web service in C++, you can leverage library like libhv, restbed:
https://github.com/ithewei/libhv
https://github.com/Corvusoft/restbed