Use KCL 1.* in Lambda: Credentials - amazon-web-services

Using the examples provided in https://github.com/aws/aws-sdk-java/tree/master/src/samples/AmazonKinesis I understand how to use the KCL to read KinesisEvents from my local machine.
I am trying to implement the same logic in Lambda Functions.
To set up the KinesisClientLibConfiguration, you need to provide an AWSCredentialsProvider.
Where do I get these AWSCredentials to create the kinesisClientLibConfiguration in a Lambda?
KinesisClientLibConfiguration kinesisClientLibConfiguration = new KinesisClientLibConfiguration(SAMPLE_APPLICATION_NAME, SAMPLE_APPLICATION_STREAM_NAME, credentialsProvider, workerId);
The full code of the handler looks like this:
public class ProcessKinesisRecords implements RequestHandler<KinesisEvent, Void> {
public static final String SAMPLE_APPLICATION_STREAM_NAME = "kinesis-s";
public static final String SAMPLE_APPLICATION_STREAM_REGION = "eu-west-1";
private static final String SAMPLE_APPLICATION_NAME = "SampleKinesisLambdaApplication";
private static final InitialPositionInStream SAMPLE_APPLICATION_INITIAL_POSITION_IN_STREAM =
InitialPositionInStream.LATEST;
private static ProfileCredentialsProvider credentialsProvider;
public Void handleRequest(KinesisEvent event, Context context) {
init();
int exitCode = 0;
try {
String workerId = InetAddress.getLocalHost().getCanonicalHostName() + ":" + UUID.randomUUID();
KinesisClientLibConfiguration kinesisClientLibConfiguration =
new KinesisClientLibConfiguration(SAMPLE_APPLICATION_NAME,
SAMPLE_APPLICATION_STREAM_NAME,
credentialsProvider,
workerId);
kinesisClientLibConfiguration.withInitialPositionInStream(SAMPLE_APPLICATION_INITIAL_POSITION_IN_STREAM);
kinesisClientLibConfiguration.withRegionName(SAMPLE_APPLICATION_STREAM_REGION);
IRecordProcessorFactory recordProcessorFactory = new AmazonKinesisApplicationRecordProcessorFactory();
Worker worker = new Worker(recordProcessorFactory, kinesisClientLibConfiguration);
System.out.printf("Running %s to process stream %s as worker %s...\n",
SAMPLE_APPLICATION_NAME,
SAMPLE_APPLICATION_STREAM_NAME,
workerId);
worker.run();
} catch (Throwable e) {
System.err.println("Caught throwable while processing data.");
e.printStackTrace();
}
System.exit(exitCode);
return null;
}
private static void init() {
// Ensure the JVM will refresh the cached IP values of AWS resources (e.g. service endpoints).
java.security.Security.setProperty("networkaddress.cache.ttl", "60");
credentialsProvider = new ProfileCredentialsProvider();
try {
credentialsProvider.getCredentials();
} catch (Exception e) {
throw new AmazonClientException("Cannot load the credentials", e);
}
}
}

Lambda does provide environment variables for the credentials:
https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html#lambda-environment-variables
Access them via the EnvironmentVariableCredentialsProvider:
https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/auth/EnvironmentVariableCredentialsProvider.java

Related

AddSingleton won't accept AddSerilog

I'm facing an issue where I'm trying to write a more smart console app with logging and configuration available.
This is what I have so far:
namespace Client
{
public class Program
{
public static IConfigurationRoot Configuration;
private static int Main(string[] args)
{
Log.Logger = new LoggerConfiguration()
.WriteTo.Console(Serilog.Events.LogEventLevel.Debug)
.MinimumLevel.Debug()
.Enrich.FromLogContext()
.CreateLogger();
try
{
MainAsync(args).ConfigureAwait(false);
return 0;
}
catch
{
return 1;
}
}
private static async Task MainAsync(string[] args)
{
// Create service collection
Log.Information("Creating service collection");
var serviceCollection = new ServiceCollection();
ConfigureServices(serviceCollection);
// Create service provider
Log.Information("Building service provider");
IServiceProvider serviceProvider = serviceCollection.BuildServiceProvider();
try
{
Log.Information("Starting service");
await serviceProvider.GetService<App>().Run();
Log.Information("Ending service");
}
catch (Exception ex)
{
Log.Fatal(ex, "Error running service");
throw;
}
finally
{
Log.CloseAndFlush();
}
}
private static void ConfigureServices(IServiceCollection services)
{
// Add logging
services.AddSingleton(
LoggerFactory.Create(
builder => { builder.AddSerilog(dispose: true); }));
services.AddLogging();
// Build configuration
Configuration = new ConfigurationBuilder()
.SetBasePath(Directory.GetParent(AppContext.BaseDirectory).FullName)
.AddJsonFile("appSettings.json", false)
.Build();
// Add access to generic IConfigurationRoot
services.AddSingleton(Configuration);
// Add app
services.AddTransient<App>();
}
}
}
I'm facing an issue in ConfigureServices method on line builder.AddSerilog for the life of me, I cannot figure out why it is not able to resolve AddSerilog
I was missing a package: Serilog.Extensions.Logging
For some reason, VS or even Resharper was not able to suggest this.

How to manage secret rotation used by spring boot app running on ECS in AWS cloud

My organization is running spring boot app on AWS ECS docker container which reads the credentials for Postgres sql from secrets manager in AWS during boot up. AS part of security complaince, we are rotating the secrets every 3 months. The spring boot app is loosing connection with the database and going down when the RDS credentials are rotated.we have to restart it in order to pick the new credentials to work properly. Is there any way I can read the credentials automatically once the credentials are rotated to avoid restarting the application manually?
After some research I found that the postgres database in AWS supports passwordless authentication using IAM roles. We can generate a token which is valid for 15 mins and can connect to database using that token. I prefer this way of connecting to database rather than using password for my database. More details about setting up password less authentication can be found here
Code example as below
import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
import com.amazonaws.services.rds.auth.GetIamAuthTokenRequest;
import com.amazonaws.services.rds.auth.RdsIamAuthTokenGenerator;
import org.apache.commons.lang3.StringUtils;
import org.apache.tomcat.jdbc.pool.ConnectionPool;
import org.apache.tomcat.jdbc.pool.PoolConfiguration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.net.URI;
import java.net.URISyntaxException;
import java.sql.SQLException;
import java.util.Properties;
public class RdsIamAuthDataSource extends org.apache.tomcat.jdbc.pool.DataSource {
private static final Logger LOGGER = LoggerFactory.getLogger(RdsIamAuthDataSource.class);
private static final int DEFAULT_PORT = 5432;
private static final String USESSL = "useSSL";
private static final String REQUIRE_SSL = "requireSSL";
private static final String BOOLEAN_TRUE = "true";
private static final String VERIFY_SERVER_CERTIFICATE = "verifyServerCertificate";
private static final String THREAD_NAME = "RdsIamAuthDataSourceTokenThread";
/**
* Constructor for RdsIamAuthDataSource.
* #param props {#link PoolConfiguration}
*/
public RdsIamAuthDataSource(PoolConfiguration props) {
this.poolProperties = props;
}
#Override
public ConnectionPool createPool() throws SQLException {
if (pool == null) {
return createPoolImpl();
} else {
return pool;
}
}
protected ConnectionPool createPoolImpl() throws SQLException {
synchronized (this) {
return pool = new RdsIamAuthConnectionPool(poolProperties);
}
}
private class RdsIamAuthConnectionPool extends ConnectionPool implements Runnable {
private RdsIamAuthTokenGenerator rdsIamAuthTokenGenerator;
private String host;
private String region;
private int port;
private String username;
private Thread tokenThread;
/**
* Constructor for RdsIamAuthConnectionPool.
* #param prop {#link PoolConfiguration}
* #throws SQLException {#link SQLException}
*/
public RdsIamAuthConnectionPool(PoolConfiguration prop) throws SQLException {
super(prop);
}
#Override
protected void init(PoolConfiguration prop) throws SQLException {
try {
final URI uri = new URI(prop.getUrl().substring(5));
this.host = uri.getHost();
this.port = uri.getPort();
if (this.port < 0) {
this.port = DEFAULT_PORT;
}
this.region = StringUtils.split(this.host,'.')[2];
this.username = prop.getUsername();
this.rdsIamAuthTokenGenerator = RdsIamAuthTokenGenerator.builder()
.credentials(new DefaultAWSCredentialsProviderChain())
.region(this.region)
.build();
updatePassword(prop);
final Properties props = prop.getDbProperties();
props.setProperty(USESSL, BOOLEAN_TRUE);
props.setProperty(REQUIRE_SSL, BOOLEAN_TRUE);
props.setProperty(VERIFY_SERVER_CERTIFICATE, BOOLEAN_TRUE);
super.init(prop);
this.tokenThread = new Thread(this, THREAD_NAME);
this.tokenThread.setDaemon(true);
this.tokenThread.start();
} catch (URISyntaxException e) {
LOGGER.error("Database URL is not correct. Please verify", e);
throw new RuntimeException(e.getMessage());
}
}
/**
* Refresh the token every 12 minutes.
*/
#Override
public void run() {
try {
while (this.tokenThread != null) {
Thread.sleep(12 * 60 * 1000);
updatePassword(getPoolProperties());
}
} catch (InterruptedException e) {
LOGGER.error("Background token thread interrupted", e);
}
}
#Override
protected void close(boolean force) {
super.close(force);
final Thread thread = tokenThread;
if (thread != null) {
thread.interrupt();
}
}
private void updatePassword(PoolConfiguration props) {
final String token = rdsIamAuthTokenGenerator.getAuthToken(GetIamAuthTokenRequest.builder()
.hostname(host)
.port(port)
.userName(this.username)
.build());
LOGGER.info("Updated IAM token for connection pool");
props.setPassword(token);
}
}
}
Supply the following DataSource as a spring bean. That's it. Now your application will automatically refresh credentials every 12 minutes
#Bean
public DataSource dataSource() {
final PoolConfiguration props = new PoolProperties();
props.setUrl("jdbc:postgresql://myapp.us-east-2.rds.amazonaws.com/myschema?ssl=true");
props.setUsername("rdsadminuser");
props.setDriverClassName("org.somedatabase.Driver");
return new RdsIamAuthDataSource(props);
}

Amazon Elasticsearch service 403-forbidden error

I am having trouble fetching result from my amazon elastic search cluster using the amazon java SDK and an IAm user credential. Now the issue is that when the PATH string is equal to "/" then I am able to fetch the result correctly but when I try with a different path for e.g "/private-search" then I get a 403 forbidden error. Even when for the path that has public access I am getting a 403 forbidden error for this IAm user but it works if I remove "signer.sign(requestToSign, credentials);" line in performSigningSteps method(for public resource only).
My policy in AWS gives this IAM user access to everything in my elastic search service. And also what can I do to avoid hard-coding the access key and secret key in source code?
private static final String SERVICE_NAME = "es";
private static final String REGION = "region-name";
private static final String HOST = "host-name";
private static final String ENDPOINT_ROOT = "http://" + HOST;
private static final String PATH = "/private-search";
private static final String ENDPOINT = ENDPOINT_ROOT + PATH;
private static String accessKey = "IAmUserAccesskey"
private static String secretKey = "IAmUserSecretkey"
public static void main(String[] args) {
// Generate the request
Request<?> request = generateRequest();
// Perform Signature Version 4 signing
performSigningSteps(request);
// Send the request to the server
sendRequest(request);
}
private static Request<?> generateRequest() {
Request<?> request = new DefaultRequest<Void>(SERVICE_NAME);
request.setContent(new ByteArrayInputStream("".getBytes()));
request.setEndpoint(URI.create(ENDPOINT));
request.setHttpMethod(HttpMethodName.GET);
return request;
}
private static void performSigningSteps(Request<?> requestToSign) {
AWS4Signer signer = new AWS4Signer();
signer.setServiceName(requestToSign.getServiceName());
signer.setRegionName(REGION);
AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
signer.sign(requestToSign, credentials);
}
private static void sendRequest(Request<?> request) {
ExecutionContext context = new ExecutionContext();
ClientConfiguration clientConfiguration = new ClientConfiguration();
AmazonHttpClient client = new AmazonHttpClient(clientConfiguration);
MyHttpResponseHandler<Void> responseHandler = new MyHttpResponseHandler<Void>();
MyErrorHandler errorHandler = new MyErrorHandler();
Void response = client.execute(request, responseHandler, errorHandler, context);
}
public static class MyHttpResponseHandler<T> implements HttpResponseHandler<AmazonWebServiceResponse<T>> {
#Override
public AmazonWebServiceResponse<T> handle(com.amazonaws.http.HttpResponse response) throws Exception {
InputStream responseStream = response.getContent();
String responseString = convertStreamToString(responseStream);
System.out.println(responseString);
AmazonWebServiceResponse<T> awsResponse = new AmazonWebServiceResponse<T>();
return awsResponse;
}
#Override
public boolean needsConnectionLeftOpen() {
return false;
}
}
public static class MyErrorHandler implements HttpResponseHandler<AmazonServiceException> {
#Override
public AmazonServiceException handle(com.amazonaws.http.HttpResponse response) throws Exception {
System.out.println("In exception handler!");
AmazonServiceException ase = new AmazonServiceException("exception.");
ase.setStatusCode(response.getStatusCode());
ase.setErrorCode(response.getStatusText());
return ase;
}
#Override
public boolean needsConnectionLeftOpen() {
return false;
}
}
public static String convertStreamToString(InputStream is) throws IOException {
// To convert the InputStream to String we use the
// Reader.read(char[] buffer) method. We iterate until the
// Reader return -1 which means there's no more data to
// read. We use the StringWriter class to produce the string.
if (is != null) {
Writer writer = new StringWriter();
char[] buffer = new char[1024];
try {
Reader reader = new BufferedReader(new InputStreamReader(is, "UTF-8"));
int n;
while ((n = reader.read(buffer)) != -1) {
writer.write(buffer, 0, n);
}
}
finally {
is.close();
}
return writer.toString();
}
return "";
}

WinRT App consume NAV web services and got this message

I did the following and got the below error msg:
The error message :
An exception of type 'System.AggregateException' occurred in mscorlib.dll but was not handled in user code
Additional information: One or more errors occurred.
If there is a handler for this exception, the program may be safely continued.
Question :
a) What seems to be the problems in above code as I just wanted to retrieve a record.
b) Must use Async Methods in WinRT or Windows store app?
c) Will below code able to retrieve record from Navision?
-----1------- Windows store App to access Nav Web Services
1.1 Added the service reference in WinRT App
1.2 Added a class1.cs in WinRT App
private async void btnImportCustomer_Click(object sender, RoutedEventArgs e)
{
Task _asyncCustomer = Class1.Customer.Listing.GetAsyncRecords("Y007");
### encounterd error here: ####
string g_strmsg = _asyncCustomer.Result.No + " “ +_asyncCustomer.Result.Name;
}
-----2---------- Class1.cs use inside WinRT App Project:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace MobileNAVSalesSystem
{
class Class1
{
public static string _webserviceurlpage = "http ://{0}:{1}/{2}/WS/{3}/Page/{4}";
public static string _webserviceurlcodeunit = "http://{0}:{1}/{2}/WS/{3}/Codeunit/{4}";
public static Uri _webserviceuripage = null;
public static Uri _webserviceuricodeunit = null;
#region Customer
public class Customer
{
public class Card
{
//Do something for Card Type
}
public class Listing
{
public static wsCustomerList.Customer_List_PortClient GetService()
{
_webserviceuripage = new Uri(string.Format(_webserviceurlpage, "msxxx", "7047", "DynamicsNAV_xxx", Uri.EscapeDataString("Global xxx Pte. Ltd."), "Customer List"));
System.ServiceModel.BasicHttpBinding _wSBinding = new System.ServiceModel.BasicHttpBinding();
_wSBinding.Security.Mode = System.ServiceModel.BasicHttpSecurityMode.TransportCredentialOnly;
_wSBinding.Security.Transport.ClientCredentialType = System.ServiceModel.HttpClientCredentialType.Windows;
_wSBinding.MaxBufferSize = Int32.MaxValue;
_wSBinding.MaxReceivedMessageSize = Int32.MaxValue;
//_wSBinding.UseDefaultWebProxy = false;
wsCustomerList.Customer_List_PortClient _ws = new wsCustomerList.Customer_List_PortClient(_wSBinding, new System.ServiceModel.EndpointAddress(_webserviceuripage));
_ws.ClientCredentials.Windows.AllowedImpersonationLevel = System.Security.Principal.TokenImpersonationLevel.Delegation;
_ws.ClientCredentials.Windows.ClientCredential = new System.Net.NetworkCredential("xxx","xxxx", "companyName");
return _ws;
}
//-------------------------- Using Async Methods
public static async Task GetAsyncRecords(string _No)
{
wsCustomerList.Customer_List_PortClient _ws = GetService();
wsCustomerList.Customer_List _List = (await _ws.ReadAsync(_No)).Customer_List;
if (_ws.State == System.ServiceModel.CommunicationState.Opened)
await _ws.CloseAsync();
return _List;
}
public static async Task GetAsyncRecords(wsCustomerList.Customer_List_Filter[] _filters)
{
wsCustomerList.Customer_List_PortClient _ws = GetService();
wsCustomerList.Customer_List[] _List;
List _filterArray = new List();
_filterArray.AddRange(_filters);
_List = (await _ws.ReadMultipleAsync(_filterArray.ToArray(), null, 0)).ReadMultiple_Result1;
if (_ws.State == System.ServiceModel.CommunicationState.Opened)
await _ws.CloseAsync();
return _List;
}
public static async Task GetAsyncRecords(wsCustomerList.Customer_List_Filter[] _filters, string _bookmarkkey)
{
wsCustomerList.Customer_List_PortClient _ws = GetService();
wsCustomerList.Customer_List[] _List;
List _filterArray = new List();
_filterArray.AddRange(_filters);
_List = (await _ws.ReadMultipleAsync(_filterArray.ToArray(), _bookmarkkey, 0)).ReadMultiple_Result1;
if (_ws.State == System.ServiceModel.CommunicationState.Opened)
await _ws.CloseAsync();
return _List;
}
public static async Task GetAsyncRecords(wsCustomerList.Customer_List_Filter[] _filters, string _bookmarkkey, int _setsize)
{
wsCustomerList.Customer_List_PortClient _ws = GetService();
wsCustomerList.Customer_List[] _List;
List _filterArray = new List();
_filterArray.AddRange(_filters);
_List = (await _ws.ReadMultipleAsync(_filterArray.ToArray(), _bookmarkkey, _setsize)).ReadMultiple_Result1;
if (_ws.State == System.ServiceModel.CommunicationState.Opened)
await _ws.CloseAsync();
return _List;
}
}
}
#endregion
}
//--- end namespace
}
i know it is some time ago this question was posted, but others might stumble across it, so here goes:
a) What seems to be the problems in above code as I just wanted to retrieve a record.
it seems like your return type is incorrect.
b) Must use Async Methods in WinRT or Windows store app?
Yes, when using windows mobile platforms(windows store apps and windows phone apps), you have to use asynchronous calls.
c) Will below code able to retrieve record from Navision?
Hard to tell, but to me it seems like your data you try to retrieve is in a incorrect format. Ill give you a simple example from one of my current projects where I retrieve a login:
private async void Button_Click(object sender, RoutedEventArgs e)
{
await call();
}
private async Task call()
{
BasicHttpBinding binding = new BasicHttpBinding();
NetworkCredential cred = new NetworkCredential("username", "password", "domain");
WS_PortClient ws = new WS_PortClient(binding, new EndpointAddress("Webservice-URL"));
binding.Security.Mode = BasicHttpSecurityMode.TransportCredentialOnly;
binding.Security.Transport.ClientCredentialType = HttpClientCredentialType.Ntlm;
ws.ClientCredentials.Windows.ClientCredential = cred;
CheckLogin_Result s = await ws.CheckLoginAsync("parameter");
string k = s.return_value.ToString();
MessageDialog d = new MessageDialog(k, "message");
await d.ShowAsync();
}
Hope it helps!

RavenDB keeps throwing a ConcurrencyException

I keep getting a ConcurrencyException trying to update the same document multiple times in succession. PUT attempted on document '<id>' using a non current etag is the message.
On every save from our UI we publish an event using MassTransit. This event is sent to the subscriberqueues, but I put the Eventhandlers offline (testing offline subscribers). Once the eventhandler comes online the queue is read and the messages are processed as intended.
However since the same object is in the queue multiple times the first write succeeds, the next doesn't and throws this concurrencyexception.
I use a factory class to have a consistent IDocumentStore and IDocumentSession in all my applications. I specifically set the UseOptimisticConcurrency = false in the GetSession() method.
public static class RavenFactory
{
public static IDocumentStore CreateDocumentStore()
{
var store = new DocumentStore() { ConnectionStringName = "RavenDB" };
// Setting Conventions
store.Conventions.RegisterIdConvention<MyType>((db, cmd, e) => e.MyProperty.ToString());
store.Conventions.RegisterAsyncIdConvention<MyType>((db, cmd, e) => new CompletedTask<string>(e.MyProperty.ToString()));
// Registering Listeners
store
.RegisterListener(new TakeNewestConflictResolutionListener())
.RegisterListener(new DocumentConversionListener())
.RegisterListener(new DocumentStoreListener());
// Initialize and return
store.Initialize();
return store;
}
public static IDocumentSession GetSession(IDocumentStore store)
{
var session = store.OpenSession();
session.Advanced.UseOptimisticConcurrency = false;
return session;
}
}
The eventhandler looks like this. The IDocumentSession gets injected using Dependency Injection.
Here is the logic to get an instance of IDocumentSession.
private static void InitializeRavenDB(IUnityContainer container)
{
container.RegisterInstance<IDocumentStore>(RavenFactory.CreateDocumentStore(), new ContainerControlledLifetimeManager());
container.RegisterType<IDocumentSession, DocumentSession>(new PerResolveLifetimeManager(), new InjectionFactory(c => RavenFactory.GetSession(c.Resolve<IDocumentStore>())));
}
And here is the actual EventHandler which has the ConcurrencyException.
public class MyEventHandler:Consumes<MyEvent>.All, IConsumer
{
private readonly IDocumentSession _session;
public MyEventHandler(IDocumentSession session)
{
if (session == null) throw new ArgumentNullException("session");
_session = session;
}
public void Consume(MyEvent message)
{
Console.WriteLine("MyEvent received: Id = '{0}'", message.MyProperty);
try
{
_session.Store(message);
_session.SaveChanges();
}
catch (Exception ex)
{
var exc = ex.ToString();
// Deal with concurrent writes ...
throw;
}
}
}
I want to ignore any concurrencyexception for now until we can sort out with the business on how to tackle concurrency.
So, any ideas why I get the ConcurrencyException? I want the save to happen no matter whether the document has been updated before or not.
I am unfamiliar with configuring Unity, but you always want Singleton of the IDocumentStore. Below, I have coded the Singleton out manually, but I'm sure Unity would support it:
public static class RavenFactory
{
private static IDocumentStore store;
private static object syncLock = new object();
public static IDocumentStore CreateDocumentStore()
{
if(RavenFactory.store != null)
return RavenFactory.store;
lock(syncLock)
{
if(RavenFactory.store != null)
return RavenFactory.store;
var localStore = new DocumentStore() { ConnectionStringName = "RavenDB" };
// Setting Conventions
localStore .Conventions.RegisterIdConvention<MyType>((db, cmd, e) => e.MyProperty.ToString());
localStore .Conventions.RegisterAsyncIdConvention<MyType>((db, cmd, e) => new CompletedTask<string>(e.MyProperty.ToString()));
// Registering Listeners
localStore
.RegisterListener(new TakeNewestConflictResolutionListener())
.RegisterListener(new DocumentConversionListener())
.RegisterListener(new DocumentStoreListener());
// Initialize and return
localStore.Initialize();
RavenFactory.store = localStore;
return RavenFactory.store;
}
}
// As before
// public static IDocumentSession GetSession(IDocumentStore store)
//
}