I am trying to switch profiles in code using appsettings.json for a .net core web service.
If I new up an instance of the client in code eg:
using var client = new AmazonSecretsManagerClient();
client.ListSecretsAsync(listRequest, default)
.GetAwaiter()
.GetResult();
this takes the default profile that was set up and works fine.
Instead of using the default profile, I want to control the profile from json file. I am trying to follow this:
https://docs.aws.amazon.com/sdk-for-net/latest/developer-guide/net-dg-config-netcore.html
Instead of newing up the client, I am using dependency injection
public SecretsService(IAmazonSecretsManager secretsManagerClient)
{
_secretsManagerClient = secretsManagerClient;
}
Here is the config:
"AWS": {
"Profile": "default",
"Region": "us-east-2"
}
When I make a call using,
_secretsManagerClient.ListSecretsAsync(listRequest, default)
.GetAwaiter()
.GetResult();
it gives an exception
No such host is known. (secretsmanager.defaultregion.amazonaws.com:443)
What am I missing?
Check your appsettings.Development.json file. Some templates have the region set as DefaultRegion and so you have to either update it or remove it to let appsettings.json take precedence.
Related
Getting error when trying to unload or count data from AWS Keyspace using dsbulk.
Error:
Operation COUNT_20221021-192729-813222 failed: Token metadata not present.
Command line:
$ dsbulk count/unload -k my_best_storage -t book_awards -f ./dsbulk_keyspaces.conf
Config:
datastax-java-driver {
basic.contact-points = [ "cassandra.us-east-2.amazonaws.com:9142"]
advanced.auth-provider {
class = PlainTextAuthProvider
username = "aw.keyspaces-at-XXX"
password = "XXXX"
}
basic.load-balancing-policy {
local-datacenter = "us-east-2"
}
basic.request {
consistency = LOCAL_QUORUM
default-idempotence = true
}
advanced {
request{
log-warnings = true
}
ssl-engine-factory {
class = DefaultSslEngineFactory
truststore-path = "./cassandra_truststore.jks"
truststore-password = "XXX"
hostname-validation = false
}
metadata {
token-map.enabled = false
}
}
}
dsbulk load - loading operator works fine...
I suspect the problem here is that your cluster is using the proprietary com.amazonaws.cassandra.DefaultPartitioner partitioner which most open-source tools and drivers don't recognise.
The DataStax Bulk Loader (DSBulk) tool uses the Cassandra Java driver under the hood to connect to Cassandra clusters. The Java driver uses the partitioner to determine which nodes own tokens [ranges]. Only the following Cassandra partitioners are supported:
Murmur3Partitioner
RandomPartitioner
ByteOrderedPartitioner
Since the Java driver doesn't know about DefaultPartitioner, it doesn't have a map of token range owners (token metadata) and so can't determine how to "split" the Cassandra ring to query the nodes.
As you already figured out, this doesn't affect the load command because it simply sends writes to coordinators and lets the coordinators figure out how the data is partitioned. But for unload and count commands which require reads, the Java driver can't determine which coordinators to pick for sub-range queries with an unsupported partitioner.
Maybe as a workaround you can try to disable token-awareness with:
$ dsbulk count [...]
--driver.advanced.metadata.token-map.enabled false
but I don't have an AWS Keyspaces cluster I could test and I'm doubtful it will work. In any case, you're welcome to try.
There is an outstanding DSBulk feature request to provide the ability to completely disable token-awareness (internal ticket ID DAT-622) but it is unassigned at the time of writing so I'm not in a position to provide any expectation on when it will be prioritised. Cheers!
Amazon Keyspaces now supports multiple partitioners including MurMr3Partitioner. See the following to update your partitioner. You will also want to set token-map.enabled to true.
metadata {
token-map.enabled = true
}
Additionally, if you are using VPC Endpoints you will need the following permissions to make sure that you will see available peers.
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"ListVPCEndpoints",
"Effect":"Allow",
"Action":[
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeVpcEndpoints"
],
"Resource":"*"
}
]
}
I would also recommend increasing the connection pool size for the data load process.
advanced.connection.pool.local.size = 3
Finally, I would recommend using AWS glue instead of DSBulk. DSBulk is single process tool and will not scale for larger data loads. Additionally, learning glue will be helpful in managing other aspects of the data lifecycle. See my example on how to unload/export data using AWS Glue.
I am following the directions at https://learn.microsoft.com/en-us/azure/app-service/app-service-key-vault-references
Essentially, I am attempting to protect the storage connection string used for AzureWebJobsDashboard and AzureWebJobsStorage behind an Azure Key vault secret. I cannot use my injected KeyVault service to fetch it because my service container has not been built yet. So I found (through the link above) I could express this intent using a "#Microsoft.KeyVault()" expression in configuration. Here is an example where I moved the configuration to inline code to keep it terse:
.ConfigureHostConfiguration(configurationBuilder =>
{
configurationBuilder
.AddConfiguration(configuration)
.AddInMemoryCollection(new Dictionary<string, string>
{
["ConnectionStrings:AzureWebJobsDashboard"] = "#Microsoft.KeyVault(SecretUri=https://host.vault.azure.net/secrets/secret-name/ec545689445a40b199c0e0a956f16fca)",
["ConnectionStrings:AzureWebJobsStorage"] = "#Microsoft.KeyVault(SecretUri=https://host.vault.azure.net/secrets/secret-name/ec545689445a40b199c0e0a956f16fca)",
});
})
If I run this, I get:
FormatException: No valid combination of account information found.
If I change the configuration values from the special annotation to the copied secret value from Key Vault (the blue copy button under the 'Show Secret Value' button), everything just works. This confirms to me the connection string I use is correct.
Also, I manually used KeyVaultClient w/AzureServiceTokenProvider to verify the process should work when running locally in Visual Studio, even before the host has been built. I am able to get the secret just fine. This tells me I have sufficient privileges to get the secret.
So now I am left wondering if this is even supported. There are pages which imply this is possible however, such as https://medium.com/statuscode/getting-key-vault-secrets-in-azure-functions-37620fd20a0b. At least for Azure Functions. I am using Azure Web Jobs which gets deployed as a console application with an ASP.NET Core service, and I cannot find an example with that configuration.
Can anybody clarify if what I am doing is supported? And if not, what is the advisable process for getting connection strings stored in Azure Key Vault before the Azure Web Jobs host has been built?
Thanks
I have gone through a lot of online resources and everything seems to indicate that the special decorated #Microsoft.KeyVault setting only works when the value lives in AppSettings on the Azure Portal, not in local configuration. Somebody please let me know if that is an incorrect assessment.
So to solve this problem, I came up with a solution which in all honesty, feels a little hacky because I am depending on the fact that the connection string is not read/cached from local configuration until the host is ran (not during build). Basically, the idea is to build a configuration provider for which I can set a value after the host has been built. For example:
public class DelayedConfigurationSource : IConfigurationSource
{
private IConfigurationProvider Provider { get; } = new DelayedConfigurationProvider();
public IConfigurationProvider Build(IConfigurationBuilder builder) => Provider;
public void Set(string key, string value) => Provider.Set(key, value);
private class DelayedConfigurationProvider : ConfigurationProvider
{
public override void Set(string key, string value)
{
base.Set(key, value);
OnReload();
}
}
}
A reference to this type gets added during host builder construction:
var delayedConfigurationSource = new DelayedConfigurationSource();
var hostBuilder = new HostBuilder()
.ConfigureHostConfiguration(configurationBuilder =>
{
configurationBuilder
.AddConfiguration(configuration)
.Add(delayedConfigurationSource);
})
...
And just make sure to set the configuration before running the host:
var host = hostBuilder.Build();
using (host)
{
var secretProvider = host.Services.GetRequiredService<ISecretProvider>();
var secret = await secretProvider.YourCodeToGetSecretAsync().ConfigureAwait(false);
delayedConfigurationSource.Set("ConnectionStrings:AzureWebJobsStorage", secret.Value);
await host.RunAsync().ConfigureAwait(false);
}
If there is a more intuitive way to accomplish this, please let me know. If not, the connection string design is plain silly.
I want to test my AWS code locally so I have to set a proxy to a AWS client.
There is a proxy host (http://user#pass:my-corporate-proxy.com:8080) set in my environment via a variable HTTPS_PROXY.
I didn't find a way how to set the proxy as whole so I came up with this code:
AmazonSNS sns = AmazonSNSClientBuilder.standard()
.withClientConfiguration(clientConfig(System.getenv("HTTPS_PROXY")))
.withRegion(Regions.fromName(System.getenv("AWS_REGION")))
.withCredentials(new DefaultAWSCredentialsProviderChain())
.build();
ClientConfiguration clientConfig(String proxy) {
ClientConfiguration configuration = new ClientConfiguration();
if (proxy != null && !proxy.isEmpty()) {
Matcher matcher = Pattern.compile("(\\w{3,5})://((\\w+):(\\w+)#)?(.+):(\\d{1,5})").matcher(proxy);
if (!matcher.matches()) {
throw new IllegalArgumentException("Proxy not valid: " + proxy);
}
configuration.setProxyHost(matcher.group(5));
configuration.setProxyPort(Integer.parseInt(matcher.group(6)));
configuration.setProxyUsername(matcher.group(3));
configuration.setProxyPassword(matcher.group(4));
}
return configuration;
}
The whole method clientConfig is only boilerplate code.
Is there any elegant way how to achieve this?
As far as I can tell while using AWS SDK V1 (1.11.840), if you have environment variables such as HTTP(S)_PROXY or http(s)_proxy set at runtime, or properties like http(s).proxyHost, proxyPort, proxyUser, and proxyPassword passed to your application, you don't have to set any of that. It gets automatically read into the newly created ClientConfigiration.
As, such you'd only want to set the ProxyAuthenticationMethod, if needed.
ClientConfiguration clientConfig(ProxyAuthenticationMethod authMethod) {
ClientConfiguration conf = new ClientConfiguration();
List<ProxyAuthenticationMethod> proxyAuthentication = new ArrayList<>(1);
proxyAuthentication.add(authMethod);
conf.setProxyAuthenticationMethods(proxyAuthentication);
return conf;
}
ProxyAuthenticationMethod can be ProxyAuthenticationMethod.BASIC or DIGEST or KERBEROS or NTLM or SPNEGO
I can confirm setting the parameters "http(s).proxyHost" (and others) work out of the box, you need however to specify a port, otherwise AWS SDK (1) will not pick it up.
java -Dhttps.proxyHost=proxy.company.com -Dhttps.proxyPort=8080 -Dhttps.proxyUser=myUsername -Dhttps.proxyPassword=myPassword <app>
Username & passsword are optional.
See for more info:
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/net/doc-files/net-properties.html
and
What Java properties to pass to a Java app to authenticate with a http proxy
I am writing a C++ application in Windows using the AWS C++ SDK and need help with Developer Authenticated Identities in order to upload/download files to/from S3 in my application.
We have a backend application using Cognito to get the temporary credentials for AWS (IdentityID & OpenIDToken). I know the ProviderName and the IdentityPoolID.
Rather than describe what I've attempted, I believe it's easier to show.
Below is the snippet. I am trying to do the equivalent of "Refresh Identity" which is available in C# and Java SDK's. C# and Java both use "RefreshIdentity" which defines an IdentityState(IdentityID, ProviderName, OpenIDToken, fromCacheFlag)
Based on some documentation I found, I created a new class (MyInheritedCognitoIdentityClient) derived from CognitoIdentityClient which overrides the GetId & GetOpenIdToken functions to return the IdentityID and openIdToken received back from the server applicaton. The GetID is called, but GetOpenIdToken is not called - as the documentation implies; instead there is a call to GetCredentialsForIdentity (which fails with NotAuthorizedException).
Below is my code snippet:
`{
std::shared_ptr<MyInheritedPersistentCognitoIdentityProvider> identityProvider = std::make_shared<MyInheritedPersistentCognitoIdentityProvider>();
identityProvider->setIdentityPool(myIdentityPoolID); // us-east-1:c373a2ca-b912-3839-a65c-8d4ce53d512e -> not real
identityProvider->setAccountId(myProviderName); // login.mycompany.net
std::shared_ptr<MyInheritedCognitoIdentityClient> cognitoIdentityClient =
std::make_shared<MyInheritedCognitoIdentityClient>(); // _cognitoID and _openIdToken
std::shared_ptr<Aws::Auth::AWSCredentialsProvider> cognitoCachCredProvider =
std::make_shared<Aws::Auth::CognitoCachingAuthenticatedCredentialsProvider>(identityProvider, cognitoIdentityClient);
Aws::S3::S3Client s3Client(cognitoCachCredProvider);
/* attempt to upload/download here */
}`
you need to call GetCredentialsForIdentity server side, if not anyone can login in your own created provider (I'm assuming login.mycompany.net is custom provider)
cppcms website doesn't include an example about using sessions in asynchronous mode. How I can create an asynchronous session management system using cppcms?
Added later:
I used this code for saving a session:
session()["name"] = ...
session().save();
and somewhere I placed this:
if(!session().load() || !session().is_set("name"))
std::cerr<<"error";
When I run program it shows error.
this is my config file (session section):
"expire": "renew",
"timeout": 604800,
"location": "both",
"client" : {
"hmac": "sha1",
"hmac_key": "...",
},
"server":{
"storage": "files"
}
See the section with heading "Now let's create our main asynchronous function", it does provide session and bind socket to session.
http://cppcms.com/wikipp/en/page/cppcms_1x_aio
Just read the manuals:
http://cppcms.com/cppcms_ref/latest/classcppcms_1_1session__interface.html#ae63e68dd2ec1d615f5a6a85bcee36605
You need to call session().load() before using session object.
By default the session configuration is disabled. Please enable that first. See the following for reference. http://cppcms.com/wikipp/en/page/cppcms_1x_config#session. After you have configured session. The rest is the same as I previously stated. Session management is described in detailed over here. http://cppcms.com/wikipp/en/page/secure_programming