I want to test my AWS code locally so I have to set a proxy to a AWS client.
There is a proxy host (http://user#pass:my-corporate-proxy.com:8080) set in my environment via a variable HTTPS_PROXY.
I didn't find a way how to set the proxy as whole so I came up with this code:
AmazonSNS sns = AmazonSNSClientBuilder.standard()
.withClientConfiguration(clientConfig(System.getenv("HTTPS_PROXY")))
.withRegion(Regions.fromName(System.getenv("AWS_REGION")))
.withCredentials(new DefaultAWSCredentialsProviderChain())
.build();
ClientConfiguration clientConfig(String proxy) {
ClientConfiguration configuration = new ClientConfiguration();
if (proxy != null && !proxy.isEmpty()) {
Matcher matcher = Pattern.compile("(\\w{3,5})://((\\w+):(\\w+)#)?(.+):(\\d{1,5})").matcher(proxy);
if (!matcher.matches()) {
throw new IllegalArgumentException("Proxy not valid: " + proxy);
}
configuration.setProxyHost(matcher.group(5));
configuration.setProxyPort(Integer.parseInt(matcher.group(6)));
configuration.setProxyUsername(matcher.group(3));
configuration.setProxyPassword(matcher.group(4));
}
return configuration;
}
The whole method clientConfig is only boilerplate code.
Is there any elegant way how to achieve this?
As far as I can tell while using AWS SDK V1 (1.11.840), if you have environment variables such as HTTP(S)_PROXY or http(s)_proxy set at runtime, or properties like http(s).proxyHost, proxyPort, proxyUser, and proxyPassword passed to your application, you don't have to set any of that. It gets automatically read into the newly created ClientConfigiration.
As, such you'd only want to set the ProxyAuthenticationMethod, if needed.
ClientConfiguration clientConfig(ProxyAuthenticationMethod authMethod) {
ClientConfiguration conf = new ClientConfiguration();
List<ProxyAuthenticationMethod> proxyAuthentication = new ArrayList<>(1);
proxyAuthentication.add(authMethod);
conf.setProxyAuthenticationMethods(proxyAuthentication);
return conf;
}
ProxyAuthenticationMethod can be ProxyAuthenticationMethod.BASIC or DIGEST or KERBEROS or NTLM or SPNEGO
I can confirm setting the parameters "http(s).proxyHost" (and others) work out of the box, you need however to specify a port, otherwise AWS SDK (1) will not pick it up.
java -Dhttps.proxyHost=proxy.company.com -Dhttps.proxyPort=8080 -Dhttps.proxyUser=myUsername -Dhttps.proxyPassword=myPassword <app>
Username & passsword are optional.
See for more info:
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/net/doc-files/net-properties.html
and
What Java properties to pass to a Java app to authenticate with a http proxy
Related
I'm refactoring a bunch of code from AWS V1 sdk to V2 and am stuck creating the proxy configuration for a GlueClient.
V1 code:
ClientConfiguration clientConfiguration = new ClientConfiguration();
clientConfiguration.setProxyPort(8443);
clientConfiguration.setProtocol(Protocol.HTTPS);
V2 code:
ClientOverrideConfiguration.builder(). ??? .build();
I can't find any good examples of how to do this. There do not seem to be any proxy options in the replacement class for V2
I'm going to assume you're using the Java SDK based on classnames, in which case you set the proxy configuration on the HTTPClientBuilder. Here's an example with the default (Apache) client-builder, taken from this gist, which shows a complete program and compares it to the V1 proxy configuration.
ProxyConfiguration config = ProxyConfiguration.builder()
.endpoint(new URI("http://localhost:3128"))
.addNonProxyHost("169.254.169.254")
.useSystemPropertyValues(Boolean.FALSE)
.build();
ApacheHttpClient.Builder clientBuilder = ApacheHttpClient.builder()
.proxyConfiguration(config);
StsClient client = StsClient.builder()
.httpClientBuilder(clientBuilder)
.build();
You might also be interested in the ways to automatically configure a proxy via environment variables or system properties, which I've documented here. TL;DR: it's a hot mess.
I am following the directions at https://learn.microsoft.com/en-us/azure/app-service/app-service-key-vault-references
Essentially, I am attempting to protect the storage connection string used for AzureWebJobsDashboard and AzureWebJobsStorage behind an Azure Key vault secret. I cannot use my injected KeyVault service to fetch it because my service container has not been built yet. So I found (through the link above) I could express this intent using a "#Microsoft.KeyVault()" expression in configuration. Here is an example where I moved the configuration to inline code to keep it terse:
.ConfigureHostConfiguration(configurationBuilder =>
{
configurationBuilder
.AddConfiguration(configuration)
.AddInMemoryCollection(new Dictionary<string, string>
{
["ConnectionStrings:AzureWebJobsDashboard"] = "#Microsoft.KeyVault(SecretUri=https://host.vault.azure.net/secrets/secret-name/ec545689445a40b199c0e0a956f16fca)",
["ConnectionStrings:AzureWebJobsStorage"] = "#Microsoft.KeyVault(SecretUri=https://host.vault.azure.net/secrets/secret-name/ec545689445a40b199c0e0a956f16fca)",
});
})
If I run this, I get:
FormatException: No valid combination of account information found.
If I change the configuration values from the special annotation to the copied secret value from Key Vault (the blue copy button under the 'Show Secret Value' button), everything just works. This confirms to me the connection string I use is correct.
Also, I manually used KeyVaultClient w/AzureServiceTokenProvider to verify the process should work when running locally in Visual Studio, even before the host has been built. I am able to get the secret just fine. This tells me I have sufficient privileges to get the secret.
So now I am left wondering if this is even supported. There are pages which imply this is possible however, such as https://medium.com/statuscode/getting-key-vault-secrets-in-azure-functions-37620fd20a0b. At least for Azure Functions. I am using Azure Web Jobs which gets deployed as a console application with an ASP.NET Core service, and I cannot find an example with that configuration.
Can anybody clarify if what I am doing is supported? And if not, what is the advisable process for getting connection strings stored in Azure Key Vault before the Azure Web Jobs host has been built?
Thanks
I have gone through a lot of online resources and everything seems to indicate that the special decorated #Microsoft.KeyVault setting only works when the value lives in AppSettings on the Azure Portal, not in local configuration. Somebody please let me know if that is an incorrect assessment.
So to solve this problem, I came up with a solution which in all honesty, feels a little hacky because I am depending on the fact that the connection string is not read/cached from local configuration until the host is ran (not during build). Basically, the idea is to build a configuration provider for which I can set a value after the host has been built. For example:
public class DelayedConfigurationSource : IConfigurationSource
{
private IConfigurationProvider Provider { get; } = new DelayedConfigurationProvider();
public IConfigurationProvider Build(IConfigurationBuilder builder) => Provider;
public void Set(string key, string value) => Provider.Set(key, value);
private class DelayedConfigurationProvider : ConfigurationProvider
{
public override void Set(string key, string value)
{
base.Set(key, value);
OnReload();
}
}
}
A reference to this type gets added during host builder construction:
var delayedConfigurationSource = new DelayedConfigurationSource();
var hostBuilder = new HostBuilder()
.ConfigureHostConfiguration(configurationBuilder =>
{
configurationBuilder
.AddConfiguration(configuration)
.Add(delayedConfigurationSource);
})
...
And just make sure to set the configuration before running the host:
var host = hostBuilder.Build();
using (host)
{
var secretProvider = host.Services.GetRequiredService<ISecretProvider>();
var secret = await secretProvider.YourCodeToGetSecretAsync().ConfigureAwait(false);
delayedConfigurationSource.Set("ConnectionStrings:AzureWebJobsStorage", secret.Value);
await host.RunAsync().ConfigureAwait(false);
}
If there is a more intuitive way to accomplish this, please let me know. If not, the connection string design is plain silly.
We need to get the service name under which a fargate task runs so we can perform some per service configuration (we have one service per customer, and use the service name to identify them).
By knowing the service discovery namespace for our cluster and the Task IP address, we are able to do find out the service by doing the following.
Get the task ip address, eaither by calling http://169.254.170.2/v2/metadata endpoint or by using the ECS_ENABLE_CONTAINER_METADATA method in my follow-up answer.
With the cluster namespace we call AWS.ServiceDiscovery.listNamespaces
From there we extract the nameSpace id.
We pass that to AWS.ServiceDiscovery.listServices
We pass the id of each service to AWS.ServiceDiscovery.listInstances
We flat map the results of that and look for an instance that matches our IP address above.
Voilà! that record gives us the service name.
Works fine, it just seems like a super circuitous path! I'm just wondering whether there is some shorter way to get this information.
Here's a working C# example in two steps. It gets the taskARN from the metadata to retrieve the task description, and then reads its Group property, which contains the name of the service. It uses AWSSDK.ECS to get the task description and Newtonsoft.Json to parse the JSON.
private static string getServiceName()
{
// secret keys, should be encoded in license configuration object
var ecsClient = new AmazonECSClient( ACCESS_KEY, SECRET_KEY );
var request = new DescribeTasksRequest();
// need cluster here if not default
request.Cluster = NAME_OF_CLUSTER;
request.Tasks.Add( getTaskArn() );
var asyncResponse = ecsClient.DescribeTasksAsync( request );
// probably need this synchronously for application to proceed
asyncResponse.Wait();
var response = asyncResponse.Result;
string group = response.Tasks.Single().Group;
// group returned in the form "service:[NAME_OF_SERVICE]"
return group.Remove( 0, 8 );
}
private static string getTaskArn()
{
// special URL for fetching internal Amazon information for ECS instances
string url = #"http://169.254.170.2/v2/metadata";
string metadata = getWebRequest( url );
// use JObject to read the JSON return
return JObject.Parse( metadata )[ "TaskARN" ].ToString();
}
private static string getWebRequest( string url )
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create( url );
request.AutomaticDecompression = DecompressionMethods.GZip;
using HttpWebResponse response = (HttpWebResponse)request.GetResponse();
using Stream stream = response.GetResponseStream();
using StreamReader reader = new StreamReader( stream );
return reader.ReadToEnd();
}
You can get the service name from startedBy property of the task definition. Using boto sdk you can call a describe_tasks (or its equivalent in aws-cli: aws ecs describe-tasks) which will provide a
'startedBy': 'string'
The tag specified when a task is started. If the task is started by an Amazon ECS service, then the startedBy parameter contains the deployment ID of the service that starts it.
From:
boto3 ecs client
aws-cli ecs client
Hope it helps.
The answer above requires reading the container metadata that appears if you set the ECS_ENABLE_CONTAINER_METADATA environment variable in the task. The work flow is then:
Read the container metadata file ecs-container-metadata.json to get the taskArn
Call the aws.ecs.describe-tasks function to get the startedBy property
Call aws.servicediscover.get-service.
Two steps instead of three, if you don't count reading the metadata file. Better, to be sure, but I'm probably not going to change the way we do it for now.
I am using IdentityServer4 with .NET Core 2.0 on AWS's ElasticBeanstalk. I have a certificate for signing tokens. What's the best way to store this certificate and retrieve it from the application? Should I just stick it with the application files? Throw it in an environment variable somehow?
Edit: just to be clear, this is a token signing certificate, not an SSL certificate.
I don't really like the term 'token signing certificate' because it sounds so benign. What you have is a private key (as part of the certificate), and everyone knows you should secure your private keys!
I wouldn't store this in your application files. If someone gets your source code, they shouldn't also get the keys to your sensitive data (if someone has your signing cert, they can generate any token they like and pretend to be any of your users).
I would consider storing the certificate in AWS parameter store. You could paste the certificate into a parameter, which can be encrypted at rest. You then lock down the parameter with an AWS policy so only admins and the application can get the cert - your naughty Devs dont need it! Your application would pull the parameter string when needed and turn it into your certificate object.
This is how I store secrets in my application. I can provide more examples/details if required.
Edit -- This was the final result from Stu's guidance
The project needs 2 AWS packages from Nuget to the project
AWSSDK.Extensions.NETCORE.Setup
AWSSDK.SimpleSystemsManagement
Create 2 parameters in the AWS SSM Parameter Store like:
A plain string named /MyApp/Staging/SigningCertificate and the value is a Base64 encoded .pfx file
An encrypted string /MyApp/Staging/SigningCertificateSecret and the value is the password to the above .pfx file
This is the relevant code:
// In Startup class
private X509Certificate2 GetSigningCertificate()
{
// Configuration is the IConfiguration built by the WebHost in my Program.cs and injected into the Startup constructor
var awsOptions = Configuration.GetAWSOptions();
var ssmClient = awsOptions.CreateServiceClient<IAmazonSimpleSystemsManagement>();
// This is blocking because this is called during synchronous startup operations of the WebHost-- Startup.ConfigureServices()
var res = ssmClient.GetParametersByPathAsync(new Amazon.SimpleSystemsManagement.Model.GetParametersByPathRequest()
{
Path = "/MyApp/Staging",
WithDecryption = true
}).GetAwaiter().GetResult();
// Decode the certificate
var base64EncodedCert = res.Parameters.Find(p => p.Name == "/MyApp/Staging/SigningCertificate")?.Value;
var certificatePassword = res.Parameters.Find(p => p.Name == "/MyApp/Staging/SigningCertificateSecret")?.Value;
byte[] decodedPfxBytes = Convert.FromBase64String(base64EncodedCert);
return new X509Certificate2(decodedPfxBytes, certificatePassword);
}
public void ConfigureServices(IServiceCollection servies)
{
// ...
var identityServerBuilder = services.AddIdentityServer();
var signingCertificate = GetSigningCertificate();
identityServerBuilder.AddSigningCredential(signingCertificate);
//...
}
Last, you may need to set an IAM role and/or policy to your EC2 instance(s) that gives access to these SSM parameters.
Edit: I have been moving my web application SSL termination from my load balancer to my elastic beanstalk instance this week. This requires storing my private key in S3. Details from AWS here: Storing Private Keys Securely in Amazon S3
As I have read, Azure Webjobs SDK requires the value of AzureWebJobsDashboard to be defined in the connection string section of an App Service. If I do that manually by going to the portal, and configuring the values, my webjob works fine.
What I want now is to provide the value of the connectionString in KeyVault and provide a secretURI provided by KeyVault in the ApplicationSettings of the App Service. Thereafter, I want the SDK to parse the secretURI and obtain the secret stored underneath in KeyVault.
I have tried to do this but am not able to get a definite solution for this. Is there any way to do this, or is currently not supported ?
Here is working example of what Mike suggested:
var config = new JobHostConfiguration();
if (config.IsDevelopment)
{
config.UseDevelopmentSettings();
}
else
{
var azureServiceTokenProvider = new AzureServiceTokenProvider();
var vaultBaseUrl = ConfigurationManager.AppSettings["KeyVaultUri"];
var client = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
config.DashboardConnectionString = client.GetSecretAsync(vaultBaseUrl, "DashboardConnectionString").Result.Value;;
config.StorageConnectionString = client.GetSecretAsync(vaultBaseUrl, "StorageConnectionString").Result.Value;;
}
config.UseTimers();
var host = new JobHost(config);
References:
https://learn.microsoft.com/en-us/azure/key-vault/service-to-service-authentication
WebJobs SDK will not automatically get from KeyVault.
But you could also fetch from keyvault yourself and then programmatically set the result on the JobHostConfiguration.Dashboard property.
You could file a request here to have more direct integration: https://github.com/Azure/azure-webjobs-sdk