I am developing an ASP.NET Core Web API project. In my project, I am using Hangfire to run the background task. Therefore, I am configuring the Hangfire to use the database like this.
public void ConfigureServices(IServiceCollection services)
{
services.AddHangfire(configuration =>
{
configuration.UseSqlServerStorage("Server=(LocalDB)\\MSSQLLocalDB;Integrated Security=true;");
});
//
}
In the above code, I am using Local DB. Now, I am trying to use AWS RDS database since I am deploying my application on the AWS Elastic Beanstalks. I created a function for getting the connection
public static string GetRDSConnectionString()
{
string dbname = "ebdb";
if(string.IsNullOrEmpty(dbname)) return null;
string username = "admin";
string password = "password";
string hostname = "cxcxcxcx.xcxcxcxc.eu-west-2.rds.amazonaws.com:1234";
string port = "1234";
return "Data Source=" + hostname + ";Initial Catalog=" + dbname + ";User ID=" + username + ";Password=" + password + ";";
}
I got the above code from the official AWS documentation. In the above code, what I am not clear is the database name, is the database name always be "ebdb"? I tried to find out the database name. But could not. In the tutorial, it is saying to use ebdb. So, I used it.
Then in configuration, I changed to this.
configuration.UseSqlServerStorage(AppConfig.GetRDSConnectionString());
When I run the code, it is giving me this error.
Win32Exception: The parameter is incorrect
Unknown location
SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 25 - Connection string is not valid)
System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, object providerInfo, bool redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, bool applyTransientFaultHandling)
Win32Exception: The parameter is incorrect
Basically, it cannot connect to the database when I run my application. But I set the correct credentials. the only thing I double is the database name (ebdb). What is wrong with my configuration and what is wrong? How can I fix it?
Calling a few things out here just incase...
You have your port specified both in your host variable and as a separate port variable...but never use port.
Can you confirm that you are able to access your SQLServer via another means, such as from SQL Management Studio?
RDS uses SSL by default now for connections, my .NET is rusty but would you need to inform the connection string that it needs to run over a secure protocol?
& finally, regarding the AWS Security Group on your RDS instance. Have you opened up the correct port to your machine/network/IP?
This is the screenshot of the RDS db instance security group section in the console.
Related
We have an application that is using AWS MemoryDB for Redis. We have setup a cluster with one shard and two nodes. One of the nodes (named 0001-001) is a primary read/write while the other one is a read replica (named 0001-002).
After deploying the application, connecting to MemoryDB sometimes fails when we use the cluster endpoint connection string to connect. If we restart the application a few times it suddenly starts working. It seems to be random when it succeeds or not. The error we get is the following:
Endpoint Unspecified/ourapp-memorydb-cluster-0001-001.ourapp-memorydb-cluster.xxxxx.memorydb.eu-west-1.amazonaws.com:6379 serving hashslot 6024 is not reachable at this point of time. Please check connectTimeout value. If it is low, try increasing it to give the ConnectionMultiplexer a chance to recover from the network disconnect. IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=0,Free=32767,Min=2,Max=32767), Local-CPU: n/a
If we connect directly to the primary read/write node we get no such errors.
If we connect directly to the read replica it always fails. It even gets the error above, compaining about the "0001-001" node.
We use .NET Core 6
We use Microsoft.Extensions.Caching.StackExchangeRedis 6.0.4 which depends on StackExchange.Redis 2.2.4
The application is hosted in AWS ECS
StackExchangeRedisCache is added to the service collection in a startup file :
services.AddStackExchangeRedisCache(o =>
{
o.InstanceName = redisConfiguration.Instance;
o.ConfigurationOptions = ToRedisConfigurationOptions(redisConfiguration);
});
...where ToRedisConfiguration returns a basic ConfigurationOptions object :
new ConfigurationOptions()
{
EndPoints =
{
{ "clustercfg.ourapp-memorydb-cluster.xxxxx.memorydb.eu-west-1.amazonaws.com", 6379 } // Cluster endpoint
},
User = "username",
Password = "password",
Ssl = true,
AbortOnConnectFail = false,
ConnectTimeout = 60000
};
We tried multiple shards with multiple nodes and it also sometimes fail to connect to the cluster. We even tried to update the dependency StackExchange.Redis to 2.5.43 but no luck.
We could "solve" it by directly connecting to the primary node, but if a failover occurs and 0001-002 becomes the primary node we would have to manually change our connection string, which is not acceptable in a production environment.
Any help or advice is appreciated, thanks!
I have been trying to connect to my EMR cluster from java code to run a presto query. Until now I created a "maven project", and added "presto dependancy" in the "pom.xml". I have been referring this link for the program
https://gist.github.com/nagataka/2c2d9fa49b03e8556faf85345b43f59c
I have two questions:
1.) How do I connect to the EMR cluster with username and password like in the " conn = DriverManager.getConnection(DB_URL, USER, PASS);" used in the above reference. Because I use a ".ppk key" to validate the connection. I dunno how to give the key in this context.
2.) How do I run a simple "show tables;" query on Presto.
The following is my program:
package presto.presto_sample;
import java.net.URI;
import java.net.URISyntaxException;
import java.sql.*;
/**
* Hello world!
*
*/
public class App
{
static final String JDBC_DRIVER = "com.facebook.presto.jdbc.PrestoDriver";
static final String DB_URL = "jdbc:presto://ec2-18-191-128-219.us-east-2.compute.amazonaws.com:8889/hive/default";
public static void main( String[] args )
{
System.out.println( "Hello World!" );
}
}
Properties properties = new Properties();
properties.setProperty("user", "test");
properties.setProperty("password", "secret");
Connection connection = DriverManager.getConnection(url, properties);
i guess ppk can be another setProperty
see How to execute Presto query using Java API?
You don't need to pass PPK key in the connection string properties. Your code may not be working because the firewall might be blocking you to connect to the server. To solve this you would have to add an inbound rule in the master security group to allow access to your IP.
See this to add an inbound rule, Cannot connect/query from Presto on AWS EMR with Java JDBC
For checking out the tables:
Set the String object "SQL" from the referenced program you are using and set its value to
either "show tables from hive.default" (ex.) or if you have set catalog and schema name in connection property then just set it to "show tables".
As I tried to run the chat app from localhost connected to MySQL database which had been coded with PHP via WebSocket it was successful.
Also when I tried to run from the PuTTY terminal logged into SSH credentials, it was displaying as Server Started with the port# 8080
ubuntu#ec3-193-123-96:/home/admin/web/ec3-193-123-96.eu-central-1.compute.amazonaws.com/public_html/application/libraries/server$ php websocket_server.php
PHP Fatal error: Uncaught React\Socket\ConnectionException: Could not bind to tcp://0.0.0.0:8080: Address already in use in /home/admin/web/ec3-193-123-96.eu-central-1.compute.amazonaws.com/public_html/application/libraries/vendor/react/socket/src/Server.php:29
Stack trace:
#0 /home/admin/web/ec3-193-123-96.eu-central-1.compute.amazonaws.com/public_html/application/libraries/vendor/cboden/ratchet/src/Ratchet/Server/IoServer.php(70): React\Socket\Server->listen(8080, '0.0.0.0')
#1 /home/admin/web/ec3-193-123-96.eu-central-1.compute.amazonaws.com/public_html/application/libraries/server/websocket_server.php(121): Ratchet\Server\IoServer::factory(Object(Ratchet\Http\HttpServer), 8080)
#2 {main}
thrown in /home/admin/web/ec3-193-123-96.eu-central-1.compute.amazonaws.com/public_html/application/libraries/vendor/react/socket/src/Server.php on line 29
ubuntu#ec3-193-123-96:/home/admin/web/ec3-193-123-96.eu-central-1.compute.amazonaws.com/public_html/application/libraries/server$
So I tried to change the port#8080 to port# 8282, it was successful
ubuntu#ec3-193-123-96:/home/admin/web/ec3-193-123-96.eu-central-1.compute.amazonaws.com/public_html/application/libraries/server$ php websocket_server.php
Keeping the shell script running, open a couple of web browser windows, and open a Javascript console or a page with the following Javascript:
var conn = new WebSocket('ws://0.0.0.0:8282');
conn.onopen = function(e) {
console.log("Connection established!");
};
conn.onmessage = function(e) {
console.log(e.data);
};
From the browser console results:
WebSocket connection to 'ws://5.160.195.94:8282/' failed: Error in
connection establishment: net::ERR_CONNECTION_TIMED_OUT
websocket_server.php
<?php
use Ratchet\Server\IoServer;
use Ratchet\Http\HttpServer;
use Ratchet\WebSocket\WsServer;
use MyApp\Chat;
require dirname(__DIR__) . '/vendor/autoload.php';
$server = IoServer::factory(
new HttpServer(
new WsServer(
new Chat()
)
),
8282
);
$server->run();
I even tried to assign Public IP and Private IP, but with no good it resulted in the same old result?
This was the composer files generated after executing and adding src folder $composer require cboden/ratchet
composer.json(On AmazonWebServer)
{
"autoload": {
"psr-4": {
"MyApp\\": "src"
}
},
"require": {
"cboden/ratchet": "^0.4.1"
}
}
composer.json(On localhost)
{
"autoload": {
"psr-4": {
"MyApp\\": "src"
}
},
"require": {
"cboden/ratchet": "^0.4.3"
}
}
How am I suppose to resolve/overcome while connecting it from the WebSocket especially from the hosted server with the domain name such as
http://ec3-193-123-96.eu-central-1.compute.amazonaws.com/
var conn = new WebSocket('ws://localhost:8282');
From the Security Group
Under Inbound tab
Under Outbound tab
When it comes to a connectivity issue with an EC2 there are few things you need to check to find the root cause.
SSH into the EC2 instance that the application is running and make sure you can access it from within the EC2 instance. If it works then its a network related issue that we need to solve.
If step 1 was successful. You have now identified it is a network issue to solve this you need to check the following.
Check if an Internet Gateway is created and attached to your VPC.
Next check if your subnets routing table has its default route pointing to the internet gateway. check this link to complete this and the above step.
Check your subnets Network ACLs rules to see if ports are not blocked
finally, you would want to check your Instances Security group as you have shown.
If you need access via a EC2 dns you will need to provision your ec2 instance in a public subnet and assign an elastic IP
If an issue still exists check if the EC2 status checks pass, or try provisioning a new instance.
I am trying to connect to an AWS Redshift database from a lambda function using c#, dotnet core 2.0, and npgsql. I am having difficulty with SSL.
I have created two non-publicly-accessible Redshift databases in a dedicated VPC. The lambda executes in the same VPC. The two databases are identical in every way except that one has the "force SSL" parameter set to true.
Using the following code snippet, I can access the non-SSL database just fine:
using (var conn = new NpgsqlConnection ("Host=x; Port=5439; Username=x;
Password=x;Database=xxx")
{
Console.WriteLine("Redshift pre-Open!");
conn.Open();
Console.WriteLine("Redshift: post-Open!");
...
}
When I access the SSL database, I get the "missing hba.conf" error message - seems standard, I've seen it before ...
When I append to the connection string: "ssl Mode=Require;Server Compatibility Mode=Redshift;Trust Server Certificate=true"
the conn.open statement hangs, and the second write statement never shows up in cloudwatch.
And yet ... this connection statement works when accessing the same database thru a rest API and C#/dotnetcore 2 WEBAPI (same runtime environment), with
an EC2 instance and load balancer.
A Python lambda connecting to the SSL database, in the same environment - subnets, security groups, lambda triggers, lambda parameters, ... is working just fine.
The csproj references Amazon.Lambda.Core 1.0.0, Amazon.Lambda.Serialization.Json 1.1.0, and
Npgsql.EntityFrameworkCore.PostgreSQL 2.0.1.
I'd try Wireshark, maybe, in another environment - but running as a lambda, I'm not sure how best to debug. I've tried many permutations and combinations, and I wouldn't put it past myself to be missing something blindingly obvious,
but I absolutely do not see why hangs. Thank you.
I am using IdentityServer4 with .NET Core 2.0 on AWS's ElasticBeanstalk. I have a certificate for signing tokens. What's the best way to store this certificate and retrieve it from the application? Should I just stick it with the application files? Throw it in an environment variable somehow?
Edit: just to be clear, this is a token signing certificate, not an SSL certificate.
I don't really like the term 'token signing certificate' because it sounds so benign. What you have is a private key (as part of the certificate), and everyone knows you should secure your private keys!
I wouldn't store this in your application files. If someone gets your source code, they shouldn't also get the keys to your sensitive data (if someone has your signing cert, they can generate any token they like and pretend to be any of your users).
I would consider storing the certificate in AWS parameter store. You could paste the certificate into a parameter, which can be encrypted at rest. You then lock down the parameter with an AWS policy so only admins and the application can get the cert - your naughty Devs dont need it! Your application would pull the parameter string when needed and turn it into your certificate object.
This is how I store secrets in my application. I can provide more examples/details if required.
Edit -- This was the final result from Stu's guidance
The project needs 2 AWS packages from Nuget to the project
AWSSDK.Extensions.NETCORE.Setup
AWSSDK.SimpleSystemsManagement
Create 2 parameters in the AWS SSM Parameter Store like:
A plain string named /MyApp/Staging/SigningCertificate and the value is a Base64 encoded .pfx file
An encrypted string /MyApp/Staging/SigningCertificateSecret and the value is the password to the above .pfx file
This is the relevant code:
// In Startup class
private X509Certificate2 GetSigningCertificate()
{
// Configuration is the IConfiguration built by the WebHost in my Program.cs and injected into the Startup constructor
var awsOptions = Configuration.GetAWSOptions();
var ssmClient = awsOptions.CreateServiceClient<IAmazonSimpleSystemsManagement>();
// This is blocking because this is called during synchronous startup operations of the WebHost-- Startup.ConfigureServices()
var res = ssmClient.GetParametersByPathAsync(new Amazon.SimpleSystemsManagement.Model.GetParametersByPathRequest()
{
Path = "/MyApp/Staging",
WithDecryption = true
}).GetAwaiter().GetResult();
// Decode the certificate
var base64EncodedCert = res.Parameters.Find(p => p.Name == "/MyApp/Staging/SigningCertificate")?.Value;
var certificatePassword = res.Parameters.Find(p => p.Name == "/MyApp/Staging/SigningCertificateSecret")?.Value;
byte[] decodedPfxBytes = Convert.FromBase64String(base64EncodedCert);
return new X509Certificate2(decodedPfxBytes, certificatePassword);
}
public void ConfigureServices(IServiceCollection servies)
{
// ...
var identityServerBuilder = services.AddIdentityServer();
var signingCertificate = GetSigningCertificate();
identityServerBuilder.AddSigningCredential(signingCertificate);
//...
}
Last, you may need to set an IAM role and/or policy to your EC2 instance(s) that gives access to these SSM parameters.
Edit: I have been moving my web application SSL termination from my load balancer to my elastic beanstalk instance this week. This requires storing my private key in S3. Details from AWS here: Storing Private Keys Securely in Amazon S3