I have to create an alert when a human user tries to access a DB instance in AWS using a service account. Programmatic access should be fine and no need to be alerted.
Could anyone suggest a best possible solution to achieve this?
You can't, at least not directly.
There are two ways to access an Aurora database:
Via a TCP connection, using the Postgres or MySQL connection protocol.
Via the RDS Data API.
In both cases, there is a program at the other end of the connection, and the database has no way to determine whether that program is an business application, a user-written program connecting using a connection library, a user making API calls from a Jupyter notebook, or a user typing directly into psql.
The best that you can get is an indirect indication.
For example, if you use usernames and passwords to access the database, and store that information in a Secrets Manager secret, then you can use CloudTrail to find all calls to GetSecretValue and alert based on user identity. You can do the same thing for ExecuteStatement if using the Data API, but I don't believe that there's a CloudTrail event if you're using IAM-generated tokens for authorization.
However, even that has limitations. First, because you have to wait 15 minutes for events to appear in CloudTrail (which makes it a forensic tool, not a good alerting tool). Second, because there are ways to conceal your true identity (although it's not that easy with the Data API).
The real solution to your problem (which you have not described) will be an architecture that makes it difficult to create ad hoc database connections, and a culture that discourages such behavior.
Related
Following Coursera Architecting with Google Kubernetes Engine for switching to Service Account.
It says create and download a key file and authenticate using the key. Is this the common way in GCP? There will be many keys created by developers and downloaded to many laptops or servers scattering the keys in many places, which seems to be not secure manner.
Answering your question, yes. The service accounts are the common way to authenticate in GCP.
There are two different service account types, and the recommendation is to use the second one:
User Managed Service Accounts: to authenticate you will then need a “password” that comes in the form of Service Account Key (json file), and if you leak the service account key, the service account can be considered compromised.
Using keys implies that you are in charge of their lifecycle and security, and it’s a lot to ask because:
You need a robust system for secrets distribution.
You need to implement a key rotation policy.
You need to implement safeguards to prevent key leaks.
Google Managed Service Account: Google Managed Service Accounts, are SAs for which you don’t need to generate keys and your applications can just assume their identity. No keys are involved: the VM will continuously request short lived authorization tokens from the metadata service.
Documentation
NO, no and no, don't use service account key file. As you smell it, you are right, it's a terrible thing for the security.
Today, there are several way to prevent the service account key usage, even if, in some corner case, you need them.
I have wrote bunch of articles on that topics:
the limits
the service account credential API
and a fight against a Google dev advocate and one of his article
Because YES, even Google tutorials, courses, documentation (...) promote that bad practice for years and continue. It was my nightmare in my previous company, and I increased my knowledge and skill to prevent key usage and find workarounds. Let me know your use case, I will try to help your the most
I am trying to write an application in Python.
Through this application I want to create AWS Cognito users and provide services like user Sign-in, Forgot password, etc.
As I understand, boto3, is the standard Python library for accessing AWS APIs, from Python.
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html
This library needs storing of AWS credentials ( Access key and secure access keys) on the host machine.
Can this be avoided?
I want to distribute this Python application to my users.
I am checking, if I can avoid this configuration of AWS credentials on every user's host.
Is there any alternative option to boto3 library?
If you absolutely need to access internal AWS API's you need to log in to AWS. Access keys is one way, it's also possible to use aws-adfs command line tool to log in though active directory, but that requires your AWS/AD administrators to do some additional setup on their side.
I would suggest looking into writing a client-server / web applications that would be hosted within AWS and only expose relevant functionality to authenticated users.
If costs are an issue for a hosted application, look into lambdas, as there you pay only for cpu/memory time. In case of setting management app it will probably not even exceed free tier.
My library is a CLI utility, and people get it by running pip install [libname]. I would like to automatically record exceptions that occur when people use it and store these logs in the cloud. I have found services that should do just that: AWS CloudWatch, GCP Stackdriver.
However, while looking at their API it appears that I would have to ship my private key in order for the library to authenticate to my account. This doesn't sound right and I am warned by the cloud providers not to do this.
Example from GCP fails, requires credentials:
from google.cloud import logging
client = logging.Client()
logger = client.logger('log_name')
logger.log_text('A simple entry') # API call
While python library exposes source, I understand that any kind of authentication I ship would bear the risk of people sending any fake logs, but this is OK to me, as I would just limit the spending on my account for the (unexpected) case that somebody does just that. Of Course the credentials that ship with the library should be restricted to logging only.
Any example of how to enable logging to a cloud service from user machines?
For Azure Application Insights' "Instrumentation Key" there is a very good article about that subject here: https://devblogs.microsoft.com/premier-developer/alternative-way-to-protect-your-application-insights-instrumentation-key-in-javascript/
While I'm not familiar with the offerings of AWS or GCP, I would assume similar points are vaild.
Generally speaking: While the instrumentation key is a method of authentication, it is not considered a very secret key in most scenarios. The worst damage somebody can do is to send unwanted logs. They cannot read any data or overwrite anything with that key. And you already stated above that you are not really worried in your case about the issue of unwated logs.
So, as long as you are using an App Insights instance only for one specific application / purpose, I would say you are fine. You can still further aggregate that data in the background with data from different sources.
To add an concrete example to this: This little tool from Microsoft (the specific use case does not matter here), collects telemetry as well and sends it to Azure Application Insights - if the user does not opt out. I won't point to the exact code line but their instrumentation key is checked-in to that public GitHub repo for anybody to find.
Alternatively, the most secure way would be to send data from the
browser to your custom API on your server then forward to Application
Insights resource with the correct instrumentation key (see diagram
below).
(Source: the link above)
App Insights SDK for python is here btw: https://github.com/microsoft/ApplicationInsights-Python
To write logs to Stackdriver requires credentials. Anonymous connections to Stackdriver are NOT supported.
Under no circumstances give non-privileged users logging read permissions. Stackdriver records sensitive information in Stackdriver Logs.
Google Cloud IAM provides the role roles/logging.logWriter. This role gives users just enough permissions to write logs. This role does not grant read permissions.
The role roles/logging.logWriter is fairly safe. A user can write logs, but cannot read, overwrite or delete logs. I say fairly safe as there is private information stored in the service account. I would create a separate project only for Stackdriver logging with no other services.
The second issue with providing external users access is cost. Stackdriver logs are $0.50 per GiB. You would not want someone uploading a ton of logfile entries. Make sure that you monitor external usage. Create an alert to monitor costs.
Creating and managing service accounts
Chargeable Stackdriver Products
Alert on Stackdriver usage
Stackdriver Permissions and Roles
I want to build an application using Amazon Web Services (AWS).
The way the application should work is this;
I make a program that lets the user import a large file in an external format and send it to AWS (S3?) in my own format.
Next many users can access the data from web and desktop applications.
I want to charge per user accessing the data.
The problem is that the data on AWS must be in an unintelligible format or the users may copy the data over to another AWS account where I can not charge them. In other words the user need to do some "decrypting" of the data before they can be used. On the web this must be done in JavaScript which is plaintext and would allow the users to figure out my unintelligible format.
How can I fix this problem?
Is there for instance a built in encryption/decryption mechanism?
Alternatively is there some easy way in AWS to make a server that decrypts the data using precompiled code that I upload to AWS?
In general when you don't want your users to access your application's raw data you just don't make that data public. You should build some sort of server-side process that reads the raw data and serves up what the user is requesting. You can store the data in a database or in files on S3 or wherever you want, just don't make it publicly accessible. Then you can require a user to login to your application in order to access the data.
You could host such a service on AWS using EC2 or Elastic Beanstalk or possibly Lambda. You could also possibly use API Gateway to manage access to the services you build.
Regarding your specific question about a service on AWS that will encrypt your public data and then decrypt it on the fly, there isn't anything that does that out of the box. You would have to build such a service and host it on Amazon, but I don't think that is the right way to go about this at all. Just don't make your data publicly accessible in the first place, and make all requests for data go through some service to verify that the user should be able to access the data. In your case that would mean verifying that the user has paid to access the data they are requesting.
So this might be a silly question, but what is the point of using Amazon SQS if it requires a private and public key? If the client has the private and public key they could probably discover the keys via decompile or some other means...
The only secure way I could think of would be to use a proxy (like php) that has the private and public keys. But then what is the point of using SQS in the first place? The main benefit of SQS (I can see) is that it can scale upwards and you don't have to worry about how many messages you are receiving. But if you are going to be using a proxy then you will have to scale that too... I hope my concerns make sense?
Thanks
Your concerns would be valid if you had to give your secret key out for clients to pull data from the queue. However the typical workflow involves using your AWS account ID for creating and modifying queues and perhaps pushing data onto the queues. Then you can set permissions with either the SQS addPermission action or setup a more finely controlled access policy. This means you would give read access to only a specific AWS account or to anonymous usage, but you would not allow for other modifications.
So basically you have a couple options. You could compile in the AWS public and private keys which you have setup in advanced that has restricted permissions on your client application. A better approach in my opinion is to make the public and private key files a configurable option on your client and tell the users of the client they are responsible for getting their own AWS account and keys and they can tell you what their AWS key is and you can give them as fine grain control as you want on a per client basis.
These resources would be good for you to look at:
Using the Access Policy Language
Controlling User Access to your AWS account
addPermission action for SQS