We have a workflow in AWS that pulls twitter related data through API. We are doing it for over 100 twitter accounts and we expect the number to increase exponentially. Each twitter account has its own API key and so we want to use each key to pull data relating to that twitter account.
Our worklow previous used AWS secret but we only stored one account before but because we are planning to run it dynamically for separate accounts using their own API keys, we are thinking of using Hashicorp vault to store
We are wondering if anyone can point to best way we can enable the integration ? Is this the best approach or is there other scalable approach?
Hashicorp provides a Lambda extension as a way to integrate with Vault
https://learn.hashicorp.com/tutorials/vault/aws-lambda
This AWS blog post also details other options in the space and their advantages / disadvantages
https://aws.amazon.com/blogs/compute/choosing-the-right-solution-for-aws-lambda-external-parameters/
Related
Currently, we use AWS IAM User permanent credentials to transfer customers' data from our company's internal AWS S3 buckets to customers' Google BigQuery tables following BigQuery Data Transfer Service documentation.
Using permanent credentials possesses security risks related to the data stored in AWS S3.
We would like to use AWS IAM Role temporary credentials, which require the support of a session token on the BiqQuery side to get authorized on the AWS side.
Is there a way that the BigQuery Data Transfer Servce can use AWS IAM roles or temporary credentials to authorise against AWS and transfer data?
We considered Omni framework (https://cloud.google.com/bigquery/docs/omni-aws-cross-cloud-transfer) to transfer data from S3 to BQ, however, we faced several concerns/limitations:
Omni framework targets data analysis use-case rather than data transfer from external services. This concerns us that the design of Omni framework may have drawbacks in relation to data transfer at high scale
Omni framework currently supports only AWS-US-EAST-1 region (we require support at least in AWS-US-WEST-2 and AWS-EU-CENTRAL-1 and corresponding Google regions). This is not backward compatible with current customers' setup to transfer data from internal S3 to customers' BQ.
Our current customers will need to signup for Omni service to properly migrate from the current transfer solution we use
We considered a workaround with exporting data from S3 through staging in GCS (i.e. S3 -> GCS -> BQ), but this will also require a lot of effort from both customers and our company's sides to migrate to the new solution.
Is there a way that the BigQuery Data Transfer Servce can use AWS IAM roles or temporary credentials to authorise against AWS and transfer data?
No unfortunately.
The official Google BigQuery Data Transfer Service only mentions AWS access keys all throughout the documentation:
The access key ID and secret access key are used to access the Amazon S3 data on your behalf. As a best practice, create a unique access key ID and secret access key specifically for Amazon S3 transfers to give minimal access to the BigQuery Data Transfer Service. For information on managing your access keys, see the AWS general reference documentation.
The irony of the Google documentation is that while it refers to best practices and links to the official AWS docs, it actually doesn't endorse best practices and ignores what AWS mention:
We recommend that you use temporary access keys over long term access keys, as mentioned in the previous section.
Important
Unless there is no other option, we strongly recommend that you don't create long-term access keys for your (root) user. If a malicious user gains access to your (root) user access keys, they can completely take over your account.
You have a few options:
hook into both sides manually (i.e. link up various SDKs and/or APIs)
find an alternative BigQuery-compatible service, which does as such
accept the risk of long-term access keys.
In conclusion, Google is at fault here of not following security best practices and you - as a consumer - will have to bear the risk.
We have a site for our customers to log onto to get their relevant data. We have set it up on AWS using Cognito for user authentication. Each customer navigates to the same URL, enters their credentials, and then gets shown their own information. One of our customers has a corporate policy for any SaaS offering requiring a SSO (using SAML2.0). Our other customers do not need the SSO mechanism.
I have read through the documents AWS provides: (https://docs.aws.amazon.com/singlesignon/index.html) but these appear to be focused on a single corporation with AWS accounts for services provided by AWS. I have not been able to find any articles that address the situation.
Specific questions I have:
Is the AWS SSO mechanism the correct mechanism to use to achieve the goals? I have read in one Q&A that it is better to manipulate this through Cognito (but I cannot find the relevant article to link here).
If we set up one company to use SSO, can other companies use the credentials we set up to go to the same site?
Can we set up multiple companies to use the SSO separately, or will the application of a second SAML overwrite the first? (this doesn't seem likely as their would be updates to applicable users).
Any articles that can help point me in the best direction is greatly appreciated
AWS SSO would be a different AWS service you would have to integrate your application with.
If you're already using Cognito, you should be adding their SAML provider as a Cognito identity pool instead of adding AWS SSO.
I'm working on a Slack app which will have to store access token per each customer using the app (ex. 1000 teams using it = 1000 tokens). Token enables the app to access Slack API for customers workspace and will be used frequently every day.
App will be running on AWS, using Lambda's and DynamoDB.
What would be the best practice to store those access tokens securly?
I cannot find any strict recomendation for this scenario. Was thinking initially to put those in DynamoDB in a dedicated table but thinking now if I should use other AWS services for that use case. I've checked Secrets Manager but looks like a rather expensive option and not sure if it applies to my scenario.
Appreciate any suggestions.
I would probably use a dedicated DynamoDB table for this purpose. At a minimum, I would configure it to use a KMS CMK to encrypt the data at-rest, and also restrict access to the table through fairly granular IAM permissions in your AWS account. If you also wanted to encrypt each value separately you could look into client-side encryption.
Your findings on the Secrets Manager costs are a good point. You could also look at Systems Manager Parameter Store as an alternative that is generally cheaper than Secrets Manager. Secrets Manager does have the added security of being able to set an IAM resource policy on the secret itself.
Ultimately it's up to you to determine how secure your solution needs to be, and how much you are willing to pay for that. You could even spin up an AWS HSM to encrypt the values, but that would increase the cost by quite a bit.
For our product we are currently storing customer credentials hashed in db (3 tier architecture) . We want the authentication to be done at 1st tier itself ,which aws solution can be used for this ,May be AWS HSM but what changes need to be done at app layer to do this .
This is a website
using cloudfront to route across across edge
using database replication
also we have active-active multi region .
any suggestions would be useful
thanks
I agree that some further details on your architecture would help. Is this a web application, mobile app, other fat client? How are you achieving the active-active multi-region architecture at the DB? I would like to suggest AWS Cognito but the multi-region needs become a bit more complex in that scenario.
Today how do you determine which region your users are routed to? If using AWS Cognito you'd likely need to create a user pool per region but this means your users would need to be routed to the correct user pool based on their region.
I have had great luck with AWS Cognito identities from web, mobile, and fat client apps and have even used many of the Lambda integrations with Cognito for commercial grade applications. Some good examples -
http://docs.aws.amazon.com/cognito/latest/developerguide/using-amazon-cognito-user-identity-pools-javascript-examples.html
http://docs.aws.amazon.com/cognito/latest/developerguide/walkthrough-using-the-ios-sdk.html
http://docs.aws.amazon.com/cognito/latest/developerguide/setting-up-android-sdk.html
I currently am running my app on CloudKit by Apple. I would like to start using Amazon Web Services instead but I am not sure which part I should be using.
The app currently allows the user to log in on multiple devices and save a date value.
It then allows all of the user to fetch the same images and strings from CloudKit.
Should I be using AWS Cognito or S3 or something else?
Are there any tutorials for this?
Thank you
Look at Amazon Cognito for authenticating and identifying your users and delivering AWS Credentials to your users' devices. These AWS Credentials enable your users to securely access your AWS Resources. See How Amazon Cognito Keeps Mobile App Users' Data Safe.
Amazon Cognito also provides data synchronization capabilities that enable you to save user-specific information that only the current user can read. Look at Amazon DynamoDB for storing application-wide data that is shared with all users (like CloudKit database/records). Cognito also integrates with DynamoDB to enable fine-grained access control that might be interesting for you to review too. Use Amazon S3 for storing large objects (like CloudKit "Assets") like images or other files.