We have applications for multiple tenants on our AWS account and would like to distinguish between them in different IAM roles. In most places this is already possible by limiting resource access based on naming patterns.
For CloudWatch log groups of SageMaker training jobs however I have not seen a working solution yet. The tenants can choose the job name arbitrarily, and hence the only part of the LogGroup name that is available for pattern matching would be the prefix before the job name. This prefix however seems to be fixed to /aws/sagemaker/TrainingJobs.
Is there a way to change or extend this prefix in order to make such limiting possible? Say, for example /aws/sagemaker/TrainingJobs/<product>-<stage>-<component>/<training-job-name>-... so that a resource limitation like /aws/sagemaker/TrainingJobs/<product>-* becomes possible?
I think it is not possible to change the log streams names for any of the SageMaker services.
Related
I need to generate a report of all AWS Services that were provisioned after a certain date (say last 3 months).
AWS Service Catalog seems relevant here; but can this be used only if the services were provisioned using CloudFormation Templates?
We did our provisioning using Terraform - can AWS Service Catalog still be used to generate an inventory?
If not, is there an alternate way to generate this report?
You can try to use the Resource Groups for that https://eu-central-1.console.aws.amazon.com/resource-groups/home?region=eu-central-1#
There you will find the Tag Editor https://eu-central-1.console.aws.amazon.com/resource-groups/tag-editor/find-resources?region=eu-central-1 and list all of your resources.
If you have tagged your resources, you can filter by them. Alternative solution would be to tag all resources with the current date...wait one day...search again and find resources without the specific date tag. So you will find the differences.
To automate this, you can use e.g. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/resourcegroupstaggingapi.html#client
To get a full solution, you can use Tag Editor, get all resources and request the resources itself with the specific API of each resource, e.g. EC2, Lambda, RDS, etc.
This could be time consuming, so maybe a solution like from aquasec could fit your needs.
I'm trying to solve a problem with AWS IAM policies.
I need to allow certain users to only delete/modify resources that are tagged with their particular username (This I've solved) while also being able to create any new aws resource.
The part I haven't solved is need to be able to create resources without ability modifying any existing resources (unless they have the right tag).
Is there an existing AWS policy example that allows a user to create any resource (without granting delete/modify)? Is there a way to allow this without having to list every single aws offering and continuously update it for new offerings?
AdministratorAccess will give all rights to create all services.
See https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.html#jf_administrator
I managed to solve this problem with a rather ugly solution, but as far as I can tell it's the only solution.
I found a list of all aws actions: https://github.com/rvedotrc/aws-iam-reference
I then parsed out potentially troubling functions like anything with Delete or Terminate in the action name. I used vim/grep for this.
After that I broke that up into multiple aws_iam_group_policy statements. Each statement was attached to a corresponding group. The target users are then added to each of those groups.
Unfortunately, this is pretty ugly and required 5 different groups and policies, but it's the solution I arrived at.
Is there a quick way to find out which regions have any resources in my account? I'm specifically using the AWS .NET SDK but the answer likely applies to other AWS SDKs and the CLI since they all seem to be just wrappers to the REST API. I can obviously run all the List* methods across all regions but I'm thinking there must be a more optimal way to decide whether to query the entire region or not. Maybe something in billing, but it also needs to be relatively up-to-date, maybe within the last 5 minutes or so. Any ideas?
There is no single way to list all resources in an AWS account or in multiple regions.
Some people say that Resource Groups are a good way to list resources, but I don't think they include "everything" in an account.
AWS Config does an excellent job of keeping track of resources and their history, but it is also limited in the types of resources it tracks.
My favourite way to list resources is to use nccgroup/aws-inventory: Discover resources created in an AWS account. It's a simple HTML/JavaScript file that makes all the 'List' calls for you and shows them in a nicely formatted list.
I'm creating a bunch of application resources with AWS CloudFormation, and when the resources are created, CloudFormation adds a hash at the end of the name to make it unique.
i.e. If you wanted to create a Kinesis stream names MyStream, the actually name would be something like my-stack-MyStream-1F8ISNCLP0W4O.
I want to be able to programmatically access the resources without having to know the hash, without having to query AWS for my resources to match the names myself, and without manual steps. Does anybody know a convenient way to use AWS resources in your application programmatically and predictably?
Here are the less ideal options I can think of:
Set a tag on the resource (i.e. name -> MyStream) and query AWS to get the actual resource name.
Query AWS for a list of resources names and look for a partial match on the expected name.
After you create your resources, manually copy the actual names into your config file (probably the sanest of these options)
You can use the CloudFormation API to get a list of resources in your stack. This will give you a list of logical ids (i.e. the name in your CloudFormation template without the hash) and matching physical ids (with the stack name and hash). Using the AWS CLI, this will show a mapping between the two ids:
aws cloudformation describe-stack-resources
--query StackResources[].[LogicalResourceId,PhysicalResourceId]
--stack-name <my-stack>
CloudFormation APIs to do the same query are provided in all the various language SDKs provided by Amazon.
You can use this as an alternative to #1, by querying CloudFormation at runtime, or #3, by querying CloudFormation at buildtime and embedding the results in a config file. I don't see any advantage to using your own tags over simply querying the CF API. #2 will cause problems if you want two or more stacks from the same template to coexist.
I've used both the runtime and build time approaches. The build time approach lets you remove the dependency on or knowledge of CloudFormation, but needs stack specific information in your config file. I like the runtime approach to allow the same build to be deployed to multiple stacks and all it needs is the stack name to find all the related resources.
I'm creating a stack with CloudFormation. When you create log groups, it automatically adds prefixes and suffixes to my log group names. For example, if I try to create log group MyLogGroup, it creates log group my-stack-name-MyLogGroup-EEWJYSCJRK2V.
I understand that for a lot of use cases, this might be desired to differentiate the same resources for different stacks. However, my team has different accounts for our different stacks, so there will be no overlap. Having dynamic prefixes and suffixes makes it hard to reference log groups from static files (i.e. CloudWatch Logs agent config file).
Is there a way to make sure that resources get named EXACTLY what I put and not add a prefix or suffix?
We have run into this same issue with our AWS ecosystem and after speaking to several folks at AWS, this is by design and is not modifiable right now.
Depending on the complexity of what you are trying to do, I would recommend replacing CloudFormation with some Lambda functions to manage the resources (can be done cross account with sts:AssumeRole).
Yes its possible. For instance in our cloud formation template we create the cloud watch conf file with log_group_name and log_stream_name parameter set to combination of different parameters. Our log groups are created without prefix and postfix. See following example:
"log_group_name = "MyLogGroup\n",
"log_stream_name = {instance_id}/", "MyLogGroup", ".log\n",