I am currently getting into Terraform and I am trying to structure the different resources that I am deploying by using tags and resource groups.
https://docs.aws.amazon.com/cli/latest/reference/resource-groups/index.html
I can easily add tags with Terraform and I can create the resource-group via aws cli but I really want to be able to do both with Terraform if possible.
The official Terraform docs currently seem to not support an aws_resource_group resource(I was able to find aws_inspector_resource_group and aws_iam_resource_group, which are different types of grouping resources) but I was wondering if anyone was able to achieve it via some kind of a workaround.
I would really appreciate any feedback on the matter.
Thanks in advance!
This has been released in aws provider 1.55.0: https://www.terraform.io/docs/providers/aws/r/resourcegroups_group.html
For anyone looking for a code example, try this:
resource "aws_resourcegroups_group" "code-resource" {
name = "code-resource"
resource_query {
query = <<JSON
{
"ResourceTypeFilters": [
"AWS::EC2::Instance"
],
"TagFilters": [
{
"Key": "Stage",
"Values": ["dev"]
}
]
}
JSON
}
}
Please update it to your liking and needs. also be sure to checkout the source documentation:
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/resourcegroups_group
Related
I'm using Sagemaker to generate to do preprocessing and generate training data and I'm following the Sagemaker API documentation here, but I don't see any way currently how to specify autoscaling within the EMR cluster. What should I include within the configuration argument that I pass to my spark_processor run() object? What shouldn't I include?
I'm aware of the this resource, but it doesn't seem comprehensive.
Below is my code; it is very much a "work-in-progress", but I would like to know if someone could provide me with or point me to a resource that shows:
Whether this PySparkProcessor object will manage autoscaling automatically. Should I put AutoScaling config within the configuration in the run() object?
An example of the full config that I can pass to the configuration variable.
Here's what I have so far for the configuration.
SPARK_CONFIG = \
{ "Configurations": [
{ "Classification": "spark-env",
"Configurations": [ {"Classification": "export"} ] }
]
}
spark_processor = PySparkProcessor(
tags=TAGS,
role=IAM_ROLE,
instance_count=2,
py_version="py37",
volume_size_in_gb=30,
container_version="1",
framework_version="3.0",
network_config=sm_network,
max_runtime_in_seconds=1800,
instance_type="ml.m5.2xlarge",
base_job_name=EMR_CLUSTER_NAME,
sagemaker_session=sagemaker_session,
)
spark_processor.run(
configuration=SPARK_CONFIG,
submit_app=LOCAL_PYSPARK_SCRIPT_DIR,
spark_event_logs_s3_uri="s3://{BUCKET_NAME}/{S3_PYSPARK_LOG_PREFIX}",
)
I'm used to interacting via Python more directly with EMR for these types of tasks. Doing that allows me to specify the entire EMR cluster config at once--including applications, autoscaling, EMR default and autoscaling roles--and then adding the steps to the cluster once it's created; however, much of this config seems to be abstracted away, and I don't know what remains or needs to be specified, specifically regarding the following config variables: AutoScalingRole, Applications, VisibleToAllUsers, JobFlowRole/ServiceRole etc.
I found the answer in the Sagemaker Python SDK github.
_valid_configuration_keys = ["Classification", "Properties", "Configurations"]
_valid_configuration_classifications = [
"core-site",
"hadoop-env",
"hadoop-log4j",
"hive-env",
"hive-log4j",
"hive-exec-log4j",
"hive-site",
"spark-defaults",
"spark-env",
"spark-log4j",
"spark-hive-site",
"spark-metrics",
"yarn-env",
"yarn-site",
"export",
]
Thus, specifying autoscaling, Visibility, and some other cluster level configurations seems not to be supported. However, the applications installed upon cluster start up seem to depend on the applications in the above list.
We're using python boto3 to detect entities out of text. Is it possible to add tags such as "product":"x" to the API calls?
comprehend = boto3.client(service_name='comprehend', region_name='us-east-1')
response = comprehend.detect_entities(Text=text, LanguageCode='en')
Today, Comprehend only supports Tags for persistent resources such as Custom models and endpoints. Comprehend inference jobs are not persistent and hence, don't support Tags. We'll take the feedback under consideration.
As per the documentation DetectEntities
Only following attributes are supported in the request
{
"EndpointArn": "string",
"LanguageCode": "string",
"Text": "string"
}
I am a beginner using AWS WAF, AWS Elasticsearch.
I want to change the AWS WAF log timestamp type from long(number) type to date type.
Because of long type timestamp not working in Kibana visualize and Dashboard time filtering.
I tried to this query code on the AWS ES Dev Tools.
But, I have just received this message.
How can I solve this problem?
Thanks a lot.
Would you show your original index queries? And by trying fixed this issued, you may wanna try reindex.
Usage:
POST _reindex
{
"source": {
"index": "test-timestamps"
},
"dest": {
"index": "test-timestamps-2"
}
}
I have a secret stored in AWS secret manager and trying to integrate that within terraform during runtime. We are using terraform 0.11.13 version, and updating to latest terraform is in the roadmap.
We all want to use the jsondecode() available as part of latest terraform, but need to get few things integrated before we upgrade our terraform.
We tried to use the below helper external data program suggested as part of https://github.com/terraform-providers/terraform-provider-aws/issues/4789.
data "external" "helper" {
program = ["echo", "${replace(data.aws_secretsmanager_secret_version.map_example.secret_string, "\\\"", "\"")}"]
}
But we ended up getting this error now.
data.external.helper: can't find external program "echo"
Google search didn't help much.
Any help will be much appreciated.
OS: Windows 10
It sounds like you want to use a data source for the aws_secretsmanager_secret.
Resources in terraform create new resources. Data sources in terraform reference the value of existing resources in terraform.
data "aws_secretsmanager_secret" "example" {
arn = "arn:aws:secretsmanager:us-east-1:123456789012:secret:example-123456"
}
data "aws_secretsmanager_secret_version" "example" {
secret_id = data.aws_secretsmanager_secret.example.id
version_stage = "example"
}
Note: you can also use the secret name
Docs: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/secretsmanager_secret
Then you can use the value from this like so:
output MySecretJsonAsString {
value = data.aws_secretsmanager_secret_version.example.secret_string
}
Per the docs, the secret_string property of this resource is:
The decrypted part of the protected secret information that was originally provided as a string.
You should also be able to pass that value into jsondecode and then access the properties of the json body individually.
but you asked for a terraform 0.11.13 solution. If the secret value is defined by terraform you can use the terraform state datasource to get the value. This does trust that nothing else is updating the secret other than terraform. But the best answer is to upgrade your terraform. This could be a useful stopgap until then.
As a recommendation, you can make the version of terraform specific to a module and not your whole organization. I do this through the use of docker containers that run specific versions of the terraform bin. There is a script in the root of every module that will wrap the terraform commands to come up in the version of terraform meant for that project. Just a tip.
I'm creating datasources/ datasets in code (boto3) but these don't show up in the console.
Even though the datasets are listed with list_data_sets, they don't seem to be available in the console.
I need to be able to create all the necessary datasets in code and then be able to use these to create new analyses/ dashboards in the console.
I'm using the Standard Edition of QuickSight.
Can this be done? Or, can it only be done in the Enterprise Edition? Or, not at all?
Thanks
According to the QuickSight pricing page "APIs" are not available in Standard Edition. Exactly what that means, I have no idea.
But, assuming it's possible to call create-data-set, one important thing to remember is that data set permissions are necessary in order for users to view them.
According to the boto docs, these permissions should be included in the following schema
Permissions=[
{
'Principal': 'string',
'Actions': [
'string',
]
},
]
In my code, I use the following to share with the all-users group (note the group principal, replace AWS_REGION and ACCOUNT_ID with your values)
Permissions= [
{
'Principal': 'arn:aws:quicksight:AWS_REGION:ACCOUNT_ID:group/default/all-users',
'Actions': [
'quicksight:DescribeDataSet',
'quicksight:DescribeDataSetPermissions',
'quicksight:PassDataSet',
'quicksight:DescribeIngestion',
'quicksight:ListIngestions'
]
}
],
I believe the same can be done for individual users, with an ARN resource of user/default/user.name instead of group/default/all-users.
For data sources, the set of permissions that I use is
'Actions': [
'quicksight:DescribeDataSource',
'quicksight:DescribeDataSourcePermissions',
'quicksight:UpdateDataSource',
'quicksight:UpdateDataSourcePermissions',
'quicksight:DeleteDataSource',
'quicksight:PassDataSource'
]