I would like to find a list of dynamic thing groups. I can see the type of field when I go to one of the thing groups in AWS IoT Core. How do I search and find a list of thing groups which has a Type as Dynamic?
e.g.
When I visit one of the Thing Group present in IoT Core.
You do not have a description for the thing group yet.
Created
Jul 26, 2019 11:21:44 AM -0700
Type
Static
0 Attributes
I tried a few variants, but they did not work.
Type: Dynamic
attributes.Type: Dynamic
Type == Dynamic
Thanks in advance for any suggestions.
Configure a Thing Group index on Fleet Indexing.
For every dynamic Thing Group created, add an attribute to distinguish it as a dynamic thing group, i.e. attribute.dynamic: true
Call SearchIndex on the index with the query attributes.dynamic: true and that will return all dynamic Thing Groups.
It looks like it is not straight-forward. Thanks to my colleague, I created a script to get that list.
import boto3
client = boto3.client('iot')
list_thing_groups = client.list_thing_groups()
while True:
for thing_group in list_thing_groups['thingGroups']:
name = thing_group['groupName']
response = client.describe_thing_group(
thingGroupName=name
)
query = response.get('queryString')
if query:
print(name)
if list_thing_groups.get('nextToken'):
list_thing_groups = client.list_thing_groups(nextToken=list_thing_groups.get('nextToken'))
else:
break
The idea is queryString for dynamic Thing Group won't be null.
Related
I'm trying to change the table name created by AWS Crawler using boto3. Here is the code:
import boto3
database_name = "eventbus"
table_name = "enrollment_user_enroll_cancel_1_0_0"
new_table_name = "enrollment_user_enroll_cancel"
client = boto3.client("glue", region_name='us-west-1')
response = client.get_table(DatabaseName=database_name, Name=table_name)
table_input = response["Table"]
table_input["Name"] = new_table_name
print(table_input)
print(table_input["Name"])
table_input.pop("CreatedBy")
table_input.pop("CreateTime")
table_input.pop("UpdateTime")
client.create_table(DatabaseName=database_name, TableInput=table_input)
Getting the below error:
botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in TableInput: "DatabaseName", must be one of: Name, Description, Owner, LastAccessTime, LastAnalyzedTime, Retention, StorageDescriptor, PartitionKeys, ViewOriginalText, ViewExpandedText, TableType, Parameters
Unknown parameter in TableInput: "IsRegisteredWithLakeFormation", must be one of: Name, Description, Owner, LastAccessTime, LastAnalyzedTime, Retention, StorageDescriptor, PartitionKeys, ViewOriginalText, ViewExpandedText, TableType, Parameters
Could you please let me know the resolution for this issue? Thanks!
To get rid of botocore.exceptions.ParamValidationError thrown by client.create_table, you need to delete the corresponding items from table_input in a similar way as you did with CreatedBy etc
...
table_input.pop("DatabaseName")
table_input.pop("IsRegisteredWithLakeFormation")
client.create_table(DatabaseName=database_name, TableInput=table_input)
In case your original table had partitions, which want to add to a new table, you need to use similar approach. First you need to retrieve meta information about those partitions with either:
batch_get_partition()
get_partition()
get_partitions()
Note: depending which one you chose, you would need to pass different parameters. There are limitation on how many partitions you can retrieve within a single request. If I remember correctly it is around 200 or so. On top of that, you might need to use page paginator to list all of the available partitions. This is the case when your table has more then 400 partitions.
In general, I would suggest to:
paginator = client.get_paginator('get_partitions')
response = paginator.paginate(
DatabaseName=database_name,
TableName=table_name
)
partitions = list()
for page in response:
for p in page['Partitions']:
partitions.append(p.copy())
# Here you need to remove "DatabaseName", "TableName", "CreationTime" from
# every partition
Now you are ready add those retrieved partition to a new table with either:
batch_create_partition()
create_partition()
I'd suggest to use batch_create_partition(), however, it limits on how many partitions can be created at the single request.
I'm attempting to figure out how to elegantly update a document's associations with existing documents with Mongoid.
If I have Users and Groups, and want to assign a User to an existing group, how could I do this via update_attributes ?
I want to be able to do something like this:
user.attributes = { groups: [{"_id":"existing group id here"}]}
user.save
When I try to do the above, Mongoid attempts to INSERT a new group, thereby causing a ID duplicate error.
I have tried doing the same via nested attributes:
user.groups_attributes = [{"_id":"existing group id here"}]
user.save
And the same error occurs. Is there anyway I can do this WITHOUT having to manually query the group id and push it into the array? The reason I'm asking is because lets say i have a model with many associations.. i dont want to have to have blocks of code to update each association manually
Assigning an an existing User to an existing Group with update_attributes is very simple (assuming you already added the Group/User relation).
user.update_attributes(:group_id => 'existing group id here')
The following code works. In other words, there is no error returned and a group is created. However, the instances launched by this group will not have a Name tag.
AS_GROUP = AutoScalingGroup(
group_name=AS_GROUP_NAME,
availability_zones=ZONE_LIST,
launch_config=LAUNCH_CONFIG_NAME,
min_size=GROUP_MIN_SIZE,
max_size=GROUP_MAX_SIZE,
default_cooldown=DEFAULT_COOLDOWN,
desired_capacity=DESIRED_CAPACITY,
tag=[Tag(
key='Name',
value='ASG Minion',
propagate_at_launch=True,
resource_id=AS_GROUP_NAME)],
)
AS_CONNECTION.create_auto_scaling_group(AS_GROUP)
I have tried the Tag method without the resource_id.
[Tag(key="Name", value="ASGMinion", propagate_at_launch=True)]
Other obviously wrong ways I have also tried:
tag='k=Name, v=ASGMinion, p=true',
tag=['k=Name, v=ASGMinion, p=true'],
tag=[Tag('k=Name, v=ASGMinion, p=true')],
No worky.
Of course, I can run this after the group is already created:
tag = Tag(key='Name', value=tag_name, propagate_at_launch=True, resource_id=groups[group_number].name)
asConnection.create_or_update_tags([tag])
But that defeats the point of the tag parameter in the AutoScalingGroup method.
Well, this is rather embarrassing. The main problem was that the correct name of the parameter is tags and not tag. Once I got that sorted out I was able to run through and find which value this parameter was looking for. This is what works:
tags=[Tag(
key='Name',
value='ASG Minion',
propagate_at_launch=True,
resource_id=AS_GROUP_NAME)],
)
I tried it also without the resource_id and it complained with Invalid resourceID: None. So the group name needs to be specified in the tag even if the group is the being created at the same time. Hopefully this is helpful to someone else.
it seems that Boto is the official Amazon API module for Python, and this one is for Tornado, so here is my questions:
does it offer pagination (requests only 10 products, since amazon offers 10 products per page, then i want only to get the first page...), then how (sample code?)
how then to parse the product parse, i've used python-amazon-simple-product-api but sadly it dont offer pagination, so all the offers keep iterating.
generally, pagination is performed by the client requesting the api. To do this in boto, you'll need to cut your systems up. So for instance, say you make a call to AWS via boto, using the get_all_instances def; you'll need to store those somehow and then keep track of which servers have been displayed, and which not. To my knowledge, boto does not have the LIMIT functionality most dev's are used to from MySQL. Personally, I scan all my instances and stash them in mongo like so:
for r in conn.get_all_instances(): # loop through all reservations
groups = [g.name for g in r.groups] # get a list of groups for this reservation
for x in r.instances: # loop through all instances with-in reservation
groups = ','.join(groups) # join the groups into a comma separated list
name = x.tags.get('Name',''); # get instance name from the 'Name' tag
new_record = { "tagname":name, "ip_address":x.private_ip_address,
"external_ip_nat":x.ip_address, "type":x.instance_type,
"state":x.state, "base_image":x.image_id, "placement":x.placement,
"public_ec2_dns":x.public_dns_name,
"launch_time":x.launch_time, "parent": ObjectId(account['_id'])}
new_record['groups'] = groups
systems_coll.update({'_id':x.id},{"$set":new_record},upsert=True)
error = db.error()
if error != None:
print "err:%s:" % str(error)
You could also wrap these in try/catch blocks. Up to you. Once you get them out of boto, should be trivial to do the cut up work.
-- Jess
I'd like to use CFLDAP to retrieve all the users in a certain distribution group used by Exchange. If this is possible, what do I use for the 'filter' attribute of CFLDAP? Also, if all I have is the email address for the group (e.g. 'sales#example.com'), can I still get the user information, or do I need the name of the group that uses that email address?
For example, what would I put in the block below?
<cfldap server = "foo.example.com"
action = "query"
name = "ldap2"
start = "dc=foo,dc=example,dc=com"
attributes = "givenName,sn,sAMAccountName,mail,employeeID,dn"
filter="?????????????"
username="BAR\eterps"
password="12345" >
To get the Group name from the email address, I used Active Directory Explorer. I'm sure there is a way to query for it as well.
Once I had the group name, I created my filter for CFLDAP: (&(objectClass=user)(memberOf=cn=Sales,ou=Email Distribution Groups,dc=foo,dc=example,dc=com))
So the resulting CFLDAP query looks like:
<cfldap server = "foo.example.com"
action = "query"
name = "ldap2"
start = "dc=foo,dc=example,dc=com"
attributes = "givenName,sn,sAMAccountName,mail,employeeID,dn"
filter="(&(objectClass=user)(memberOf=cn=Sales,ou=Email Distribution Groups,dc=foo,dc=example,dc=com))"
username="BAR\eterps"
password="12345" >
a filter is not required when using cfldap in my experience. What happens when you run the query without a filter?
If I understand your question correctly, you can modify the start attribute with the specific Group dn, and not just the Root dn, it should only return the info from that group. If there is an attribute that points to users that are members of that group, make sure you include that in the attribute list.
If you dont modify the start attribute, your filter would be something like (cn=groupname) that points to the group you want.