I'm creating a AWS glue crawler using API, I have created a crawler already and it is working fine. Now i want to implemt IncrementalCrawler functionality to it. When I try to give Recrawler parameter for update_crawler() it is throwing me with the error. The same thing is happening is happening with create_crawler() also.
mycode
`crawler_args = {"Name": args['CRAWLER_NAME'], "Role": args['IAM_ROLE'], "DatabaseName": args['DB_NAME'],"RecrawlPolicy": {"RecrawlBehavior": "CRAWL_NEW_FOLDERS_ONLY"}
"Targets":
{"S3Targets": [{"Path": f"s3:/../{src_obj}/"}]},
"TablePrefix": ""}
glue.create_crawler(**crawler_args)`
"Parameter validation failed:
Unknown parameter in input: "RecrawlPolicy", must be one of: Name, Role, DatabaseName, Description, Targets, Schedule, Classifiers, TablePrefix, SchemaChangePolicy, Configuration, CrawlerSecurityConfiguration"
I believe it is because of the boto3 version im using, if that is the case how to update boto3 to latest version.
It looks like you are missing a comma, can you try:
crawler_args = {"Name": args['CRAWLER_NAME'], "Role": args['IAM_ROLE'], "DatabaseName": args['DB_NAME'],"RecrawlPolicy": {"RecrawlBehavior": "CRAWL_NEW_FOLDERS_ONLY"},
"Targets":
{"S3Targets": [{"Path": f"s3:/../{src_obj}/"}]},
"TablePrefix": ""}
Related
I have my product uploaded to AWS as an AMI through Hashicorp's Packer. Now I'ld like to automate the last step, publishing it to the marketplace. The product already exists, it's only about adding a revision.
After reading this article, the API_StartChangeSet doc, this add revisions user guide & fiddling with the marketplace console, I think I just have to
aws marketplace-catalog start-change-set --catalog AWSMarketplace --change-set-name "$VERSION" --change-set '[ {"ChangeType": "AddRevisions", "Entity": {"Identifier": "REDACTED#29","Type": "ServerProduct#1.0"}, "Details": "{\"DataSetArn\": \"?????\", \"RevisionArns\": [\"?????\"] }" ]'
I'm having a hard time coming up with "Details" part. I've my AMI id. I guess that goes in the RevisionsArns ? What should I put in the DataSetArn, the "EntityArn" from the output of aws marketplace-catalog describe-entity --catalog AWSMarketplace --entity-id REDACTED ?
Details facet here is just a product type specific facet, encoded as json string. For the AMI that you are offering in the AWS Marketplace, it could include support information, region availability or any other info that provides a descriptive text regarding your change. For example:
"Details": "{\"Description\":{}, \"PromotionalResources\":{}, \"RegionAvailability\":{}, \"SupportInformation\":{}}",
The example you found does not necessarily mean that you have to have EntityArn and RevisionsArns. The Details facet is used as an information describing the details of your change.
Check here.
Turns out I didn't found the good documentation, my last link being about AWS Data Exchange, whose "Details" field's contents were confusing.
Here the relevant documentation: Marketplace catalog AMI add version, and here's the snippet I was looking for
"Details": "{
\"Version\": {
\"VersionTitle\": \"*My new title*\",
\"ReleaseNotes\": \"*My new Release notes*\"
},
\"DeliveryOptions\": [
{
\"Details\": {
\"AmiDeliveryOptionDetails\": {
\"AmiSource\": {
\"AmiId\": \"ami-1234567890abcdef\",
\"AccessRoleArn\": \"arn:aws:iam::12345678901:role/AwsMarketplaceAmiIngestion\",
\"UserName\": \"ec2-user\",
\"OperatingSystemName\": \"AMAZONLINUX\",
\"OperatingSystemVersion\": \"Amazon Linux 2 AMI 2.0.20210126.0 x86_64 HVM gp2\"
},
\"UsageInstructions\": \"Easy to use AMI\",
\"RecommendedInstanceType\": \"m4.xlarge\",
\"SecurityGroups\": [
{
\"IpProtocol\": \"tcp\",
\"FromPort\": 443,
\"ToPort\": 443,
\"IpRanges\": [
\"0.0.0.0/0\"
]
}
]
}
}
}
]
}"
I am currently assigning AWS media-live channels to a specific group by a custom tag and want to get the (CostExplorer) GetReservationUtilization for a group's channels by filtering by tag. The AWS documentation for GetReservationUtilization lists the Filtering options as:
"Filter": {
.
.
"Tags": {
"Key": "string",
"MatchOptions": [ "string" ],
"Values": [ "string" ]
}
.
.
}
I interpret it as it should be possible to sort by a custom set tag via:
"Key": "Group",
"Value": [customId]
But I get an error that says "An error occurred (ValidationException) when calling the GetReservationUtilization operation: Tags expression is not allowed, allowed expression(s): And, Not, Dimensions"
Feels like I have tried everything possible but I cant seem to get it to work.
Have you looked at the examples boto3 documentation?
Seems you may need to wrap the tag element inside of the And, Not or supply dimensions
For anyone coming here in the future, sorting reservation-utilization by the tag dimension is currently not supported. The following dimensions are supported:
AZ
CACHE_ENGINE
DEPLOYMENT_OPTION
INSTANCE_TYPE
LINKED_ACCOUNT
OPERATING_SYSTEM
PLATFORM
REGION
SERVICE
SCOPE
TENANCY
As specified in the AWS API docs.
I'm looking for a programmatic way to retrieve parameters just by giving the name or a part of the complete path (instead of giving the full path with the name).
It's pretty easy using the Parameter Store AWS Systems Manager console, if I type tokens, I retrieve all parameters where the Name contains tokens :
Is there a way to do the same but using AWS CLI or AWS SDK (python or Go preferably) ?
I think this is what you are after:
aws ssm describe-parameters --parameter-filters Key=Name,Values=token,Option=Contains
Or with Python:
import boto3
response = boto3.client("ssm").describe_parameters(
ParameterFilters=[
{
'Key': 'Name',
'Option': 'Contains',
'Values': [
'token',
]
},
]
)
I have below table in DynamoDB
{
"id": 1,
"user": {
"age": "26",
"email": "testuser#gmail.com",
"name": "test user"
}
}
Using AWS console, I want to scan all the records whose email address contains gmail.com
I am trying this but it is giving no results.
I am new to AWS, not sure what's wrong here. Is it not possible to scan on nested fields?
I've been trying to figure this out myself but it would seem that nested item scans are not supported through the console.
I'm going based off of this which offer some alternative options via CLI or SDK: https://forums.aws.amazon.com/thread.jspa?messageID=931016
I am new at programming with AWS services, so some fundamental things are pretty hard for me. Recently, I was asked to develop an app that used Amazon Pinpoint to send push notifications, as a test for considering future implementations.
As you can see in another question I posted in here (Amazon Pinpoint and Ionic - Push notifications not working when app is in background), I was having trouble trying to send push notifications to users when my app is running in the background. The app was developed using Ionic by following these steps.
When I was almost giving up, I decided to try sending the pushes directly through Firebase, and it finally worked. Some research took me to this question, in which another user described the problem as only happening in AWS Console, so the solution would be to use CLI. After searching a little about it, I found this tutorial about how to sending pinpoint messages to users using CLI, that seems to be what I wanted. Combining it with this documentation about phonegap plugin, I was able to generate a JSON I thought could be a solution:
{
"ApplicationId":"io.ionic.starter",
"MessageRequest":{
"Addresses": {
"": {
"BodyOverride": "",
"ChannelType": "GCM",
"Context": {
"": ""
},
"RawContent": "",
"Substitutions": {},
"TitleOverride": ""
}
},
"Context": {
"": ""
},
"Endpoints": {"us-east-1": {
"BodyOverride": "",
"Context": {},
"RawContent": "",
"Substitutions": {},
"TitleOverride": ""
}
},
"MessageConfiguration": {
"GCMMessage": {
"Action": "OPEN_APP",
"Body": "string",
"CollapseKey": "",
"Data": {
"": ""
},
"IconReference": "",
"ImageIconUrl": "",
"ImageUrl": "",
"Priority": "High",
"RawContent": "{\"data\":{\"title\":\"sometitle\",\"body\":\"somebody\",\"url\":\"insertyourlinkhere.com\"}}",
"RestrictedPackageName": "",
"SilentPush": false,
"SmallImageIconUrl": "",
"Sound": "string",
"Substitutions": {},
"TimeToLive": 123,
"Title": "",
"Url": ""
}
}
}
}
But when I executed it in cmd with aws pinpoint send-messages --color on --region us-east-1 --cli-input-json file://test.json, I got the response An error occurred (NotFoundException) when calling the SendMessages operation: Resource not found.
I believe I didn't write the JSON file correctly, since it's my first time doing this. So please, if any of you know what I am doing wrong, no mattering which step I misunderstood, I would appreciate the help!
"Endpoints" field in the Message request deals with the endpoint id (the identifier associated with an end user device while registering to pinpoint and not the region.)
In case if you haven't registered any endpoints with Pinpoint, you can use the "Addresses" field. After registering the GCM Channel in Amazon Pinpoint, you can get the GCM device token from your device and specify it here.
Here is a sample for sending direct messages using Amazon Pinpoint Note: The example deals with sending SMS message. You should have registered a SMS channel first and created an endpoint with the endpoint id as "test-endpoint1". Otherwise, you can use the "Addresses" field instead of "Endpoints" field.
aws pinpoint send-messages --application-id $APP_ID --message-request '{"MessageConfiguration": {"SMSMessage":{"Body":"hi hello"}},"Endpoints": {"test-endpoint1": {}}}
Also Note: ApplicationId is generated by Pinpoint. When you visit the Pinpoint console and choose your application, the URL will be of the format
https://console.aws.amazon.com/pinpoint/home/?region=us-east-1#/apps/someverybigstringhere/
Here "someverybigstringhere" is the ApplicationId and not the name you give for your project.