I have following code written in DynamoDB for table creation. I am running this with Eclise. I have configured Tomcat server. I deployed my app on Tomcat and open the localhost URL.
DynamoDB dynamoDB = new DynamoDB(dynamo);
ArrayList<AttributeDefinition> attributeDefinitions = new ArrayList<AttributeDefinition>();
attributeDefinitions.add(new AttributeDefinition()
.withAttributeName("Id").withAttributeType("N"));
ArrayList<KeySchemaElement> keySchema = new ArrayList<KeySchemaElement>();
keySchema.add(new KeySchemaElement().withAttributeName("Id")
.withKeyType(KeyType.HASH));
CreateTableRequest request1 = new CreateTableRequest()
.withTableName("abcdef")
.withKeySchema(keySchema)
.withAttributeDefinitions(attributeDefinitions)
.withProvisionedThroughput(new ProvisionedThroughput()
.withReadCapacityUnits(5L)
.withWriteCapacityUnits(6L));
System.out.println("Issuing CreateTable request for abcde");
Table table = dynamoDB.createTable(request1);
System.out.println("Waiting for abcde to be created...this may take a while...");
table.waitForActive();
It runs successfully. It also shows the table created successfully.
But when I open Amazon DynamoDB console, it does not reflect the newly created table. Can anyone suggest me what goes wrong here ? I have properly configured secretKey and accessKey.
Could it be a different region where the tables created and the console shows?
Related
I have an Athena database set up pointing at an S3 bucket containing ALB logs, and it all works correctly. I partition the table by a column called datetime and the idea is that it has the format YYYY/MM/DD.
I can manually create partitions through the Athena console, using the following command:
ALTER TABLE alb_logs ADD IF NOT EXISTS PARTITION (datetime='2019-08-01') LOCATION 's3://mybucket/AWSLogs/myaccountid/elasticloadbalancing/eu-west-1/2019/08/01/'
I have created a lambda to run daily to create a new partition, however this doesn't seem to work. I use the boto3 python client and execute the following:
result = athena.start_query_execution(
QueryString = "ALTER TABLE alb_logs ADD IF NOT EXISTS PARTITION (datetime='2019-08-01') LOCATION 's3://mybucket/AWSLogs/myaccountid/elasticloadbalancing/eu-west-1/2019/08/01/'",
QueryExecutionContext = {
'Database': 'web'
},
ResultConfiguration = {
"OutputLocation" : "s3://aws-athena-query-results-093305704519-eu-west-1/Unsaved/"
}
)
This appears to run successfully without any errors and the query execution even returns a QueryExecutionId as it should. However if I run SHOW PARTITIONS web.alb_logs; via the Athena console it hasn't created the partition.
I have a feeling it could be down to permissions, however I have given the lambda execution role full permissions to all resources on S3 and full permissions to all resources on Athena and it still doesn't seem to work.
Since Athena query execution is asynchronous your Lambda function never sees the result of the query execution, it just gets the result of starting the query.
I would be very surprised if this wasn't a permissions issue, but because of the above the error will not appear in the Lambda logs. What you can do is to log the query execution ID and look it up with the GetQueryExecution API call to see that the query succeeded.
Even better would be to rewrite your code to use the Glue APIs directly to add the partitions. Adding a partition is a quick and synchronous operation in Glue, which means you can make the API call and get a status in the same Lambda execution. Have a look at the APIs for working with partitions: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-partitions.html
enter image description hereI'm able to connect to MYSQL while running my Pyspark Code Locally in juypter notebook, but the same code I am getting Communication error in AWS Glue while running the code. I have added MySQL jar in jar files required while creating the job in AWS Glue.
Reading from MYSQL
dataframe_mysql = sqlContext.read.format("jdbc").option("url", "jdbc:mysql://localhost/read").option("driver", "com.mysql.jdbc.Driver").option("dbtable", "student").option("user", "root").option("password", "root").load()
Writing to MYSQL
df = sc.parallelize([[25, 'Prem'],
[20, 'Kate'],
[20, 'Kate'],
[40, 'Cheng']]).toDF(["Depy_id","Dept_name"])
df.write.format('jdbc').options(
url='jdbc:mysql://localhost/test',
driver='com.mysql.jdbc.Driver',
dbtable='dept',
user='root',
password='root').mode('overwrite').save()
Please note that you have to provide a valid database URL, not a localhost. I believe your jupyter notebook was run locally on a laptop, in the same local environment where your mysql is running as well.
AWS Glue runs within an AWS environment, and behind the scene it would launch number of EC2 instances depending upon the DPU configuration. If your URL is configured as LOCALHOST, then the EC2 instance where the pyspark code is running, would look for a mysql database on the same node.
Please make sure you have a valid public IP for the mysql database, and try to setup a connection in the AWS Glue as suggested by bdcloud, and retry again. If you dont want to create a connection, you can hard code the connection parameters in the code and try again. If you cannot get a public IP for the installed mysql database, maybe you can try setup an RDS Mysql on AWS, and use it for testing.
Sample code snippet:
conn = mysql.connector.connect(host=url, user=uname, password=pwd, database=dbase)
cur = conn.cursor()
insertQry = "INSERT INTO emp (id, emp_name, dept, designation, address1, city, state, active_start_date, is_active) SELECT (SELECT coalesce(MAX(ID),0) + 1 FROM atlas.emp) id, tmp.emp_name, tmp.dept, tmp.designation, tmp.address1, tmp.city, tmp.state, tmp.active_start_date, tmp.is_active from EMP_STG tmp ON DUPLICATE KEY UPDATE dept=tmp.dept, designation=tmp.designation, address1=tmp.address1, city=tmp.city, state=tmp.state, active_start_date=tmp.active_start_date, is_active =tmp.is_active ;"
n = cur.execute(insertQry)
print (" CURSOR status :", n)
Please refer to the AWS Glue connections section:
yes that's true I am able to connect it as above by just adding the connection to the job as well as changing the local host to the respective
My IOT device populates the following data:
12-09-2017 12:05:26PeopleIn: 2
and this gets streamed in AWS IOT server when I subscribe to the topic. When I export a message as csv I get the following getting exported:
format,payload,qos,timestamp,topic
string,12-09-2017 12:05:26PeopleIn: 2,0,1505190320098,TestSite
I am trying to configure AWS DynamoDB to store the above data as a table. In AWS IOT rules setting I have provided the following:
Attribute = *
topicFilter = # (i have also tried with TestSite)
Selected Action = Insert a message into DynamoDB table
I created a dynamoDB table with the following settings:
Primary Partition Key = timestamp (Number)
I have set any sort key. Upon setting up the rule, the data gets populated into the dynamoDB. However, the timestamp is populated as Unix Timestamp and the data is populated in an encrypted format. The following is the output that I am getting in AWS DynamoDB.
timestamp data_raw
1505198126899 MTItMDktMjAxNyAxMjowNToyNlBlb3BsZUluOiAy
Not sure what I am missing. What I ideally want is the timestamp + payload. The desired output would be:
timestamp data
1505198126899 12-09-2017 12:05:26PeopleIn: 2
Can someone help?
There is a dynamodb table which has triggers in the triggers tab. How to get the list of those triggers using AWS Java SDK for a given table.
I have gone through the JAVA SDK but so far have not been able to find any API to get the list.
https://docs.aws.amazon.com/cli/latest/reference/lambda/list-event-source-mappings.html
Java:
AWSLambda client = AWSLambdaClientBuilder.standard().build();
ListEventSourceMappingsRequest request = new ListEventSourceMappingsRequest().withEventSourceArn("your table stream arn");
ListEventSourceMappingsResult result = client.listEventSourceMappings(request);
List<EventSourceMappingConfiguration> eventSourceMappings = result.getEventSourceMappings();
Believe these operations are part of DynamoDB Streams - see listStreams
I am trying to connect my app to DynamoDB. I have set everything up the way Amazon recommends. But i still keep getting the same error over and over again:
7-21 11:02:29.856 10027-10081/com.amazonaws.cognito.sync.demo E/AndroidRuntime﹕ FATAL EXCEPTION: AsyncTask #1
Process: com.amazonaws.cognito.sync.demo, PID: 10027
java.lang.RuntimeException: An error occured while executing doInBackground()
at android.os.AsyncTask$3.done(AsyncTask.java:304)
at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:355)
at java.util.concurrent.FutureTask.setException(FutureTask.java:222)
at java.util.concurrent.FutureTask.run(FutureTask.java:242)
at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:231)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
at java.lang.Thread.run(Thread.java:818)
Caused by: com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException: Requested resource not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: GIONOKT7E3AMTC4PO19CPLON93VV4KQNSO5AEMVJF66Q9ASUAAJG)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:710)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:385)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:196)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2930)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.updateItem(AmazonDynamoDBClient.java:930)
at com.amazonaws.mobileconnectors.dynamodbv2.dynamodbmapper.DynamoDBMapper$SaveObjectHandler.doUpdateItem(DynamoDBMapper.java:1173)
at com.amazonaws.mobileconnectors.dynamodbv2.dynamodbmapper.DynamoDBMapper$2.executeLowLevelRequest(DynamoDBMapper.java:873)
at com.amazonaws.mobileconnectors.dynamodbv2.dynamodbmapper.DynamoDBMapper$SaveObjectHandler.execute(DynamoDBMapper.java:1056)
at com.amazonaws.mobileconnectors.dynamodbv2.dynamodbmapper.DynamoDBMapper.save(DynamoDBMapper.java:904)
at com.amazonaws.mobileconnectors.dynamodbv2.dynamodbmapper.DynamoDBMapper.save(DynamoDBMapper.java:688)
at com.amazonaws.cognito.sync.Utils.FriendsSyncManager.initalize_credentialprovider(FriendsSyncManager.java:43)
at com.amazonaws.cognito.sync.ValU.FriendListActivity$SyncFriends.doInBackground(FriendListActivity.java:168)
at com.amazonaws.cognito.sync.ValU.FriendListActivity$SyncFriends.doInBackground(FriendListActivity.java:160)
at android.os.AsyncTask$2.call(AsyncTask.java:292)
What could be the solution?
Okey it seems you need to add:
ddbClient.setRegion(Region.getRegion(Regions.EU_WEST_1));
// Add correct Region. In my case its EU_WEST_1
after the following line:
AmazonDynamoDBClient ddbClient = new AmazonDynamoDBClient(credentialsProvider);
Now it works. The table was successfully created.
Have a nice day and thanks!
It seems that the table you are trying to connect to doesn't exist. Verify the table name in your code agains the name of the table you created.
Please note that table name is case sensative.
You need to check a few things:
Check your credentials in your code:
private static String awsSecretKey = "your_secret_key"; //get it in AWS web UI
private static String awsAccessKey = "your_access_key"; //get it in AWS web UI
Check your Region code and set correct value:
client.setRegion(Region.getRegion(Regions.US_EAST_1));
You can get this value from your AWS Web Console. Details
Check does you have already created DynamoDB table & indexes under your Region.
If no, check your code
#DynamoDBTable(tableName = "Event")
public class Event implements Serializable {
public static final String CITY_INDEX = "City-Index";
public static final String AWAY_TEAM_INDEX = "AwayTeam-Index";
And create manualy from AWS Console or somehow else your table (Event in my case) and indexes (City-Index, AwayTeam-Index in my case). Please note - table and index name in case sensative.
Good sample - https://github.com/aws-samples/lambda-java8-dynamodb
From the docs it's either you don't have a Table with that name or it is in CREATING status.
I would double check to verify that the table does in fact exist, in the correct region, and you're using an access key that can reach it.
Or you might select the wrong region.
Along with #Yuliia Ashomok's answer
AWS C++ SDK 1.7.25
Aws::Client::ClientConfiguration clientConfig;
clientConfig.region = Aws::Region::US_WEST_2;
If using Spring boot, you can configure the region via application properties:
in src/main/resources/application.yaml
cloud:
aws:
region:
static: eu-west-1
As this problem is some what platform agnostic. For anyone coming at the same problem from .NET/C# ...
You can instantiate your client with the Endpoint in the constructor:
AmazonDynamoDBClient client = new AmazonDynamoDBClient(credentials, Amazon.RegionEndpoint.USEast1 );
I would have assumed that this would have been brought in from you AWS Profile but seems not although you could do something like this, where profile is imported from your SharedCredentialsFile:
new AmazonDynamoDBClient(credentials, profile.Region );
If you are sure that you have already created the table in DynamoDB but still getting this error. That means you may chances that your region is not correct. Just look at the top right corner of your AWS portal, with your profile dropdown. One another dropdown will give you the option to select your region. And now follow the process again with the right region.
Hope this works. This works for me.