I'm really new with AWS and IoT, and my goal is to:
Use the Java SDK v.2 from my serverless application to create/get/update/attach/... certificates and things.
Create client side MQTT demo application to connect publish and subscribe to messages used by my new certificates and thing created in phase 1.
Publish/subscribe messages in the server side in order to talk to my things/clients.
1 & 2 I've managed to do perfectly.
But I don't understand how should I do the 3rd one.
Should I use the IoT device SDK as well in the server side ? If so with what credentials do I connect ?
Is there some objects in the SDK that I've missed?
In order to connect to IoT Core from the server I first configure my SSO connection using the AWSCLI and in the code I simply use my profile name and region to connect.
Your serverless Java application needs to be configured as a "Thing" in the same account/region as your IoT devices. In the console, go to
AWS IoT -> Manage -> Things
and create a thing for your app. In this case you shouldn't need a "Device Shadow", and you can select "Auto Generate Certificates".
For the IoT Policy, you will need the following :
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "iot:Connect",
"Resource": "arn:aws:iot:us-east-1:YOUR_AWS_ACCOUNT_ID:client/*"
},
{
"Effect": "Allow",
"Action": "iot:Subscribe",
"Resource": "arn:aws:iot:us-east-1:YOUR_AWS_ACCOUNT_ID:topicfilter/*"
},
{
"Effect": "Allow",
"Action": "iot:Receive",
"Resource": "arn:aws:iot:us-east-1:YOUR_AWS_ACCOUNT_ID:topic/*"
},
{
"Effect": "Allow",
"Action": "iot:Publish",
"Resource": "arn:aws:iot:us-east-1:YOUR_AWS_ACCOUNT_ID:topic/*"
}
]
}
Your application will communicate with IoTCore using the endpoint shown in the Settings screen in IoTCore for the region where you have created your thing. Your application will authenticate using the key/cert you downloaded when creating the thing (username/password auth is not allowed).
Once your application connects to the endpoint, you will want to "subscribe" to the same topic your devices use to send messages. You can also publish to one or more topics.
In order to debug communications, you can use the MQTT client in the AWS IoTCore console, just note the console needs to be refreshed periodically when communication times out. I recommend marking your topics as favorites so they are easy to re-subscribe to on a refresh.
As for coding in Java, you should be able to leverage examples from the AWS IoT Device SDK here :
https://github.com/aws/aws-iot-device-sdk-java-v2/tree/main/samples
Here's a link to the MQTT client class :
http://aws-iot-device-sdk-java-docs.s3-website-us-east-1.amazonaws.com/com/amazonaws/services/iot/client/AWSIotMqttClient.html
Please note that your app will not have access to messages when not in use. There are a few strategies to deal with message persistence, but that's outside the scope of your question, so I won't cover it here.
Hopefully this gets you pointed in the right direction.
Related
I have a Glue job to push data into AWS OpenSearch. Everythings works perfectly when I have an "open" permission on OpenSearch, for example:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "es:*",
"Resource": "arn:aws:es:<region>:<accountId>:domain/<domain>/*"
}
]
}
This works without issue. The problem is I want to secure my OpenSearch domain to only the role running the glue job.
I attempted to do that starting basic with:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::<accountId>:role/AWSGluePowerUser"
]
},
"Action": [
"*"
],
"Resource": [
"*"
]
}
]
}
This disables all access to OpenSearch which I want, however it also blocks it for Glue even though the jobs a running with the AWSGluePowerUser role set.
An error occurred while calling o805.pyWriteDynamicFrame. Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
Which I assume is because the Glue job can no longer see the OpenSearch cluster. Keep in mind everything works when using the "default" access policy for OpenSearch.
I have my glue job configured to use the IAM role AWSGluePowerUser which also has AmazonOpenSearchServiceFullAccess policy attached.
I'm not sure where I've gone wrong here?
Edit: Here is where/how I've set the roles for the Glue job, I assume this is all I needed to do?
From Glue Job Details
I believe this is not possible because the AWS Glue Elasticsearch connector is based on an open-source Elasticsearch Spark library that doest not sign requests using AWS Signature Version 4 which is required for enforcing domain access policies.
If you take a look at the key concepts for fine-grained access control in OpenSearch, you'll see:
If you choose IAM for your master user, all requests to the cluster must be signed using AWS Signature Version 4.
If you visit the Elasticsearch Connector for AWS Glue AWS Marketplace page, you'll notice that the connector itself is based on an open-source implementation:
For more details about this open-source Elasticsearch spark connector, please refer to this open-source connector online reference
Under the hood, AWS Glue is using this library to index data from Spark dataframes to the Elasticsearch endpoint. Since this open-source library (maintained by the Elasticsearch community) does not have support for signing requests using using AWS Signature Version 4, it will only work with the "open permission" you've referenced. This is hinted at in the big picture on fine-grained access control:
In general, if you enable fine-grained access control, we recommend using a domain access policy that doesn't require signed requests.
Note that you can always fall back us using a master user based on username/password:
Create a master user (username/password) for the OpenSearch domain's fine-grained access control configuration.
Store the username/password in an AWS Secrets Manager secret as described here.
Attach the secret to the AWS Glue connector as described here.
Hope this helps!
I usually take a "deny everyone except" approach in these situations
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Principal": "*",
"Action": "es:*",
"Resource": [
"*"
],
"Condition": {
"ArnNotLike": {
"aws:PrincipalArn": [
"arn:aws:iam::<accountId>:role/AWSGluePowerUser"
]
}
}
}
]
}
Based on the documentation, I understand how to create IoT things and also how to create authenticated users using AWS IoT. My question is geared towards how to effectively combine these services so that each user can access several of his or her devices securely.
Let's say Jane has just signed up for the platform and wants to connect her lightbulb device to her account. Let's also assume that her lightbulb device already has a certificate on it and a policy in IoT so that it can connect to the IoT platform and then publish and subscribe to a few topics. For the sake of simplicity, let's say that Jane can create this connection by simply making an API call named pairDevice which takes in a cognito identity (i.e. 59700b18-94c7-XXXX-857a-d820a68c0ec6) and a device serial number.
Basically I envision this function doing two things:
It will call "AttachPrincipalPolicy" that will link the policy associated with that lightbulb to the cognito user. Which I would assume at this point the cognito user would be able to publish and subscribe to topics for that particular lightbulb and only that lightbulb.
It would add a DynamoDB entry in the users account of the thing ARN so that way it can be easily referenced and queried later.
So if my understanding is correct I would have a policy like this for each of my devices in IoT (should they also publish and subscribe to topics with the the serial number too? I want to make sure that users cannot connect to devices they aren't allowed to obviously):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iot:Connect"
],
"Resource": [
"arn:aws:iot:us-west-1:123456789012:client/SerialNumber",
]
},
{
"Effect": "Allow",
"Action": [
"iot:Publish",
"iot:Subscribe",
"iot:Receive"
],
"Resource": [
"*"
]
}
]
}
And then I would simply attach this policy using AttachPrincipalPolicy to the cognito user? Do I need to have an explicit policy in Amazon Cognito identity pool for IoT Access, or is that linkage done specifically through AttachPrincipalPolicy?
In order to authenticate an Amazon Cognito identity to publish MQTT messages over HTTP, you must specify two policies. The first policy must be attached to an Amazon Cognito identity pool role. This first policy is most likely the managed policy AWSIoTDataAccess.
The second policy must be attached to an Amazon Cognito user using the AWS IoT AttachPrincipalPolicy API.
An example application demonstrating this is:
https://github.com/awslabs/aws-iot-chat-example
For explicit instructions, you can read:
https://github.com/awslabs/aws-iot-chat-example/blob/master/docs/authentication.md
I have a SNS topic and subscriber set up like so :
The subscribers are correctly notified when the Topic is tested via "Publish to Topic" from AWS console
I have a IoT rule like so :
I have a policy attached to the rule like so :
{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": "sns:Publish",
"Resource": "arn:aws:sns:ap-southeast-xxxxxx:MySNSTopic"
}
}
Yet when I try and test from the MQTT browser client, the notification is not fired
What am I missing?
Started working after a few minutes. Guess it needs some time to set up
I've been converting an existing application to an EC2, S3 and RDS model within AWS, so far it's going well but I've got a problem I can't seem to find any info on.
My Web application accesses the S3 box for images and documents, the way this is stored is by client code,
Data/ClientCode1/Images
Data/ClientCode2/Images
Data/ClientABC/Images -- etc
The EC2 hosting the web application also works within a similar structure, so www.programname.com/ClientCode1/Index.aspx as an example, this has working security to prevent cross client access.
Now when www.programname.com/ClientCode1/Index.aspx goes to access the S3 for images, I need to make sure it can only access the ClientCode1 folder on the S3, the goal is to prevent client A seeing the images/documents of client B if you had a tech sort trying.
Is there perhaps a way to get the page referrer, or is there a better approach to this issue?
There is no way to use the URL or referrer to control access to Amazon S3, because that information is presented to your application (not S3).
If all your users are accessing the data in Amazon S3 via the same application, it will be the job of your application to enforce any desired security. This is because the application will be using a single set of credentials to access AWS services, so those credentials will need access to all data that the application might request.
To clarify: Amazon S3 has no idea which page a user is viewing. Only your application knows this. Therefore, your application will need to enforce the security.
I found the solution, seems to work well
{
"Version": "2012-10-17",
"Id": "http referer policy example",
"Statement": [
{
"Sid": "Allow get requests referred by www.example.com and example.com.",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::clientdata/Clients/Client1/*",
"Condition": {
"StringLike": {"aws:Referer": ["http://www.example.com/Client1/*","http://example.com/Client1/*"]}
}
},
{
"Sid": "Explicit deny to ensure requests are allowed only from specific referer.",
"Effect": "Deny",
"Principal": "*",
"Action": "s3:*",
"Resource": "arn:aws:s3:::clientdata/Clients/Client1/*",
"Condition": {
"StringNotLike": {"aws:Referer": ["http://www.example.com/Client1/*","http://example.com/Client1/*"]}
}
}
]
}
This allows you to check the referer to see if the URL is from a given path, in my case I have each client sitting in their own path, the bucket follows the same rule, in the above example only a user coming from Client1 can access the bucket data for Client1, if I log in to Client2 and try force an image to the Client1 bucket I'll get access denied.
I have set up a elasticsearch server using AWS elasticsearch service (Not EC2). It gave me an endpoint https://xxx-xxxxxxxx.us-west-2.es.amazonaws.com/ and if I click this endpoint(Note that there is no port specified) I can get the expected
{
status: 200,
name: "Mastermind",
cluster_name: "xxxx",
version: {
number: "1.5.2",
build_hash: "yyyyyy",
build_timestamp: "2015-04-27T09:21:06Z",
build_snapshot: false,
lucene_version: "4.10.4"
},
tagline: "You Know, for Search"
}
The question is how do I get this through the elasticsearch java client without a port number? The sample code I get is
Client client = TransportClient.builder().build()
.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("host1"), 9300));
If I use this code and just replace "host1" with my endpoint, I'll get "NoNodeAvailableException"
ps:
The java client version I'm using is 2.0.0.
Edit
I finally decided to go with Jest, a 3rd party REST client. But what Brooks answered below is also very helpful - AWS do use port 80 for http and 443 for https. The blocker for me was the firewall I guess.
Edit2
The AWS ES service documentation explicitly says:
The service supports HTTP on port 80, but does not support TCP transport.
Believe it or not, AWS doesn't launch Elasticsearch using 9200 and 9300. It's launched via plain old port 80.
So, to demonstrate, try this...
curl -XPOST "http://xxx-xxxxxxxx.us-west-2.es.amazonaws.com:80/myIndex/myType" -d '["name":"Edmond"}'
Or
curl -XPOST "https://xxx-xxxxxxxx.us-west-2.es.amazonaws.com:443/myIndex/myType" -d '["name":"Edmond"}'
It should respond with:
{"_index":"myIndex","_type":"myType","_id":"SOME_ID_#","_version":1,"created":true}
Check in Kibana and you'll see it's there.
So, then in your code, it should be:
Client client = TransportClient.builder().build()
.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName("xxx-xxxxxxxx.us-west-2.es.amazonaws.com"), 80));
Unfortunately, I don't off-hand know how to transmit encrypted via SSL/HTTPS using the transport client. You could try using regular REST calls instead using JERSEY.
Finally, make sure your Elasticsearch access policy is configured properly. Something along the lines of:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": "*",
"Action": "es:*",
"Resource": "arn:aws:es:us-east-1:yyyyyyy:domain/myDomain/*"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": "*",
"Action": "es:*",
"Resource": "arn:aws:es:us-east-1:yyyyyyyyy:domain/myDomain"
}
]
}
NOTE: The above access policy is completely wide open and is not recommended for anything remotely close to production. Just so you know....
Managed elastic search service in AWS does not provide the port for the transport protocol until now.
This question has been answered here ,
Elastic Transport client on AWS Managed ElasticSearch1
There is also a discussion in the AWS forum regarding the transport protocol. Here is the link
What is the port for the transport protocol ?
After lot of search i found an example which used a GET request, so I made minor changes to it for allowing POST requests so that complex queries can be submitted via POST body. The implementation is available at https://github.com/dy10/aws-elasticsearch-query-java
Apart from properly configuring access to you AWS ES (i.e. dont open it to Public), make sure to use https (the above code uses http; just replace http with https in the code and it will work).
Another useful looking but partial implementation is at https://github.com/aws/aws-sdk-java/issues/861