Since AWS MSK supports Kafka versions up to 3.3.1, are there any resources which show how to configure a cluster to authenticate Producers & Consumers with Bearer Tokens, as per KIP-768?
Even though MSK supports Apache Kafka versions up to 3.3.1, at this moment MSK doesn't expose listeners for sasl.mechanism=OAUTHBEARER.
Available mechanisms for SASL_SSL protocol implementation are:
AWS_MSK_IAM - with IAM integration (listeners are on a port 9098)
SCRAM-SHA-512 - with AWS Secrets Manager integration (listeners are on a port 9096)
In addition, MSK supports mTLS implementation for SSL protocol with AWS Certificate Manager integration (listeners are on a port 9094)
Related
In MSK to communicate with brokers by using TLS(SSL) we have used 9094 port and able to communicate MSK cluster with the required keystore and truststore configurations.
And to communicate with brokers by using SASL_SCRAM we have used 9096 port and able to communicate MSK cluster with username and password configurations.
And now we want to configure both SASL_SSL to authenticate client while connecting to broker nodes as a mandatory configurations at client side.
NOTE: We are able to configure both SASL_SSL security protocol on 9096 port but the client configuration considered as the TLS configurations are optional because if we remove TLS properties, still it is able to connect broker nodes.
Im testing this connection with kafka console producer like below:
./bin/kafka-console-producer.sh --broker-list b-1.xxx:9096,b-2xx:9096 --producer.config ./bin/client_sasl_ssl.properties --topic TestTopicSASLSSL
File: client_sasl_ssl.properties
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
#ssl.truststore.location=/path/to/kafka.client.truststore.jks
#ssl.keystore.location=/path/to/kafka.client.keystore.jks
#ssl.keystore.password=psswd
#ssl.key.password=psswd
Please note that im able to produce data with or without the above commented ssl properties, which seems optional.
Any option to mandate both SASL and TLS auth protocol to restrict client connection.
I came across this article for migrating rabbitmq to amazon mq (rabbitmq).
https://aws.amazon.com/blogs/compute/migrating-message-driven-applications-to-amazon-mq-for-rabbitmq/
I see this sentence:
"In Amazon MQ for RabbitMQ, we only support the secure version of AMQP using TLS. The code snippet below demonstrates AMQPS connection using Pika library. Please note that we do not support peer verification on server side."
Does it mean its mandatory to always have a secure connection for amazon mq (rabbitmq) OR its simply saying that if we decide to go for secure connections we must use TLS
from the documentation Encryption in transit
By default, Amazon MQ brokers use the recommended TLS 1.2 to encrypt data. Amazon MQ does not currently support configuring brokers to use different TLS versions.
Key Concepts Security and authentication
Connections to Amazon MQ brokers use Transport Layer Security (TLS)
Supported wire-level protocols
You can access your brokers by using any programming language that ActiveMQ supports and by enabling TLS explicitly for the following protocols:
Infrastructure security in Amazon MQ
You use AWS published API calls to access Amazon MQ through the network. Clients must support Transport Layer Security (TLS) 1.0 or later. We recommend TLS 1.2 or later. Clients must also support cipher suites with perfect forward secrecy (PFS) such as Ephemeral Diffie-Hellman (DHE) or Elliptic Curve Ephemeral Diffie-Hellman (ECDHE). Most modern systems such as Java 7 and later support these modes.
You have to use TLS.
I am hosting a rabbitmq cluster on AWS EC2. Is it possible for my remote application to authenticate a connection to the rabbitmq server using AWS IAM roles or any other way using AWS services? The reason is that my IoT devices will have AWS credentials configured, but I do not want to install SSL certificates individually on each of them. I am using Python pika library and my application currently works with plain credentials (username/password) in the dev environment.
No you cannot although why not store the credentials in secrets manager then allow the credentials for your IoT device to read from that?
I have to connect AWS Elasticsearch with Alexa. According to a file, I have to put
client = new ElasticSearchClient("your.elastic.url", 9300, "your.cluster.name");
What is the elastic URL and 9300 for my AWS hosted ES Cluster?
As mentioned in the comment, AWS ES does not offer the possibility to connect via TCP.
TCP transport
The service supports HTTP on port 80, but does not support TCP
transport.
developerguide aes-supported-resources
But you can try this how-to-use-java-high-level-rest-client-with-spring-boot-to-talk-to-aws-elasticsearch.
Another thing, it totally depends upon the underlying library that is https://github.com/unterstein/elastic-alexa you can give it try with https with out port.
As per your question you can get AWS ES cluster details and endpoint here.
ES -> Dashboard -> select domain
in the Top, you see cluster name and below in endpoint.
I am trying to understand Amazon AWS IoT and I see there is a Device Gateway connected between Sensors/Actuators and AWS (Rules Engine). I understand Sensors/Actuators are MQTT Clients which can Publish/Subscribe to the topics.
Device Gateway also supports Publish/Subscribe pattern is that means
Device Gateway is also a MQTT Client ?
Then where is the MQTT broker?
OR Device Gateway is Broker + communication interface to AWS-IoTservices?
I have attached an Image Amazon presentation on AWS IoT for the reference:
The gateway is a MQTT broker, but not a fully compliant one. Some of the differences are:
no retained message support
no QOS2 support
no message persistence
Full details can be found here http://docs.aws.amazon.com/iot/latest/developerguide/protocols.html