I created a DCOS setup on AWS using default config
I add two kafka brokers using CLI
(DCOS) PS C:\DCOS> dcos kafka broker list
brokers:
id: 1
active: false
state: stopped
resources: cpus:2.00, mem:8192, heap:1024, port:auto
failover: delay:1m, max-delay:10m
stickiness: period:10m, expires:2016-03-22 15:58:51-04
When I start broker I see that offer from master was declined
I0322 20:56:38.954476 1316 master.cpp:5350] Sending 2 offers to framework d8c03032-ebab-4c88-80cb-e2de92e3c4c4-0001 (kafka) at scheduler-fff6da19-e31e-4518-864e-2dfcdc31a5d2#10.0.3.104:53766
I0322 20:56:38.966846 1320 master.cpp:3673] Processing DECLINE call for offers: [ d8c03032-ebab-4c88-80cb-e2de92e3c4c4-O7389 ] for framework d8c03032-ebab-4c88-80cb-e2de92e3c4c4-0001 (kafka) at scheduler-fff6da19-e31e-4518-864e-2dfcdc31a5d2#10.0.3.104:53766
I0322 20:56:38.967591 1319 master.cpp:3673] Processing DECLINE call for offers: [ d8c03032-ebab-4c88-80cb-e2de92e3c4c4-O7390 ] for framework d8c03032-ebab-4c88-80cb-e2de92e3c4c4-0001 (kafka) at scheduler-fff6da19-e31e-4518-864e-2dfcdc31a5d2#10.0.3.104:53766
I0322 20:56:40.043771 1318 http.cpp:512] HTTP GET for /master/state-summary from 10.0.6.116:60000 with User-Agent='python-requests/2.6.0 CPython/3.4.2 Linux/4.1.7-coreos-r1'
I'm not able to find any relevant logs on the slaves to see what is going on.
/var/log/mesos has some files with no relevant info. As per the doc I should see syslogs in /var/log/messages but I don't see that file. The default config provisions CoreOS. I tried journalctl command but didn't find anything there too. Not sure how to debug this.
Related
I´m trying to create a Cognito using localstack locally but when I run:
awslocal cognito-idp create-user-pool --pool-name test
as mentioned on the docs I get the following error:
2022-11-01T19:21:56.136 ERROR --- [ asgi_gw_0] l.aws.handlers.logging : exception during call chain:
2022-11-01T19:21:56.136 INFO --- [ asgi_gw_0] l.aws.handlers.service : API action 'CreateUserPool' for service 'cognito-idp' not yet implemented or pro feature - check https://docs.localstack.cloud/aws/feature-coverage for further information
2022-11-01T19:21:56.137 INFO --- [ asgi_gw_0] localstack.request.aws : AWS cognito-idp.CreateUserPool => 501 (InternalFailure)
Anyone face this issue?
As documented on the on the localstack getting started page, certain features are limited to paying members of Localstack Pro.
The Pro version of LocalStack supports additional APIs and advanced features. You can find a comprehensive list of supported APIs on our ⭐ Feature Coverage page.
Following the link (which is the same link as in your error message you posted), Cognito is a paid feature of localstack. You have to pay for localstack Pro (or use the Pro trial) to get access to paid features.
Cognito Identity Provider (IdP) (Pro)
There is a guide on how to get started with localstack pro here.
Specs:
The serverless Amazon MSK that's in preview.
t2.xlarge EC2 instance with Amazon Linux 2
Installed Kafka from https://dlcdn.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz
openjdk version "11.0.13" 2021-10-19 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.13+8-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8-LTS, mixed mode,
sharing)
Gradle 7.3.3
https://github.com/aws/aws-msk-iam-auth, successfully built.
I also tried adding IAM authentication information, as recommended by the Amazon MSK Library for AWS Identity and Access Management. It says to add the following in config/client.properties:
# Sets up TLS for encryption and SASL for authN.
security.protocol = SASL_SSL
# Identifies the SASL mechanism to use.
sasl.mechanism = AWS_MSK_IAM
# Binds SASL client implementation.
# sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required;
# Encapsulates constructing a SigV4 signature based on extracted credentials.
# The SASL client bound by "sasl.jaas.config" invokes this class.
sasl.client.callback.handler.class = software.amazon.msk.auth.iam.IAMClientCallbackHandler
# Binds SASL client implementation. Uses the specified profile name to look for credentials.
sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required awsProfileName="kafka-client";
And kafka-client is the IAM role attached to the EC2 instance as an instance profile.
Networking: I used VPC Reachability Analyzer to confirm that the security groups are configured correctly and the EC2 instance I'm using as a Producer can reach the serverless MSK cluster.
What I'm trying to do: create a topic.
How I'm trying: bin/kafka-topics.sh --create --partitions 1 --replication-factor 1 --topic quickstart-events --bootstrap-server boot-zclcyva3.c2.kafka-serverless.us-east-2.amazonaws.com:9098
Result:
Error while executing topic command : Timed out waiting for a node assignment. Call: createTopics
[2022-01-17 01:46:59,753] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createTopics
(kafka.admin.TopicCommand$)
I'm also trying: with the plaintext port of 9092. (9098 is the IAM-authentication port in MSK, and serverless MSK uses IAM authentication by default.)
All the other posts I found on SO about this node assignment error didn't include MSK. I tried suggestions like uncommenting the listener setting in server.properties, but that didn't change anything.
Installing kcat for troubleshooting didn't work for me, since there's no out-of-the box installation for the yum package manager, which Amazon Linux 2 uses, and since these instructions failed for me at checking for libcurl (by compile)... failed (fail).
The Question: Any other tips on solving this "node assignment" error?
The documentation has been updated recently, I was able to follow it end to end without any issue (The IAM policy is now correct)
https://docs.aws.amazon.com/msk/latest/developerguide/serverless-getting-started.html
The created properties file is not automatically used; your command needs to include --command-config client.properties, where this properties file is documented at the MSK docs on the linked IAM page.
Extract...
ssl.truststore.location=<PATH_TO_TRUST_STORE_FILE>
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
Alternatively, if the plaintext port didn't work, then you have other networking issues
Beyond these steps, I suggest reaching out to MSK support, and telling them to update the "Create a Topic" page to no longer use Zookeeper, keeping in mind that Kafka 3.0 is not (yet) supported
I have set-up Kafka on my GCP cloud shell terminals and I am trying to read data from Kafka topic to cloud dataflow using flex template.
I am able to read data using console consumer on the terminal.
user#cloudshell:~/kafka_project/kafka/bin $ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic txtopic --from-beginning
Sample data:
{"prod_id": 100, "prod_name": "Redmi 100 X", "price": 8000,
"event_time": "2021-12-04 16:32:32"}
{"prod_id": 102, "prod_name": "Redmi 102 X", "price": 8500,
"event_time": "2021-12-04 16:33:02"}
But while running dataflow pipeline I am getting the below error:
{"container_id":"2ab55e9b52e11de98a17a4546ca119af7b607490008bdb0be273e0cb4c13f6dc","severity":"INFO","time":"2021/12/04 16:39:35.135010","line":"exec.go:64","message":"Caused by: **org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata**"}
I suspect cloud dataflow is not able to connect to the bootstrap server or topic. But I am unable to figure out the solution. Please help.
I have been following the documentation in every step, and I didn't face any errors. Configured, deployed and made a subscription to hello/world topic just as the documentation detailed. However, when I arrived at the testing step here: https://docs.aws.amazon.com/greengrass/latest/developerguide/lambda-check.html
No messages were showing up on the IoT console (subscription view hello/world)! I am using Greengrass core daemon which runs on my Ubuntu machine, it is active and listens to port 8000. I don't think there is anything wrong with my local device because the group was deployed successfully and because I see the communications going both ways on Wireshark.
I have these logs on my machine: /home/##/Desktop/greengrass/ggc/var/log/system/runtime.log:
[2019-09-28T06:57:42.492-07:00][INFO]-===========================================
[2019-09-28T06:57:42.492-07:00][INFO]-Greengrass Version: 1.9.3-RC3
[2019-09-28T06:57:42.492-07:00][INFO]-Greengrass Root: /home/##/Desktop/greengrass
[2019-09-28T06:57:42.492-07:00][INFO]-Greengrass Write Directory: /home/##/Desktop/greengrass/ggc
[2019-09-28T06:57:42.492-07:00][INFO]-Group File Directory: /home/##/Desktop/greengrass/ggc/deployment/group
[2019-09-28T06:57:42.492-07:00][INFO]-Default Lambda UID: 122
[2019-09-28T06:57:42.492-07:00][INFO]-Default Lambda GID: 127
[2019-09-28T06:57:42.492-07:00][INFO]-===========================================
[2019-09-28T06:57:42.492-07:00][INFO]-The current core is using the AWS IoT certificates with fingerprint. {"fingerprint": "90##4d"}
[2019-09-28T06:57:42.492-07:00][INFO]-Will persist worker process info. {"dir": "/home/##/Desktop/greengrass/ggc/ggc/core/var/worker/processes"}
[2019-09-28T06:57:42.493-07:00][INFO]-Will persist worker process info. {"dir": "/home/##/Desktop/greengrass/ggc/ggc/core/var/worker/processes"}
[2019-09-28T06:57:42.494-07:00][INFO]-No proxy URL found.
[2019-09-28T06:57:42.495-07:00][INFO]-Started Deployment Agent to listen for updates. [2019-09-28T06:57:42.495-07:00][INFO]-Connecting with MQTT. {"endpoint": "a6##ws-ats.iot.us-east-2.amazonaws.com:8883", "clientId": "simulators_gg_Core"}
[2019-09-28T06:57:42.497-07:00][INFO]-The current core is using the AWS IoT certificates with fingerprint. {"fingerprint": "90##4d"}
[2019-09-28T06:57:42.685-07:00][INFO]-MQTT connection successful. {"attemptId": "GVko", "clientId": "simulators_gg_Core"}
[2019-09-28T06:57:42.685-07:00][INFO]-MQTT connection established. {"endpoint": "a6##ws-ats.iot.us-east-2.amazonaws.com:8883", "clientId": "simulators_gg_Core"}
[2019-09-28T06:57:42.685-07:00][INFO]-MQTT connection connected. Start subscribing. {"clientId": "simulators_gg_Core"}
[2019-09-28T06:57:42.685-07:00][INFO]-Deployment agent connected to cloud.
[2019-09-28T06:57:42.685-07:00][INFO]-Start subscribing. {"numOfTopics": 2, "clientId": "simulators_gg_Core"}
[2019-09-28T06:57:42.685-07:00][INFO]-Trying to subscribe to topic $aws/things/simulators_gg_Core-gda/shadow/update/delta
[2019-09-28T06:57:42.727-07:00][INFO]-Trying to subscribe to topic $aws/things/simulators_gg_Core-gda/shadow/get/accepted
[2019-09-28T06:57:42.814-07:00][INFO]-All topics subscribed. {"clientId": "simulators_gg_Core"}
[2019-09-28T06:58:57.888-07:00][INFO]-Daemon received signal: terminated. [2019-09-28T06:58:57.888-07:00][INFO]-Shutting down daemon.
[2019-09-28T06:58:57.888-07:00][INFO]-Stopping all workers.
[2019-09-28T06:58:57.888-07:00][INFO]-Lifecycle manager is stopped.
[2019-09-28T06:58:57.888-07:00][INFO]-IPC server stopped.
/home/##/Desktop/greengrass/ggc/var/log/system/localwatch/localwatch.log:
[2019-09-28T06:57:42.491-07:00][DEBUG]-will keep the log files for the following lambdas {"readingPath": "/home/##/Desktop/greengrass/ggc/var/log/user", "lambdas": "map[]"}
[2019-09-28T06:57:42.492-07:00][WARN]-failed to list the user log directory {"path": "/home/##/Desktop/greengrass/ggc/var/log/user"}
Thanks in advance.
I had a similar issue on another platform (Jetson Nano). I could not get a response after going through the AWS instructions for setting up a simple Lambda using IOT Greengrass. In my search for answers I discovered that AWS has a qualification test script for any device you connect.
It goes through an automated process of deploying and testing a lambda function(as well as other functionality) and reports results for each step and docs provide troubleshooting info for failures.
By going through those tests I was able to narrow down the issues with my setup, installation, and configuration. The testing docs give pointers to troubleshoot test results. Here is a link to the test: https://docs.aws.amazon.com/greengrass/latest/developerguide/device-tester-for-greengrass-ug.html
If you follow the 'Next Topic' links, it will take you through the complete test. Let me warn you that its extensive, and will take some time, but for me it gave a lot of detailed insight that a hello world does not.
Our AWS MSK cluster is not automatically creating topics when a producer first publishes a message to the topic. This is surprising because the default behaviour according to https://kafka.apache.org/documentation/#brokerconfigs should have auto.create.topics.enable = true.
I have attempted to set this configuration on my brokers but I am unable because the config is read-only for dynamic updates.
$ kafka-configs --bootstrap-server $KAFKA_BROKER --entity-type brokers --entity-default --alter --add-config auto.create.topics.enable=true
Error while executing config command with args '--bootstrap-server $KAFKA_BROKER --entity-type brokers --entity-default --alter --add-config auto.create.topics.enable=true'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Invalid config value for resource ConfigResource(type=BROKER, name=''): Cannot update these configs dynamically: Set(auto.create.topics.enable)
note: I replaced our broker's IP with $KAFKA_BROKER in the output
How can I configure my AWS MSK Kafka cluster to enable auto creation of topics?
Update
You can now customise MSK configuration:
auto.create.topics.enable: Enables topic autocreation on the server.
Therefore, auto.create.topics.enable=true (defaults to false) should now do the trick as usual.
Although auto.create.topics.enable normally defaults to true, this is not the case for AWS MSK. According to these slides presented in Berlin's AWS Kafka Summit this February, the following are the default configuration parameters for Amazon MSK:
offsets.topic.replication.factor 3
transaction.state.log.replication.factor 3
transaction.state.log.min.isr 2
auto.create.topics.enable False
default.replication.factor 3
min.insync.replicas 2
unclean.leader.election.enable True
auto.leader.rebalance.enable True
authorizer.class.name kafka.security.auth.SimpleAclAuthorizer
group.initial.rebalance.delay.ms 3000
log.retention.hours 168
Note that even Kafka docs mention that the suggested production configuration should be set to false.
Moving forward you can now set this parameter to true by customizing the configuration, however it will stay false by default since that is the recommended production configuration.