I am using ansible to create an aws ami and instance from that ami
but however when i run "ansible-playbook -i ../hosts ec2.yml" command it fails with a following error,
"An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ClientError: An error occurred (AuthFailure) when calling the CreateImage operation: AWS was not able to validate the provided access credentials
fatal: [localhost]: FAILED! => {"boto3_version": "1.9.50", "botocore_version": "1.12.50", "changed": false, "error": {"code": "AuthFailure", "message": "AWS was not able to validate the provided access credentials"}, "msg": "Error registering image: An error occurred (AuthFailure) when calling the CreateImage operation: AWS was not able to validate the provided access credentials", "response_metadata": {"http_headers": {"date": "Sat, 24 Nov 2018 11:03:48 GMT", "server": "AmazonEC2", "transfer-encoding": "chunked"}, "http_status_code": 401, "request_id": "17ae8e55-7cf4-44e3-9c23-498a2913b4f0", "retry_attempts": 0}}
to retry, use: --limit #/etc/ansible/yml/ec2.retry
"
I have setup awscli and configure my access and secret key with default region (with administrative rights).
I am using ubuntu 14.04 version and my date command output is, "Sat Nov 24 16:47:44 IST 2018"
Any solution?
Related
Using AWS CDK, I am trying to deploy the Docker image with lambda function on AWS. And I am getting the following error.
[100%] fail: docker login --username AWS --password-stdin https://XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com exited with error code 1: Error saving credentials: error storing credentials - err: exit status 1, out: `Post "http://ipc/registry/credstore-updated": dial unix /Users/my_mac/Library/Containers/com.docker.docker/Data/backend.sock: connect: connection refused`
❌ MyService (prj-development) failed: Error: Failed to publish one or more assets. See the error messages above for more information.
at publishAssets (/Users/my_mac/.npm/_npx/8365afa3375eae8d/node_modules/aws-cdk/lib/util/asset-publishing.ts:44:11)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at CloudFormationDeployments.publishStackAssets (/Users/my_mac/.npm/_npx/8365afa3375eae8d/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:464:7)
at CloudFormationDeployments.deployStack (/Users/my_mac/.npm/_npx/8365afa3375eae8d/node_modules/aws-cdk/lib/api/cloudformation-deployments.ts:339:7)
at CdkToolkit.deploy (/Users/my_mac/.npm/_npx/8365afa3375eae8d/node_modules/aws-cdk/lib/cdk-toolkit.ts:209:24)
at initCommandLine (/Users/my_mac/.npm/_npx/8365afa3375eae8d/node_modules/aws-cdk/lib/cli.ts:341:12)
Failed to publish one or more assets. See the error messages above for more information.
make: *** [deploy-local] Error 1
What can I do, please?
Before deployment, open the Docker app/daemon on your machine.
Trying to install and configure AWS toolkit to Visual Studio Code.
Command Command palette->Create Credentials profile brings two files :
credentials file content
[default]
aws_access_key_id = XXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXX
config file content
[default]
region = eu-central-1
output = text
Choose Command palette->AWS profile profile:default generates error:
2022-02-03 10:03:51 [ERROR]: log level: info
2022-02-03 10:03:52 [INFO]: Retrieving AWS endpoint data
2022-02-03 10:03:52 [INFO]: OS: Windows_NT x64 10.0.19043
2022-02-03 10:03:52 [INFO]: Visual Studio Code Extension Host Version: 1.63.2
2022-02-03 10:03:52 [INFO]: AWS Toolkit Version: 1.35.0
2022-02-03 10:03:52 [INFO]: telemetry cache not found: 'c:\Users\g\AppData\Roaming\Code\User\globalStorage\amazonwebservices.aws-toolkit-vscode\telemetryCache'
2022-02-03 10:04:18 [ERROR]: Error getting AccountId: [InvalidClientTokenId: The security token included in the request is invalid.
at constructor.h (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:5:9005)
at constructor.callListeners (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:6:21079)
at constructor.emit (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:6:20788)
at constructor.emitEvent (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:6:6641)
at constructor.e (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:6:2227)
at U.runTo (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:18:1767)
at c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:18:1979
at constructor.<anonymous> (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:6:2438)
at constructor.<anonymous> (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:6:6696)
at constructor.callListeners (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:6:21183)
at constructor.emit (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:6:20788)
at constructor.emitEvent (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:6:6641)
at constructor.e (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:6:2227)
at U.runTo (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:18:1767)
at c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:18:1979
at constructor.<anonymous> (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:6:2438)
at constructor.<anonymous> (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:6:6696)
at constructor.callListeners (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:6:21183)
at e (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:6:20964)
at IncomingMessage.<anonymous> (c:\Users\g\.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.35.0\dist\extension.js:1:209012)
at IncomingMessage.emit (events.js:327:22)
at IncomingMessage.EventEmitter.emit (domain.js:467:12)
at endReadableNT (internal/streams/readable.js:1327:12)
at processTicksAndRejections (internal/process/task_queues.js:80:21)] {
code: 'InvalidClientTokenId',
time: 2022-02-03T08:04:18.158Z,
requestId: '00c18899-6f97-40c1-9788-b2156b350ebb',
statusCode: 403,
retryable: false,
retryDelay: 83.95345343935642
}
2022-02-03 10:04:18 [ERROR]: login: failed to connect with "profile:default": Could not determine Account Id for credentials
How to connect AWS toolkit to my VSCode?
You'll need to get the access and secret key from AWS and insert them in place of the XXXXXXX placeholders.
You can get this information in the AWS Cloud -> IAM -> Access Management -> Users -> Select your user -> Security credentials -> Access Keys
You will find here the Access Key ID, but the Secret Key is only shown once when you are creating this item. You maybe have it stored somewhere, or you can create another Access Key pair and use that.
I have done this and I can connect to AWS Toolkit fine.
AWS Toolkit Config
Couple of things I tried to make it work:
Ensure that the credentials file is in C:\Users\UserName\ .aws\ {credentials}
Was prompted that default region for this profile was xxx. Changed it accordingly.
Restarted the VS code(yeah, I know) :D
Chose the profile and it opened okay.
I have been banging my head on this for a few days and I am coming up empty. I have an API that I can run locally just fine via the command line-
sam local start-api
or
sam local invoke HelloWorldFunction
I USED to be able to run the same api via my launch.json configuration as well but suddenly for no reason I have been able to identify, I am now getting:
2021-05-19 12:18:32 [ERROR]: Timeout while waiting for command: "sam local start-api"
2021-05-19 12:18:32 [ERROR]: Failed to start local API Gateway: Timeout while waiting for command: "sam local start-api"
2021-05-19 12:18:32 [INFO]: Sending request to local API: http://127.0.0.1:5858/hello?
My launch.json looks like this:
{
"type": "aws-sam",
"request": "direct-invoke",
"name": "Simple API lambda-nodejs14.x:HelloWorldFunction (nodejs14.x)",
"invokeTarget": {
"target": "api",
"templatePath": "${workspaceFolder}/lambda-nodejs14.x/template.yaml",
"logicalId": "HelloWorldFunction"
},
"api": {
"path": "/hello",
"httpMethod": "get",
"payload": {
"json": {}
}
},
"lambda": {
"runtime": "nodejs14.x"
},
"sam": {},
"aws":{}
}
I have reduced this to the simplest possible scenario by trying with the helloWorld test project.
My aws_toolkit log looks like this:
2021-05-19 12:27:07 [INFO]: Build complete. 2021-05-19 12:27:07
[INFO]: Starting SAM application locally 2021-05-19 12:27:07 [INFO]:
Running: (not started) [C:\Program Files\Amazon\AWSSAMCLI\bin\sam.cmd
local start-api --template
C:\Users\USERTEST\AppData\Local\Temp\aws-toolkit-vscode\vsctklc1oS3\output\template.yaml
--env-vars C:\Users\USERTEST\AppData\Local\Temp\aws-toolkit-vscode\vsctklc1oS3\env-vars.json
--port 5858 --debug-port 5859] 2021-05-19 12:27:07 [INFO]: AWS.running.command 2021-05-19 12:27:07 [ERROR]: Timeout while waiting
for command: "sam local start-api" 2021-05-19 12:27:07 [ERROR]: Failed
to start local API Gateway: Timeout while waiting for command: "sam
local start-api" 2021-05-19 12:27:07 [WARN]: [Error: Timeout while
waiting for command: "sam local start-api" at
c:\Users\BogartLisa.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.25.0\dist\extension.js:2452:1782
at async lv.invoke
(c:\Users\BogartLisa.vscode\extensions\amazonwebservices.aws-toolkit-vscode-1.25.0\dist\extension.js:2452:1677)]
2021-05-19 12:27:07 [INFO]: Sending request to local API:
http://127.0.0.1:5858/hello? 2021-05-19 12:27:07 [VERBOSE]: SAM:
command exited (code: null): PID 14376: [C:\Program
Files\Amazon\AWSSAMCLI\bin\sam.cmd local start-api --template
C:\Users\USERTEST\AppData\Local\Temp\aws-toolkit-vscode\vsctklc1oS3\output\template.yaml
--env-vars C:\Users\USERTEST\AppData\Local\Temp\aws-toolkit-vscode\vsctklc1oS3\env-vars.json
--port 5858 --debug-port 5859] 2021-05-19 12:27:12 [INFO]: Successfully sent a telemetry batch of 1 2021-05-19 12:27:13 [ERROR]:
Local API failed to respond (ECONNREFUSED) after 30 retries, path:
/hello
My vs code:
Version: 1.56.2 (user setup)
Commit: 054a9295330880ed74ceaedda236253b4f39a335
Date: 2021-05-12T17:13:13.157Z
Electron: 12.0.4
Chrome: 89.0.4389.114
Node.js: 14.16.0
V8: 8.9.255.24-electron.0
OS: Windows_NT x64 10.0.17134
aws --version
aws-cli/2.1.27 Python/3.7.9 Windows/10 exe/AMD64 prompt/off
Does Anyone have any ideas for me?? I am out of ideas
The answer to this problem was to increase the configured max time to wait to attach to the sam cli:
Settings > Extensions > AWS Configuration > Samcli > Debug > Attach > Timeout:Millis
Maximum time (in milliseconds) to wait for SAM output while starting a Local Lambda session
Mine was set at the default of 30 seconds and unfortunately for me, it was taking in excess of 1.5 minutes to attach (but that's a whole 'nother problem).
I have a running debezium cluster in AWS, no issues with that. I want to give a try with AWS MSK. So I launched a cluster. Then I launched an EC2 for running my connectors.
Then installed confluent-kafka
sudo apt-get update && sudo apt-get install confluent-platform-2.12
By default the AWS MSK doesn't have schema registry, So I configured it from the connector EC2
Schema registry conf file:
kafkastore.connection.url=z-1.bhuvi-XXXXXXXXX.amazonaws.com:2181,z-3.bhuvi-XXXXXXXXX.amazonaws.com:2181,z-2.bhuvi-XXXXXXXXX.amazonaws.com:2181
kafkastore.bootstrap.servers=PLAINTEXT://b-2.bhuvi-XXXXXXXXX.amazonaws.com:9092,PLAINTEXT://b-4.bhuvi-XXXXXXXXX.amazonaws.com:9092,PLAINTEXT://b-1.bhuvi-XXXXXXXXX.amazonaws.com:9092
Then /etc/kafka/connect-distributed.properties file
bootstrap.servers=b-4.bhuvi-XXXXXXXXX.amazonaws.com:9092,b-3.bhuvi-XXXXXXXXX.amazonaws.com:9092,b-2.bhuvi-XXXXXXXXX.amazonaws.com:9092
plugin.path=/usr/share/java,/usr/share/confluent-hub-components
Install connector:
confluent-hub install debezium/debezium-connector-mysql:latest
start the service
systemctl start confluent-schema-registry
systemctl start confluent-connect-distributed
Now everything started. Then I created a mysql.json file.
{
"name": "mysql-connector-db01",
"config": {
"name": "mysql-connector-db01",
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"database.server.id": "1",
"tasks.max": "3",
"database.history.kafka.bootstrap.servers": "172.31.47.152:9092,172.31.38.158:9092,172.31.46.207:9092",
"database.history.kafka.topic": "schema-changes.mysql",
"database.server.name": "mysql-db01",
"database.hostname": "172.31.84.129",
"database.port": "3306",
"database.user": "bhuvi",
"database.password": "my_stong_password",
"database.whitelist": "proddb,test",
"internal.key.converter.schemas.enable": "false",
"key.converter.schemas.enable": "false",
"internal.key.converter": "org.apache.kafka.connect.json.JsonConverter",
"internal.value.converter.schemas.enable": "false",
"value.converter.schemas.enable": "false",
"internal.value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"transforms": "unwrap",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState"
"transforms.unwrap.add.source.fields": "ts_ms",
}
}
Create debezium connector
curl -X POST -H "Accept: application/json" -H "Content-Type: application/json" http://localhost:8083/connectors -d #mysql.josn
Then its stated giving this error in the connector EC2.
Dec 20 11:42:36 ip-172-31-44-220 connect-distributed[2630]: [2019-12-20 11:42:36,290] WARN [Producer clientId=producer-3] Got error produce response with correlation id 844 on topic-partition connect-configs-0, retrying (2147482809 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender:637)
Dec 20 11:42:36 ip-172-31-44-220 connect-distributed[2630]: [2019-12-20 11:42:36,391] WARN [Producer clientId=producer-3] Got error produce response with correlation id 845 on topic-partition connect-configs-0, retrying (2147482808 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender:637)
Dec 20 11:42:36 ip-172-31-44-220 connect-distributed[2630]: [2019-12-20 11:42:36,492] WARN [Producer clientId=producer-3] Got error produce response with correlation id 846 on topic-partition connect-configs-0, retrying (2147482807 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender:637)
Dec 20 11:42:36 ip-172-31-44-220 connect-distributed[2630]: [2019-12-20 11:42:36,593] WARN [Producer clientId=producer-3] Got error produce response with correlation id 847 on topic-partition connect-configs-0, retrying (2147482806 attempts left). Error: NOT_ENOUGH_REPLICAS (org.apache.kafka.clients.producer.internals.Sender:637)
It never stops this error message.
Describe of connect-configs
Topic:connect-configs PartitionCount:1 ReplicationFactor:1 Configs:cleanup.policy=compact
Topic: connect-configs Partition: 0 Leader: 2 Replicas: 2 Isr: 2
MSK sets min.in.sync.replicas to 2 for all topics by default (see https://docs.aws.amazon.com/msk/latest/developerguide/msk-default-configuration.html)
It possible that Kafka Connect is producing using ACKs="all" and, since you only have one copy of your topic, it never achieves enough quorum.
I have created a service account in Gcloud.
Installed Gcloud on my mac.
When ever I run my packer template, it complains about this account which I have no idea where it is coming from.
Packer template:
{
"builders": [
{
"type": "googlecompute",
"account_file": "/Users/Joe/Downloads/account.json",
"project_id": "rare-truck-123456",
"source_image": "centos-7-v20180129",
"zone": "us-west1-a",
"ssh_username": "centos"
}
]
}
Error:
➜ packer git:(master) ✗ packer build release_google_image.json
googlecompute output will be in this color.
==> googlecompute: Checking image does not exist...
==> googlecompute: Creating temporary SSH key for instance...
==> googlecompute: Using image: centos-7-v20180129
==> googlecompute: Creating instance...
googlecompute: Loading zone: us-west1-a
googlecompute: Loading machine type: n1-standard-1
googlecompute: Loading network: default
googlecompute: Requesting instance creation...
googlecompute: Waiting for creation operation to complete...
==> googlecompute: Error creating instance: 1 error(s) occurred:
==> googlecompute:
==> googlecompute: * The resource '123412341234-compute#developer.gserviceaccount.com' of type 'serviceAccount' was not found.
Build 'googlecompute' errored: Error creating instance: 1 error(s) occurred:
* The resource '123412341234-compute#developer.gserviceaccount.com' of type 'serviceAccount' was not found.
==> Some builds didn't complete successfully and had errors:
--> googlecompute: Error creating instance: 1 error(s) occurred:
* The resource '123412341234-compute#developer.gserviceaccount.com' of type 'serviceAccount' was not found.
==> Builds finished but no artifacts were created.
Why is it trying to use 123412341234-compute#developer.gserviceaccount.com?
I created a service account with compute admin v1 permissions under my project in google cloud and I downloaded my json file and renamed it to accounts.json. The name of that service account is different (release-builder#rare-truck-123456.iam.gserviceaccount.com), but packer seems to ignore it and go after some strange account.
Even my cli command gcloud info gives back the right service account:
Google Cloud SDK [188.0.1]
Platform: [Mac OS X, x86_64] ('Darwin', 'Alexs-MacBook-Pro.local', '16.7.0', 'Darwin Kernel Version 16.7.0: Thu Jan 11 22:59:40 PST 2018; root:xnu-3789.73.8~1/RELEASE_X86_64', 'x86_64', 'i386')
Python Version: [2.7.14 (default, Sep 25 2017, 09:53:22) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)]]
Python Location: [/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python]
Site Packages: [Disabled]
Installation Root: [/Users/Joe/Downloads/google-cloud-sdk]
Installed Components:
core: [2018.02.08]
gsutil: [4.28]
bq: [2.0.28]
System PATH: [/Users/Joe/Downloads/google-cloud-sdk/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin]
Python PATH: [/Users/Joe/Downloads/google-cloud-sdk/lib/third_party:/Users/Joe/Downloads/google-cloud-sdk/lib:/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python27.zip:/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7:/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin:/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac:/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages:/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk:/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old:/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload]
Cloud SDK on PATH: [True]
Kubectl on PATH: [False]
Installation Properties: [/Users/Joe/Downloads/google-cloud-sdk/properties]
User Config Directory: [/Users/Joe/.config/gcloud]
Active Configuration Name: [default]
Active Configuration Path: [/Users/Joe/.config/gcloud/configurations/config_default]
Account: [release-builder#rare-truck-123456.iam.gserviceaccount.com]
Project: [rare-truck-123456]
Current Properties:
[core]
project: [rare-truck-123456]
account: [release-builder#rare-truck-123456.iam.gserviceaccount.com]
disable_usage_reporting: [True]
[compute]
region: [us-west1]
zone: [us-west1-a]
Logs Directory: [/Users/Joe/.config/gcloud/logs]
Last Log File: [/Users/Joe/.config/gcloud/logs/2018.02.09/15.51.18.911677.log]
git: [git version 2.14.3 (Apple Git-98)]
ssh: [OpenSSH_7.4p1, LibreSSL 2.5.0]
Google Compute Engine instances use default services account to have a better integration with Google Platform.
As you can read on the documentation [1]: "(...)If the user will be managing virtual machine instances that are configured to run as a service account, you must also grant the roles/iam.serviceAccountActor role."
You need to add the role "Service Account User" to your service account in order to be able to create Google Compute Engine instances.
[1] https://cloud.google.com/iam/docs/understanding-roles#compute_name_short_roles