I'm trying to create a cloudformation for a Task Definition but I keep running into validation errors on the command portion of the Task Definition. I've tried a few approaches and can't seem to get it right.
I've been following the documentation here.
Below are 2 of my main attempts and their respective errors:
Command: !Sub
- "--config config.yaml --env ${environment}"
- { environment: !FindInMap [EnvironmentVariables, !Ref Environment, Environment] }
Environment] }
Error:
#/ContainerDefinitions/0/Command: expected type: JSONArray, found: String)
Command: !Sub
- [--config, config.yaml, --env, ${environment}]
- { environment: !FindInMap [EnvironmentVariables, !Ref Environment, Environment] }
Template format error: YAML not well-formed.
There seems to be some issue with how !Sub interacts with a Yaml list but I'm not exactly sure what causes this to be the case.
However the below piece of Yaml managed to solve my problem.
Command:
- --config
- config.yaml
- --env
- !FindInMap [EnvironmentVariables, !Ref Environment, Environment]
Related
Just curious, why isn't there a helm cloud builder officially supported? It seems like a very common requirement, yet I'm not seeing one in the list here:
https://github.com/GoogleCloudPlatform/cloud-builders
I was previously using alpine/helm in my cloudbuild.yaml for my helm deployment as follows:
steps:
# Build app image
- name: gcr.io/cloud_builders/docker
args:
- build
- -t
- $_IMAGE_REPO/$_CONTAINER_NAME:$COMMIT_SHA
- ./cloudbuild/$_CONTAINER_NAME/
# Push my-app image to Google Cloud Registry
- name: gcr.io/cloud-builders/docker
args:
- push
- $_IMAGE_REPO/$_CONTAINER_NAME:$COMMIT_SHA
# Configure a kubectl workspace for this project
- name: gcr.io/cloud-builders/kubectl
args:
- cluster-info
env:
- CLOUDSDK_COMPUTE_REGION=$_CUSTOM_REGION
- CLOUDSDK_CONTAINER_CLUSTER=$_CUSTOM_CLUSTER
- KUBECONFIG=/workspace/.kube/config
# Deploy with Helm
- name: alpine/helm
args:
- upgrade
- -i
- $_CONTAINER_NAME
- ./cloudbuild/$_CONTAINER_NAME/k8s
- --set
- image.repository=$_IMAGE_REPO/$_CONTAINER_NAME,image.tag=$COMMIT_SHA
- -f
- ./cloudbuild/$_CONTAINER_NAME/k8s/values.yaml
env:
- KUBECONFIG=/workspace/.kube/config
- TILLERLESS=false
- TILLER_NAMESPACE=kube-system
- USE_GKE_GCLOUD_AUTH_PLUGIN=True
timeout: 1200s
substitutions:
# substitutionOption: ALLOW_LOOSE
# dynamicSubstitutions: true
_CUSTOM_REGION: us-east1
_CUSTOM_CLUSTER: demo-gke
_IMAGE_REPO: us-east1-docker.pkg.dev/fakeproject/my-docker-repo
_CONTAINER_NAME: app2
options:
logging: CLOUD_LOGGING_ONLY
# In this option we are providing the worker pool name that we have created in the previous step
workerPool:
'projects/fakeproject/locations/us-east1/workerPools/cloud-build-pool'
And this was working with no issues. Then recently it just started failing with the following error so I'm guessing a change was made recently:
Error: Kubernetes cluster unreachable: Get "https://10.10.2.2/version": getting credentials: exec: executable gke-gcloud-auth-plugin not found"
I get this error regularly on VM's and can workaround it by setting USE_GKE_GCLOUD_AUTH_PLUGIN=True, but that does not seem to fix the issue here if I add it to the env section. So I'm looking for recommendations on how to use helm with Cloud Build. alpine/helm was just something I randomly tried and was working for me up until now, but there's probably better solutions out there.
Thanks!
I am trying to setup the GCP cloudbuild pipeline for my bitbucket repo. However, it's not being triggered.
I am using following sample from their own website.
steps:
- name: 'ubuntu'
entrypoint: 'bash'
args:
- '-c'
- |
echo "Hello, world!" > /persistent_volume/file
volumes:
- name: 'myvolume'
path: '/persistent_volume'
- name: 'ubuntu'
entrypoint: 'bash'
args:
- '-c'
- |
cat /persistent_volume/file
volumes:
- name: 'myvolume'
path: '/persistent_volume'
I have following action enabled for webhook
even if if I keep this simple singe step it's not working.
steps:
- name: 'ubuntu'
entrypoint: 'bash'
args:
- '-c'
- |
echo "Hello, world!"
Error
If I keep in trigger in global region it complain about vpc-sc error
RESOURCES_NOT_IN_SAME_SERVICE_PERIMETER
if I move the pipeline to northamerical-northeast1 it says
triggerError spanner trigger (923134934784, test) not found
I've managed to replicate your issue by following this link on creating webhook triggers and got the same error.
{
"error": {
"code": 404,
"message": "triggerError spanner trigger (${PROJECT_ID}, ${TRIGGER_NAME}) not found",
"status": "NOT_FOUND"
}
}
The weird thing is even if I selected a non-global region, the webhook URL preview generated is for a global region.
Upon doing some research online, there's an ongoing issue when creating regional webhooks. You can check the status through this issue tracker.
You can track the link above for updates.
Info:
I created a flask app and on my Dockerfile last command CMD gunicorn -b 0.0.0.0:5000 --access-logfile - "app:create_app()"
I build,tag and upload image on ECR
I used this docker image to create an ECS Fargate instance with the following configs (just posting the one needed for the question):
ECSTaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
Cpu: "256"
Memory: "1024"
RequiresCompatibilities:
- FARGATE
ContainerDefinitions:
- Name: contained_above
.
.
.
ECSService:
Type: AWS::ECS::Service
DependsOn: ListenerRule
Properties:
Cluster: !Sub "${EnvName}-ECScluster"
DesiredCount: 1
LaunchType: FARGATE
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 50
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
Subnets:
- Fn::ImportValue: !Sub "${EnvName}-PUBLIC-SUBNET-1"
- Fn::ImportValue: !Sub "${EnvName}-PUBLIC-SUBNET-2"
SecurityGroups:
- Fn::ImportValue: !Sub "${EnvName}-CONTAINER-SECURITY-GROUP"
ServiceName: !Sub "${EnvName}-ECS-SERVICE"
TaskDefinition: !Ref ECSTaskDefinition
LoadBalancers:
- ContainerName: contained_above
ContainerPort: 5000
TargetGroupArn: !Ref TargetGroup
(App is working normally)
Question
Now my question is what number should be the workers on gunicorn command (my last command in dockerfile)?
On gunicorn design it is stated to use Generally we recommend (2 x $num_cores) + 1 as the number of workers to start off with.
So whats the number of cores on a fargate? Does actually make sense to combine gunicorn with Fargate like the above process? Is there 'compatibility' between loadbalancers and gunicorn workers? What is the connection between DesiredCount of ECS Service and the gunicorn -w workers value? Am I missing or miss-understanding something?
Possible solution(?)
One way that I could call it is the following:
CMD gunicorn -b 0.0.0.0:5000 -w $(( 2 * `cat /proc/cpuinfo | grep 'core id' | wc -l` + 1 )) --access-logfile - "app:create_app()"
But I am not sure if that would be a good solution.
Any insights? Thanks
EDIT: I'm using a configuration file for gunicorn to use when starting:
gunicorn.conf.py
import multiprocessing
bind = "0.0.0.0:8080"
workers = multiprocessing.cpu_count() * 2 + 1
worker_class = "uvicorn.workers.UvicornH11Worker"
keepalive = 0
you can tell gunicorn which config file to use with the --config flag.
Sadly I can't find the source anymore, but I've read that 4-12 workers should be enough to handle hundreds if not thousands of simultaneous requests - depending on your application structure, worker class and payload size.
Do take this with a grain of salt tho, since I can't find the source anymore, but it was in an accepted SO answer from a well-reputated person if I remember correctly.
Offical gunicorn docs state somthing in the 2-4 x $(NUM_CORES)range.
Another option would be as gunicorn docs state at another point:
Generally we recommend (2 x $num_cores) + 1 as the number of workers
to start off with. While not overly scientific, the formula is based
on the assumption that for a given core, one worker will be reading or
writing from the socket while the other worker is processing a
request.
Obviously, your particular hardware and application are going to
affect the optimal number of workers. Our recommendation is to start
with the above guess and tune using TTIN and TTOU signals while the
application is under load.
So far I've been running well with holding true to the 4-12 worker recommendation. My company runs several APIs, which connect to other APIs out there, which results in mostly 1-2seconds request time, with the longest taking up to a whole minute (a lot of external API calls here).
Another colleague I talked to mentioned, they are using 1 worker per 5 simultaneous requests they expect - with similar APIs to ours. Works fine for them as well.
I am using Elastic Beanstalk to deploy a worker tier environment using SQS.
In my .ebextensions I have the following file:
option_settings:
aws:elasticbeanstalk:sqsd:
WorkerQueueURL:
Ref: WorkerQueue
HttpPath: "/sqs/"
InactivityTimeout: 1650
VisibilityTimeout: 1680
MaxRetries: 1
Resources:
WorkerQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: "tpc-clients-aws-queue"
VisibilityTimeout: 1680
However, this fails with the following error:
"option_settings" in one of the configuration files failed validation. More details to follow.
Invalid option value: 'Ref=WorkerQueue' (Namespace: 'aws:elasticbeanstalk:sqsd', OptionName: 'WorkerQueueURL'): Value does not satisfy regex: '^$|^http(s)?://.+$' [Valid non empty URL starting with http(s)]
It seems that the AWSCloudFormation Ref function cannot be used in the option_settings. Can someone confirm if this is the case?
I have seen some code snippets here on StackOverflow using intrinsic functions in the option_settings, such as in the mount-config.config of this answer and also on this question. So, are these examples using an invalid syntax? Or there are some intrinsic functions or specific resources that can be used on the option_settings?
And lastly, if I cannot use the Ref function, how can I go about this?
Yes, you can reference in .ebextentions, but the syntax is a bit strange. It is shown in the docs here.
You can try something along these lines (note the various quotations marks):
option_settings:
aws:elasticbeanstalk:sqsd:
WorkerQueueURL: '`{"Ref" : "WorkerQueue"}`'
HttpPath: "/sqs/"
InactivityTimeout: 1650
VisibilityTimeout: 1680
MaxRetries: 1
Resources:
WorkerQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: "tpc-clients-aws-queue"
VisibilityTimeout: 1680
You can also use ImportValue, if you export the WorkerQueue in outputs.
Update
To check the value obtained, you can set it as an env variable, and inspect in EB console:
option_settings:
aws:elasticbeanstalk:application:environment:
SQS_NAME: '`{"Ref" : "WorkerQueue"}`'
After digging further in this issue I made some discoveries I would like to share with future readers.
Ref can be used on option_settings
As #Marcin answer states, the Ref intrinsic function can be used in the option_settings. The syntax is different though:
'`{"Ref" : "ResourceName"}`'
Using Ref on aws:elasticbeanstalk:application:environment (environment variable)
An use case of the above is to store the queue URL in an environment variable, as follows:
option_settings:
aws:elasticbeanstalk:application:environment:
QUEUE_URL: '`{"Ref" : "WorkerQueue"}`'
This will let your .sh script access the URL of the queue:
Note that if you check the Elastic Beanstalk console (Environment > Config > Software), you won't see the actual value:
Using Ref on aws:elasticbeanstalk:sqsd:WorkerQueueURL
If you try to use the following setting:
option_settings:
aws:elasticbeanstalk:sqsd:
WorkerQueueURL: '`{"Ref" : "WorkerQueue"}`'
HttpPath: "/sqs/"
It will fail:
Invalid option value: '`{"Ref" : "WorkerQueue"}`' (Namespace: 'aws:elasticbeanstalk:sqsd', OptionName: 'WorkerQueueURL'): Value does not satisfy regex: '^$|^http(s)?://.+$' [Valid non empty URL starting with http(s)]
It seems that this configuration option don't accept a reference.
Instead of creating a new queue and assign it to the sqs daemon, you can just update the queue that Elastic Beanstalk creates:
option_settings:
# SQS daemon will use default queue created by EB (AWSEBWorkerQueue)
aws:elasticbeanstalk:sqsd:
HttpPath: "/sqs/"
Resources:
# Update the queue created by EB
AWSEBWorkerQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: "tpc-clients-aws-queue"
I have been trying to figure out how to configure the docker version of Concourse (https://github.com/concourse/concourse-docker) to use the AWS Secrets Manager and I added the following environment variables into the docker-compose file but from the logs it doesn't look like it ever reaches out to AWS to fetch the creds. Am I missing something or should this automatically happen when adding these environment variables under environment in the docker-compose file? Here are the docs I have been looking at https://concourse-ci.org/aws-asm-credential-manager.html
version: '3'
services:
concourse-db:
image: postgres
environment:
POSTGRES_DB: concourse
POSTGRES_PASSWORD: concourse_pass
POSTGRES_USER: concourse_user
PGDATA: /database
concourse:
image: concourse/concourse
command: quickstart
privileged: true
depends_on: [concourse-db]
ports: ["9090:8080"]
environment:
CONCOURSE_POSTGRES_HOST: concourse-db
CONCOURSE_POSTGRES_USER: concourse_user
CONCOURSE_POSTGRES_PASSWORD: concourse_pass
CONCOURSE_POSTGRES_DATABASE: concourse
CONCOURSE_EXTERNAL_URL: http://XXX.XXX.XXX.XXX:9090
CONCOURSE_ADD_LOCAL_USER: test: test
CONCOURSE_MAIN_TEAM_LOCAL_USER: test
CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: overlay
CONCOURSE_AWS_SECRETSMANAGER_REGION: us-east-1
CONCOURSE_AWS_SECRETSMANAGER_ACCESS_KEY: <XXXX>
CONCOURSE_AWS_SECRETSMANAGER_SECRET_KEY: <XXXX>
CONCOURSE_AWS_SECRETSMANAGER_TEAM_SECRET_TEMPLATE: /concourse/{{.Secret}}
CONCOURSE_AWS_SECRETSMANAGER_PIPELINE_SECRET_TEMPLATE: /concourse/{{.Secret}}
pipeline.yml example:
jobs:
- name: build-ui
plan:
- get: web-ui
trigger: true
- get: resource-ui
- task: build-task
file: web-ui/ci/build/task.yml
- put: resource-ui
params:
repository: updated-ui
force: true
- task: e2e-task
file: web-ui/ci/e2e/task.yml
params:
UI_USERNAME: ((ui-username))
UI_PASSWORD: ((ui-password))
resources:
- name: cf
type: cf-cli-resource
source:
api: https://api.run.pivotal.io
username: ((cf-username))
password: ((cf-password))
org: Blah
- name: web-ui
type: git
source:
uri: git#github.com:blah/blah.git
branch: master
private_key: ((git-private-key))
When storing parameters for concourse pipelines in AWS Secrets Manager, it must follow this syntax,
/concourse/TEAM_NAME/PIPELINE_NAME/PARAMETER_NAME`
If you have common parameters that are used across the team in multiple pipelines, use this syntax to avoid creating redundant parameters in secrets manager
/concourse/TEAM_NAME/PARAMETER_NAME
The highest level that is supported is concourse team level.
Global parameters are not possible. Thus these variables in your compose environment will not be supported.
CONCOURSE_AWS_SECRETSMANAGER_TEAM_SECRET_TEMPLATE: /concourse/{{.Secret}}
CONCOURSE_AWS_SECRETSMANAGER_PIPELINE_SECRET_TEMPLATE: /concourse/{{.Secret}}
Unless you want to change the prefix /concourse, these parameters shall be left to their defaults.
And, when retrieving these parameters in the pipeline, no changes required in the template. Just pass the PARAMETER_NAME, concourse will handle the lookup in secrets manager as per the team and pipeline name.
...
params:
UI_USERNAME: ((ui-username))
UI_PASSWORD: ((ui-password))
...