I have an application deployed to Elasticbeanstalk and run as worker, I wanted to add a periodic task ti run each hour, so I create a cron.yaml with this conf:
version: 1
cron:
- name: "task1"
url: "/task"
schedule: "00 * * * *"
But during the deploy I always got this error:
[Instance: i-a072e41d] Command failed on instance. Return code: 1 Output: missing required parameter params[:table_name] - (ArgumentError). Hook /opt/elasticbeanstalk/addons/sqsd/hooks/start/02-start-sqsd.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
I added the right permission to EBT role, and I verified the cron.yaml maybe it formatted for Windows (CR/LF), but always got the same error.
missing required parameter params[:table_name] looks like DynamoDB table name is missing, where I can define it ? ,
Any idea how I can fix that.
Thanks !
Well I didn't figure out a solution with this issue so I moved to another approach which use CloudWatch Event to create a Rule type:schedule and select a target as SQS Queue (the one configured with the worker).
Works perfectly!
I encountered this same error when I was dynamically generating a cron.yaml file in a container command instead of already having it in my application root.
The DynamoDB table for the cron is created in the PreInitStage which occurs before any of your custom code executes so if there is no cron.yaml file than no DynamoDB table is created. When the file later appears and the cron jobs are being scheduled it fails because the table was never created.
I solved this problem by having a skeleton cron.yaml in my application root. It must have a valid cron job (I just hit my health check URL once a month) but it doesn't get scheduled since the job registration does happen after your custom commands which can reset the file with only the jobs you need.
This might not be your exact problem but hopefully it helps you find yours as it appears the error happens when the DynamoDB table does not get created.
I looks like your yaml formatting is off. That might be the issue here.
version: 1
cron:
- name: "task1"
url: "/task"
schedule: "00 * * * *"
Formatting is critical in Yaml. Give this a try at the very least.
Related
I am using serverless framework configuration and aws for the serverless.yml file and also the yaml plugin on VSCode and it says that is incorrect type and is expecting a string.
events:
- cloudwatchEvent:
name: ${self:custom.transcribeJobName.${self:provider.stage}}
event:
source:
- 'aws.transcribe'
detail-type:
- 'Transcribe Job State Change'
detail:
TranscriptionJobStatus:
- COMPLETED
- FAILED
The documentation of Serverless Framework no AWS events doesn't list AWS Transcribe at all. I also tried googling but couldn't find any native event for that.
https://www.serverless.com/framework/docs/providers/aws/events
So can you share which documentation or blog you're following to implement your solution?
Got this in my feed, the error is coming from serverless json-schema which isn't always up to date since it isn't maintained by the same team.
I've created the PRs for both serverless json schema and for serverless types as well to get this issue sorted out at the source.
It's obviously not thorough and compelete, but the best I cloud do.
https://github.com/lalcebo/json-schema/pull/13
https://github.com/DefinitelyTyped/DefinitelyTyped/pull/59966
Edit:- schema updated, vscode should stop complaining on restart.
Error message:
Error: There was an error deploying functions
firebease-debug.log holds this:
[debug] [2021-11-16T12:12:16.165Z] Error: Failed to upsert schedule function lab2 in region europe-west3
function code:
exports.lab2 =
functions
.region('europe-west3')
.pubsub.schedule('*/10 * * * *')
.onRun(lab);
What can I do? Google support leads to stackoverflow, so I post it here. Are there better ways to deal with the Google Cloud problems?
When you are using scheduled functions in Firebase Functions, an App Engine instance is created that is needed for Cloud Scheduler to work. You can read about it here. During its setup you're prompted to select your project's default Google Cloud Platform (GCP) resource location (if it wasn't already selected when setting up another service).
You are getting that error because there is a difference between the default GCP resource location you specified and the region of your scheduled Cloud Function. If you click on the cogwheel next to project-overview in Firebase you can see where your resources are located. Setting the default GCP resource location same as the scheduler function region, solves the issue.
The reason our schedule function deployment failed was because of the cron job formula. For some reason firebase couldn't understand 0 0 3 1/1 * ? *, but could understand every 5 minutes.
It's a shame that firebase doesn't provide a better error message. Error: Failed to upsert schedule function xxx in region xxx is way too generic.
I got the same problem of deploying my schedule functions, and I solved it with a few steps as below:
Step 1: Rename the schedule function (for example: newScheduleABC), then only deploy it:
$ Firebase deploy --only functions:newScheduleABC
NOTE: as #Nicholas mentioned, the Unix Cron Format The GCP accepts has only 5 fields: schedule(* * * * *)
Step 2: Delete old (schedule) functions by going to your Firebase Console : https://console.firebase.google.com/.../functions , you will see all of your functions there. click on the vertical ellipsis at the end of a function and click delete.
That's it. There should be no problem from now.
You can read more from Manage functions
I have created a glue crawler to run every 6 hours , I am using "Crawl new folders only" option. Every time crawler runs it fails with "Internal Service Exception" error.
What I tried so far ?
Created another crawler with option "Crawl all the folders" set to run every 6 hours. It works perfectly without any issue.
Created another crawler with option "Crawl new folders only" but with "Run on demand" option. It works perfectly without any issue.
All above 3 scenarios point to same S3 bucket with same IAM policy rule. I also tried reducing 6 hours run time to 15 mins / 1 hour but no luck.
What I am missing ? Any help is appreciated.
If you are setting RecrawlPolicy as CRAWL_NEW_FOLDERS_ONLY then please make sure that the UpdateBehavior is LOG only otherwise you will get the error like The SchemaChangePolicy for "Crawl new folders only" Amazon S3 target can have only LOG DeleteBehavior value and LOG UpdateBehavior value.
I'm trying to migrate and synchronize a PostgreSQL database using AWS DMS and I'm getting the following error.
Last Error Task error notification received from subtask 0, thread 0
[reptask/replicationtask.c:2673] [1020101] When working with Configured Slotname, user must
specify LSN; Error executing source loop; Stream component failed at subtask 0, component
st_0_D27UO7SI6SIKOSZ4V6RH4PPTZQ ; Stream component 'st_0_D27UO7SI6SIKOSZ4V6RH4PPTZQ'
terminated [reptask/replicationtask.c:2680] [1020101] Stop Reason FATAL_ERROR Error Level FATAL
I already created a replication slot and configured its name in the source endpoint.
DMS Engine version: 3.1.4
Does anyone knows anything that could help me?
Luan -
I experienced the same issue - I was trying to replicate data from Postgres to an S3 bucket.I would check two things - your version of Postgres and the DMS version being used.
I downgraded my RDS postgres version to 9.6 and my DMS version to 2.4.5 to get replication working.
You can find more details here -
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.PostgreSQL.html
I wanted to try the newer versions of DMS (3.1.4 and 3.3.0[beta]) as it has parquet support but I have gotten the same errors you mentioned above.
Hope this helps.
It appears AWS expects you to use the pglogical extension rather than test_decoding. You have to:
add pglogical to shared_preload_libraries in parameter options
reboot
CREATE EXTENSION pglogical;
On dms 3.4.2 and postgres 12.3 without the slotName= setting DMS created the slot for itself. Also make sure you exclude the pglogical schema from the migration task as it has unsupported data types.
P.S. When DMS hits resource limits it silently fails. After resolving the LSN errors, I continued to get failures of the type Last Error Task 'psql2es' was suspended due to 6 successive unexpected failures Stop Reason FATAL_ERROR Error Level FATAL without any errors in the logs. I resolved this issue using the Advanced task settings > Full load tuning settings and tuning the parameters downward.
For some reason I am getting the following error:
Waiter TasksStopped failed: taskId length should be one of [32,36]
I really don't know what taskId is supposed to mean and aws documentation isn't helping. Does anyone know what is going wrong in this pipeline script?
- step:
name: Run DB migrations
script:
- >
export BackendTaskArn=$(aws cloudformation list-stack-resources \
--stack-name=${DEXB_PRODUCTION_STACK} \
--output=text \
--query="StackResourceSummaries[?LogicalResourceId=='BackendECSTask'].PhysicalResourceId")
- >
SequelizeTask=$(aws ecs run-task --cluster=${DEXB_PRODUCTION_ECS_CLUSTER} --task-definition=${BackendTaskArn} \
--overrides='{"containerOverrides":[{"name":"NodeBackend","command":["./node_modules/.bin/sequelize","db:migrate"]}]}' \
--launch-type=EC2 --output=text --query='tasks[0].taskArn')
- aws ecs wait tasks-stopped --cluster=${DEXB_PRODUCTION_ECS_CLUSTER} --tasks ${SequelizeTask}
AWS introduced a new ARN format for tasks, container instances, and services. This format now contains the cluster name, which might break scripts and applications that were counting on the ARN only containing the task resource ID.
# Previous format (taskId contains hyphens)
arn:aws:ecs:$region:$accountID:task/$taskId
# New format (taskI does not contain hyphens)
arn:aws:ecs:$region:$accountId:task/$clusterName/$taskId
Until March 31, 2021, it will be possible to opt-out of this change per-region, using https://console.aws.amazon.com/ecs/home?#/settings. In order to change the behavior for the whole account, you will need to use the Root IAM user.
It turns out I had a duplicate task running in the background. I went to the ECS clusters page and stopped the duplicate task. However this may be dangerous to do if you have used cloudformation to set up your tasks and services. Proceed cautiously if you're in the same boat.
We were bit with this cryptic error message, and what it actually means is that the task_id you're sending to cloudformation script is invalid. Task ids must have a length of 32 or 36 chars.
In our case, an undocumented change in the way AWS sent back taskArn key value was causing us to grab the incorrect value, and sending an unrelated string as the task_id. AWS detected this and blew up. So double check the task_id string and you should be good.