Go pipelines can sometimes take an age to start after a Git commit and push. Even after manually hitting the start button it can take a long time. Is there a setting or way to get it to pick up changes quicker and start quicker? I don't mind if it uses more resources on the server.
Something was wrong with out Go Server. Rebuilding a new server solved the problem. It can still take upt to 30 seconds to start a pipeline (it was taking ten minutes).
Related
I have a big glue workflow (about 100 jobs / crawlers), and it was executing properly until last week. Since then, my first conditional trigger (ALL), is executing 20 time the same job.
I've configured the job it self, to just allow 1 parallel execution, but every time the workflow executes, it tries to launch 20 times (the same job).
Also configured the workflow, to allow a max concurrency of 1, but that doesn't fix the problem.
Since i started working with glue workflow, i've noticed that the tool it self is buggy, old and maybe deprecated ?
Any tips on how to fix this problem ?
I too have faced similar problems. Even one is fixed, some other issue would occur lately. So my suggestion is to try using step functions.
Really basic question, but I can't seem to find it in the documentation. When continuous scanning is turned on, at what frequency and at which time the registry will be scanned? I've turned it on a couple of hours ago but it hasn't scanned yet.
Continuous scanning will be executed when a new vulnerability is received by AWS. I couldn't pinpoint when the initial scan happened. It took approximately a half day.
I had a same issue and came across with this question. I will post the docs quote.
https://docs.aws.amazon.com/AmazonECR/latest/userguide//image-scanning.html?icmpid=docs_ecr_hp-registry-private
As new vulnerabilities appear, the scan results are updated and Amazon Inspector emits an event to EventBridge to notify you.
did you turn one the "Scan on push" option, go the Edit option end then enable that option, it will automatically scan your repo after each push.
if you go for manual scan it's generally trigger immediately.
please give a try
I am deploying a pipeline to Google Cloud DataFlow using Apache Beam. When I want to deploy a change to the pipeline, I drain the running pipeline and redeploy it. I would like to make this faster. It appears from the logs that on each deploy DataFlow builds up new worker nodes from scratch: I see Linux boot messages going by.
Is it possible to drain the pipeline without tearing down the worker nodes so the next deployment can reuse them?
rewriting Inigo's answer here:
Answering the original question, no, there's no way to do that. Updating should be the way to go. I was not aware it was marked as experimental (probably we should change that), but the update approach has not changed in the last 3 i have been using DF. About the special cases of update not working, supposing your feature existed, the workers would still need the new code, so no really much to save, and update should work in most of the other cases.
I have an AWS Glue job, with max concurrent runs set to 1. The job is currently not running. But when I try to run it, I keep getting the error: "Max concurrent runs exceeded".
Deleting and re-creating the job does not help. Also, other jobs in the same account run fine, so it cannot be a problem with account wide service quotas.
Why am I getting this error?
I raised this issue with AWS support, and they confirmed that it is a known bug:
I would like to inform you that this is a known bug, where an internal distributed counter that keeps track of job concurrency goes into a stale state due to an edge case, causing this error. Our internal Service team has to manually reset the counter to fix this issue. Service team has already added the bug fix in their product roadmap and will be working on it. Unfortunately I may not be able to comment on the ETA on the deployment, as we don’t have any visibility on product teams road map and fix release timeline.
The suggested workarounds are:
Increase the max concurrency to 2 or higher
Re-create the job with a different name
Glue container is start and its taking some time same when your job end container shutdown taking some time in between if you try to execute new Jon and default concurrency is 1 so you will get this error.
How to resolve:
Go to your Glue Job --> Under Job detail tab you can find "Maximum concurrency" default value is 1 change it to 3 or more as per your need.
I tried changing "Maximum concurrency" to 2 and then run it !
It worked but again running it cause the same issue, but I looked into my s3 ,it has dumped the data ,so it run for once!
I'm still looking for a stable solution but this may work!
I have a job that I want to run every time a commit is made to a repository. I want to avoid pulling this code down, I only want the notification build trigger. So, is there either a way to not pull down certain repositories in your SCM upon a build or a way to poll things that aren't in the SCM for a build?
you could use a post commit hook to trigger your hudson job.
Since you want to avoid changing SVN, you have to write a job that gets executed every so often (may be every 5 Minutes). This jobs runs a svn command using windows bach or shell script task to get the current revision for the branch in question. You can set the status of the job to unstable if there is a change. Don't use failure because you can't distinguish than between a real failure and a repository change. I think there is a plugin that sets the job status depending on the contents of you output.
You can then use the email extension plugin to send an email every time the revision changes. You can get the revision number from the last (or better the last successful or unstable) job. You can archive a file containing the revision number on the jobs or you can set the description for the job to the revision using the description setter plugin. Have a look at Hudsons remote API for ideas on how to get the information from the previous job.
Since you run your job very often during the day. don't forget to delete old job runs. But I would keep at least two days worth of history, just in case your svn is down for 24 hours.