I have airflow (v1.10.4) running in Kubernetes cluster using Celery Executor. We are using RDS Postgres as an external metadata db.
Problem Statement: All dags are getting scheduled and run on time but the problem happens when we clear any log (old run) to re-run the pipeline Airflow puts this Dag in running state but none of the tasks gets executed. We also saw when we re-mark those cleared logs as success Airflow schedules the next run but none of the tasks get executed (none of the tasks goes into queue or running state).
The dag in question has depends_on_past=True and catchup=True.
Can anyone know what's going wrong here?
Related
I have created a basic EMR cluster in AWS, and I'm trying to use the Jupyter Notebooks provided through the AWS Console. Launching the notebooks seems to work fine, and I'm also able to run basic python code in notebooks started with the pyspark kernel. Two variables are set up in the notebook: spark is a SparkSession instance, and sc is a SparkContext instance. Displaying sc yields <SparkContext master=yarn appName=livy-session-0> (the output can of course vary slightly depending on the session).
The problem arises once I perform operations that actually hit the spark machinery. For example:
sc.parallelize(list(range(10))).map(lambda x: x**2).collect()
I am no spark expert, but I believe this code should distribute the integers from 0 to 9 across the cluster, square them, and return the results in a list. Instead, I get a lengthy stack trace, mostly from the JVM, but also some python components. I believe the central part of the stack trace is the following:
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 4 times, most recent failure: Lost task 0.3 in stage 4.0 (TID 116, ip-XXXXXXXXXXXXX.eu-west-1.compute.internal, executor 17): java.lang.RuntimeException: Failed to run command: /usr/bin/virtualenv -p python3 --system-site-packages virtualenv_application_1586243436143_0002_0
The full stack trace is here.
A bit of digging in the AWS portal led me to log output from the nodes. stdout from one of the nodes includes the following:
The path python3 (from --python=python3) does not exist
I tried running the /usr/bin/virtualenv command on the master node manually (after logging in through), and that worked fine, but the error is of course still present after I did that.
While this error occurs most of the time, I was able to get this working in one session, where I could run several operations against the spark cluster as I was expecting.
Technical information on the cluster setup:
emr-6.0.0
Applications installed are "Ganglia 3.7.2, Spark 2.4.4, Zeppelin 0.9.0, Livy 0.6.0, JupyterHub 1.0.0, Hive 3.1.2". Hadoop is also included.
3 nodes (one of them as master), all r5a.2xlarge.
Any ideas what I'm doing wrong? Note that I am completely new to EMR and Spark.
Edit: Added the stdout log and information about running the virtualenv command manually on the master node through ssh.
I have switched to using emr-5.29.0, which seems to resolve the problem. Perhaps this is an issue with emr-6.0.0? In any case, I have a functional workaround.
The issue for me was that the virtualenv was being made on the executors with a python path that didn't exist. Pointing the executors to the right one did the job for me:
"spark.pyspark.python": "/usr/bin/python3.7"
Here is how I reconfiged the spark app at the beginning of the notebook:
{"conf":{"spark.pyspark.python": "/usr/bin/python3.7",
"spark.pyspark.virtualenv.enabled": "true",
"spark.pyspark.virtualenv.type": "native",
"spark.pyspark.virtualenv.bin.path":"/usr/bin/virtualenv"}
}
(Docker container on AWS-ECS exits before all the logs are printed to CloudWatch Logs)
Why are some streams of a CloudWatch Logs Group incomplete (i.e., the Fargate Docker Container exits successfully but the logs stop being updated abruptly)? Seeing this intermittently, in almost all log groups, however, not on every log stream/task run. I'm running on version 1.3.0
Description:
A Dockerfile runs node.js or Python scripts using the CMD command.
These are not servers/long-running processes, and my use case requires the containers to exit when the task completes.
Sample Dockerfile:
FROM node:6
WORKDIR /path/to/app/
COPY package*.json ./
RUN npm install
COPY . .
CMD [ "node", "run-this-script.js" ]
All the logs are printed correctly to my terminal's stdout/stderr when this command is run on the terminal locally with docker run.
To run these as ECS Tasks on Fargate, the log driver for is set as awslogs from a CloudFormation Template.
...
LogConfiguration:
LogDriver: 'awslogs'
Options:
awslogs-group: !Sub '/ecs/ecs-tasks-${TaskName}'
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: ecs
...
Seeing that sometimes the cloduwatch logs output is incomplete, I have run tests and checked every limit from CW Logs Limits and am certain the problem is not there.
I initially thought this is an issue with node js exiting asynchronously before console.log() is flushed, or that the process is exiting too soon, but the same problem occurs when i use a different language as well - which makes me believe this is not an issue with the code, but rather with cloudwatch specifically.
Inducing delays in the code by adding a sleep timer has not worked for me.
It's possible that since the docker container exits immediately after the task is completed, the logs don't get enough time to be written over to CWLogs, but there must be a way to ensure that this doesn't happen?
sample logs:
incomplete stream:
{ "message": "configs to run", "data": {"dailyConfigs":"filename.json"]}}
running for filename
completed log stream:
{ "message": "configs to run", "data": {"dailyConfigs":"filename.json"]}}
running for filename
stdout: entered query_script
... <more log lines>
stderr:
real 0m23.394s
user 0m0.008s
sys 0m0.004s
(node:1) DeprecationWarning: PG.end is deprecated - please see the upgrade guide at https://node-postgres.com/guides/upgrading
UPDATE: This now appears to be fixed, so there is no need to implement the workaround described below
I've seen the same behaviour when using ECS Fargate containers to run Python scripts - and had the same resulting frustration!
I think it's due to CloudWatch Logs Agent publishing log events in batches:
How are log events batched?
A batch becomes full and is published when any of the following conditions are met:
The buffer_duration amount of time has passed since the first log event was added.
Less than batch_size of log events have been accumulated but adding the new log event exceeds the batch_size.
The number of log events has reached batch_count.
Log events from the batch don't span more than 24 hours, but adding the new log event exceeds the 24 hours constraint.
(Reference: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html)
So a possible explanation is that log events are buffered by the agent but not yet published when the ECS task is stopped. (And if so, that seems like an ECS issue - any AWS ECS engineers willing to give their perspective on this...?)
There doesn't seem to be a direct way to ensure the logs are published, but it does suggest one could wait at least buffer_duration seconds (by default, 5 seconds), and any prior logs should be published.
With a bit of testing that I'll describe below, here's a workaround I landed on. A shell script run_then_wait.sh wraps the command to trigger the Python script, to add a sleep after the script completes.
Dockerfile
FROM python:3.7-alpine
ADD run_then_wait.sh .
ADD main.py .
# The original command
# ENTRYPOINT ["python", "main.py"]
# To run the original command and then wait
ENTRYPOINT ["sh", "run_then_wait.sh", "python", "main.py"]
run_then_wait.sh
#!/bin/sh
set -e
# Wait 10 seconds on exit: twice the `buffer_duration` default of 5 seconds
trap 'echo "Waiting for logs to flush to CloudWatch Logs..."; sleep 10' EXIT
# Run the given command
"$#"
main.py
import logging
import time
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()
if __name__ == "__main__":
# After testing some random values, had most luck to induce the
# issue by sleeping 9 seconds here; would occur ~30% of the time
time.sleep(9)
logger.info("Hello world")
Hopefully the approach can be adapted to your situation. You could also implement the sleep inside your script, but it can be trickier to ensure it happens regardless of how it terminates.
It's hard to prove that the proposed explanation is accurate, so I used the above code to test whether the workaround was effective. The test was the original command vs. with run_then_wait.sh, 30 runs each. The results were that the issue was observed 30% of the time, vs 0% of the time, respectively. Hope this is similarly effective for you!
Just contacted AWS support about this issue and here is their response:
...
Based on that case, I can see that this occurs for containers in a
Fargate Task that exit quickly after outputting to stdout/stderr. It
seems to be related to how the awslogs driver works, and how Docker in
Fargate communicates to the CW endpoint.
Looking at our internal tickets for the same, I can see that our
service team are still working to get a permanent resolution for this
reported bug. Unfortunately, there is no ETA shared for when the fix
will be deployed. However, I've taken this opportunity to add this
case to the internal ticket to inform the team of the similar and try
to expedite the process
In the meantime, this can be avoided by extending the lifetime of the
exiting container by adding a delay (~>10 seconds) between the logging
output of the application and the exit of the process (exit of the
container).
...
Update:
Contacted AWS around August 1st, 2019, they say this issue has been fixed.
I observed this as well. It must be an ECS bug?
My workaround (Python 3.7):
import atexit
from time import sleep
atexit.register(finalizer)
def finalizer():
logger.info("All tasks have finished. Exiting.")
# Workaround:
# Fargate will exit and final batch of CloudWatch logs will be lost
sleep(10)
I had the same problem with flushing logs to CloudWatch.
Following asavoy's answer I switched from exec form to shell form of the ENTRYPOINT and added a 10 sec sleep at the end.
Before:
ENTRYPOINT ["java","-jar","/app.jar"]
After:
ENTRYPOINT java -jar /app.jar; sleep 10
The problem:
I have a managed Cloud composer environment, under a 1.9.7-gke.6 Kubernetes cluster master.
I tried to upgrade it (as well as the default-pool nodes) to 1.10.7-gke.1, since an upgrade was available.
Since then, Airflow has been acting randomly. Tasks that were working properly are failing for no given reason. This makes Airflow unusable, since the scheduling becomes unreliable.
Here is an example of a task that runs every 15 minutes and for which the behavior is very visible right after the upgrade:
airflow_tree_view
On hover on a failing task, it only shows an Operator: null message (null_operator). Also, there is no log at all for that task.
I have been able to reproduce the situation with another Composer environment in order to ensure that the upgrade is the cause of the dysfunction.
What I have tried so far :
I assumed the upgrade might have screwed up either the scheduler or Celery (Cloud composer defaults to CeleryExecutor).
I tried restarting the scheduler with the following command:
kubectl get deployment airflow-scheduler -o yaml | kubectl replace --force -f -
I also tried to restart Celery from inside the workers, with
kubectl exec -it airflow-worker-799dc94759-7vck4 -- sudo celery multi restart 1
Celery restarts, but it doesn't fix the issue.
So I tried to restart the airflow completely the same way I did with airflow-scheduler.
None of these fixed the issue.
Side note, I can't access Flower to monitor Celery when following this tutorial (Google Cloud - Connecting to Flower). Connecting to localhost:5555 stay in 'waiting' state forever. I don't know if it is related.
Let me know if I'm missing something!
1.10.7-gke.2 is available now [1]. Can you further upgrade to 1.10.7-gke.2 to see if the issue persists?
[1] https://cloud.google.com/kubernetes-engine/release-notes
I am running a batch job on dataflow, querying from BigQuery. When I use the DirectRunner, everything works, and the results are written to a new BigQuery table. Things seem to break when I change to DataflowRunner.
The logs show that 30 worker instances are spun up successfully. The graph diagram in the web UI shows the job has started. The first 3 steps show "Running", the rest show "not started". None of the steps show any records transformed (i.e. outputcollections all show '-'). The logs show many messages that look like this, which may be the issue:
skipping: failed to "StartContainer" for "python" with CrashLoopBackOff: "Back-off 10s restarting failed container=python pod=......
I took a step back and just ran the minimal wordcount example, and that completed successfully. So all the necessary APIs seem to be enabled for Dataflow runner. I'm just trying to get a sense of what is causing my Dataflow job to hang.
I am executing the job like this:
python2.7 script.py --runner DataflowRunner --project projectname --requirements_file requirements.txt --staging_location gs://my-store/staging --temp_location gs://my-store/temp
I'm not sure if my solution was the cause of the error pasted above, but fixing dependencies problems (which were not showing up as errors in the log at all!) did solve the hanging dataflow processes.
So if you have a hanging process, make sure your workers have all their necessary dependencies. You can provide them through the --requirements_file argument, or through a custom setup.py script.
Thanks to the help I received in this post, the pipeline appears to be operating, albeit VERY SLOWLY.
In my Django project, I use Celery and Rabbitmq to run tasks in background.
I am using celery beat scheduler to run periodic tasks.
How can i check if celery beat is up and running, programmatically?
Make a task to HTTP requests to a Ping URL at regular intervals. When the URL is not pinged on time, the URL monitor will send you an alert.
import requests
from yourapp.celery_config import app
#app.task
def ping():
print '[healthcheck] pinging alive status...'
# healthchecks.io works for me:
requests.post("https://hchk.io/6466681c-7708-4423-adf0-XXXXXXXXX")
This celery periodic task is scheduled to run every minute, if it doesn't hit the ping, your beat service is down*, the monitor will kick in your mail (or webhook so you can zapier it to get mobile push notifications as well).
celery -A yourapp.celery_config beat -S djcelery.schedulers.DatabaseScheduler
*or overwhelmed, you should track tasks saturation, this is a nightmare with Celery and should be detected and addressed properly, happens frequently when the workers are busy with blocking tasks that would need optimization
Are you use upstart or supervison or something else to run celery workers + celery beat as a background tasks? In production you should use one of them to run celery workers + celery beat in background.
Simplest way to check celery beat is running: ps aux | grep -i '[c]elerybeat'. If you get text string with pid it's running. Also you can make output of this command more pretty: ps aux | grep -i '[c]elerybeat' | awk '{print $2}'. If you get number - it's working, if you get nothing - it's not working.
Also you can check celery workers status: celery -A projectname status.
If you intrested in advanced celery monitoring you can read official documentation monitoring guide.
If you have daemonized celery following the tutorial of the celery doc, checking if it's running or not can be done through
sudo /etc/init.d/celeryd status
sudo /etc/init.d/celerybeat status
You can use the return of such commands in a python module.
You can probably look up supervisor.
It provides a celerybeat conf which logs everything related to beat in /var/log/celery/beat.log.
Another way of going about this is to use Flower. You can set it up for your server (make sure its password protected), it somewhat becomes easier to notice in the GUI the tasks which are being queued and what time they are queued thus verifying if your beat is running fine.
I have recently used a solution similar to what #panchicore suggested, for the same problem.
Problem in my workplace was an important system working with celery beat, and once in a while, either due to RabbitMQ outage, or some connectivity issue between our servers and RabbitMQ server, due to which celery beat just stopped triggering crons anymore, unless restarted.
As we didn't have any tool handy, to monitor keep alive calls sent over HTTP, we have used statsd for the same purpose. There's a counter incremented on statsd server every minute(done by a celery task), and then we setup email & slack channel alerts on the grafana metrics. (no updates for 10 minutes == outage)
I understand it's not purely a programatic approach, but any production level monitoring/alerting isn't complete without a separate monitoring entity.
The programming part is as simple as it can be. A tiny celery task running every minute.
#periodic_task(run_every=timedelta(minutes=1))
def update_keep_alive(self):
logger.info("running keep alive task")
statsd.incr(statsd_tags.CELERY_BEAT_ALIVE)
A problem that I have faced with this approach, is due to STATSD packet losses over UDP. So use TCP connection to STATSD for this purpose, if possible.
You can check scheduler running or not by the following command
python manage.py celery worker --beat
While working on a project recently, I used this:
HEALTHCHECK CMD ["stat celerybeat.pid || exit 1"]
Essentially, the beat process writes a pid file under some location (usually the home location), all you have to do is to get some stats to check if the file is there.
Note: This worked while launching a standalone celery beta process in a Docker container
The goal of liveness for celery beat/scheduler is to check if the celery beat/scheduler is able to send the job to the message broker so that it can be picked up by the respective consumer. [Is it still working or in a hung state]. The celery worker and celery scheduler/beat may or may not be running in the same pod or instance.
To handle such scenarios, we can create a method update_scheduler_liveness with decorator #after_task_publish.connect which will be called every time when the scheduler successfully publishes the message/task to the message broker.
The method update_scheduler_liveness will update the current timestamp to a file every time when the task is published successfully.
In Liveness probe, we need to check the last updated timestamp of the file either using:
stat --printf="%Y" celery_beat_schedule_liveness.stat command
or we can explicitly try to read the file (read mode) and extract the timestamp and compare if the the timestamp is recent or not based on the liveness probe criteria.
In this approach, the more minute liveness criteria you need, the more frequent a job must be triggered from the celery beat. So, for those cases, where the frequency between jobs is pretty huge, a custom/dedicated liveness heartbeat job can be scheduled every 2-5 mins and the consumer can just process it. #after_task_publish.connect decorator provides multiple arguments that can be also used for filtering of liveness specific job that were triggered
If we don't want to go for file based approach, then we can rely on Redis like data-source with instance specific redis key as well which needs to be implemented on the same lines.