Google App Engine, tasks in Task Queue are not executed automatically - python-2.7

My tasks are added to Task Queue, but nothing executed automatically. I need to click the button "Run now" to run tasks, tasks are executed without problem. Have I missed some configurations ?
I use default queue configuration, standard App Engine with python 27.
from google.appengine.api import taskqueue
taskqueue.add(
url='/inserturl',
params={'name': 'tablename'})

This documentation is for the API you are now mentioning. The idea would be the same: you need to specify the parameter for when you want the task to be executed. In this case, you have different options, such as countdown or eta. Here is the specific documentation for the method you are using to add a task to the queue (taskqueue.add)
ORIGINAL ANSWER
If you follow this tutorial to create queues and tasks, you will see it is based on the following github repo. If you go to the file where the tasks are created (create_app_engine_queue_task.py). There is where you should specify the time when the task must be executed. In this tutorial, to finally create the task, they use the following command:
python create_app_engine_queue_task.py --project=$PROJECT_ID --location=$LOCATION_ID --queue=$QUEUE_ID --payload=hello
However, it is missing the time when you want to execute it, it should look like this
python create_app_engine_queue_task.py --project=$PROJECT_ID --location=$LOCATION_ID --queue=$QUEUE_ID --payload=hello --in_seconds=["countdown" for when the task will be executed, in seconds]
Basically, the key is in this part of the code in create_app_engine_queue_task.py:
if in_seconds is not None:
# Convert "seconds from now" into an rfc3339 datetime string.
d = datetime.datetime.utcnow() + datetime.timedelta(seconds=in_seconds)
# Create Timestamp protobuf.
timestamp = timestamp_pb2.Timestamp()
timestamp.FromDatetime(d)
# Add the timestamp to the tasks.
task['schedule_time'] = timestamp
If you create the task now and you go to your console, you will see you task will execute and disappear from the queue in the amount of seconds you specified.

Related

Return Glue Job Status in Airflow

I am using an older version of Airflow (1.10). We are using Python operators to trigger Glue jobs because Glue operators aren't available in this version. We have multiple jobs that need to run in a particular order. When we run the DAG, our first job triggers and then it passes as succeeded since the job was successfully started.
We are trying to use boto3 to check the status of the job, but we need it to do so continually. Any thoughts on how to check the status continually then only move on to the next Python operator based upon success?
Well, you could try to replicate the .job_completion method from the GlueJobSensor. So basically:
def my_glue_job_that_waits():
# botocore call that starts the job
while True:
try:
# botocore call to retrieve job state
if job_state == "SUCCEEDED":
# log statement
return #what you want the operator to return
else:
# log statement
time.sleep(POKE_INTERVAL)
except:
# what you want to happen if the call above fails
But I highly encourage you to upgrade to Airflow 2 if you can. Long term it will save you a lot of time both being able to use new features and not running into conflicts with provider packages.

Tasks are not performed using the admin interface

I use django-celery-beat to create the task. Everything is registered, connected, the usual Django tascas using beat_schedule work. I add a new task, I don't register it in beat_schedule:
#app.task
def say_hi():
print("hello test")
I go into admin, add Periodic tasks, this task in Task (registered) is visible, I select it, I also select the interval every minute and save, Task (registered) is zeroed, and its value appears in Task (custom).
The task itself does not start its execution, in the console print is not displayed, but Last Run Datetime is updated in the admin. What could go wrong?

Where to find the number of active concurrent invocations in Google Cloud Functions

I am looking for a way to see how many concurrent invocations there are active at any point in time, e.g. in a minute range. I am looking for this as I received the error:
Forbidden: 403 Exceeded rate limits: too many concurrent queries for
this project_and_region. For more information, see
https://cloud.google.com/bigquery/
The quotas are listed here: https://cloud.google.com/functions/quotas
I am fine with having quotas, but I would like to see this number in a chart. Where can I find this?
Currently there is no way of seeing that information directly. There is a workaround though. You can do as follows:
Go to Google Cloud Console > Stackdriver Logging
At the text box that says "Filter by label or text search", click on the small arrow at the end of the text box.
Choose "Convert to advanced filter"
Type that query inside:
resource.type="cloud_function"
resource.labels.function_name="[GOOGLE_CLOUD_FUNCTION_NAME]"
"Function execution started"
At "Last hour" drop down menu, choose "Custom"
Fix the start and end time
This will list all the times that the Cloud Function was executed in the time range. If it was executed multiple times, instead of counting one by one you can use the following Python script:
Open Google Cloud Shell
Install Google Cloud Logging Library $ pip install google-cloud-logging
Create a main.py file using my GitHub code example. (I have tested it and it is working as expected)
Change the date_a_str and set it as start date.
Change the date_b_str and set it as end date.
In function_name = "[CLOUD_FUNCTION_NAME]" change [CLOUD_FUNCTION_NAME] to the name of your Cloud Function.
Execute the Python code $ python main.py
You should see a response as follows:
Found entries: [XX]
Waiting up to 5 seconds.
Sent all pending logs.

Sitecore Scheduled Job: Unable to run

I am very new with Sitecore, I am trying to create one task, but after creating task I configured command and task at content editor. Still I don't see run now option for my task at content editor. Need help.I want to know where the logs of scheduled jobs are written?
There are 2 places where you can define custom task.
In database
In config file
If you decide to go with 1st option
a task item must be created under /sitecore/system/tasks/schedules
item in the “core” database (default behavior).
no matter what schedule you set on that item, it may never be executed, if you do not have right DatabaseAgent looking after that task item.
DatabaseAgent periodically checks task items and if tasks must be
executed (based on the value set in the Scheduling field), it
executes actual code
By default the DatabaseAgent is called each 10
minutes
If you decide to go with 2nd option, check this article first.
In short, you need to define your task class and start method in the config files (check out the /sitecore/admin/showconfig.aspx page, to make sure config changes are applied successfully)
<scheduling>
<!--
Time between checking for scheduled tasks waiting to execute
-->
<frequency>00:00:05</frequency>
<agent type="NameSpace.TaskClass" method="MethodName" interval="00:10:00"/>
</agent>
</scheduling>
As specified in the other answers, you can use config file or database to execute your task. However, it seems that you want to run it manually.
I have a custom module on the Sitecore Marketplace which allows you to select the task you want run. Here is the link
In brief, you need to go to the Sitecore control panel, then click on administration and lastly click on the Run Agent.
It will open a window where you can select the task. I am assuming that the task you have implemented does not take the Item details on which you are when triggering the job.

Django cron-job for user-specified times

I am wanting to send notifications based on a user-specified time. IE, in google calendar, I can receive a text message when my task time is hit.
Is the solution to this to run a cron job, have it execute every minute and scan which users have a time equaling the current time?
Since you tagged your question with celery, I assume you have celery running. You could use the eta kwarg to apply_async() to schedule a task to run at a specific time, see here:
http://docs.celeryproject.org/en/latest/userguide/calling.html#eta-and-countdown
If you need to use a cron job, I would not check if notification_time == current_time, but rather track unsent notifications with a boolean is_sent field on the model and check for notification_time <= current_time and not is_sent. This seems to be slightly less error prone. You could also add some form of check to prevent mass-sending notifications in case your system goes down for a few hours.