I want to get the date, time and sequence of all the Informatica jobs running on dev server for the last 30 days. How can we get it?
Easiest would be query on rep_sess_log in Infromatica metadatabase (see Repository Guide).
SELECT
SUBJECT_AREA, WORKFLOW_NAME,MAPPING_NAME,
SUCCESSFUL_SOURCE_ROWS, FAILED_SOURCE_ROWS
ACTUAL_START, SESSION_TIMESTAMP END_TIME
FROM REP_SESS_LOG
WHERE
ACTUAL_START BETWEEN TO_DATE('01/JAN/2021','dd/mon/yyyy') AND
TO_DATE('31/JAN/2021','dd/mon/yyyy');
Related
I'm using Objectify to access my GCP Datastore set of Entites. I have a full list of around 22000 items that I need to load into the frontend:
List<Record> recs = ofy().load().type(Record.class).order("-sync").list();
The number of records has recently increased and I get an error from the backend:
com.google.apphosting.runtime.HardDeadlineExceededError: This request (00000185caff7b0c) started at 2023/01/19 17:06:58.956 UTC and was still executing at 2023/01/19 17:08:02.545 UTC.
I thought that the move to Cloud Firestore in Datastore mode last year would have fixed this problem.
My only solution is to break down the load() into batches using 2 or 3 calls to my Ofy Service.
Is there a better way to grab all these Entities in one go?
Thanks
Tim
Quicksight only supports 24 refreshes / 24 Hrs for FULL REFRESH.
I want to refresh the data every 30 Mins.
Answer:
Scenario:
Let us say I want to fetch the data from the source (Jira) and push it to SPICE and render it in Quicksight Dashboards.
Requirement:
Push the data every 30 Mins once.
Quicksight supports the following:
Full refresh
Incremental refresh
Full refresh:
Process - Old data is replaced with new data.
Frequency - Every 1 Hr once
Refresh count - 24 / Day
Incremental refresh:
Process - New data get appended to the dataset.
Frequency - Every 15 Min once
Refresh count - 96 / Day
Issue:
We need to push the data every 30 Min once.
It is going to be a FULL_REFRESH
When it comes to Full Refresh Quicksight only supports Hourly refresh.
Solution:
We can leverage API support from AWS.
Package - Python Boto 3
Class - Quicksight.client
Method - create_ingestion
Process - You can manually refresh datasets by starting new SPICE ingestion.
Refresh cycle: Each 24-hour period is measured starting 24 hours before the current date and time.
Limitations:
Enterprise edition accounts 32 times in a 24-hour period.
Standard edition accounts 8 times in a 24-hour period.
Sample code:
Python - Boto for AWS:
import boto3
client = boto3.client('quicksight')
response = client.create_ingestion(
DataSetId='string',
IngestionId='string',
AwsAccountId='string',
IngestionType='INCREMENTAL_REFRESH'|'FULL_REFRESH'
)
awswrangler:
import awswrangler as wr
wr.quicksight.cancel_ingestion(ingestion_id="jira_data_sample_refresh", dataset_name="jira_db")
CLI:
aws quicksight create-ingestion --data-set-id dataSetId --ingestion-id jira_data_sample_ingestion --aws-account-id AwsAccountId --region us-east-1
API:
PUT /accounts/AwsAccountId/data-sets/DataSetId/ingestions/IngestionId HTTP/1.1
Content-type: application/json
{
"IngestionType": "string"
}
Conclusion:
Using this approach we can achieve 56 Full Refreshes for our dataset also we can go one step further and get the peak hours of our source tool (Jira) and configure the data refresh accordingly. This way we can even achieve a refresh frequency of 10 Min once.
Ref:
Quicksight
Quicksight Gallery
SPICE
Boto - Python
Boto - Create Ingestion
AWS Wrangler
CLI
API
I have created a job in Cloud scheduler like below:
Name : Start_BOT1
Frequency : 0 */15 * * * (Asia/Calcutta)
Target : A topic in Pub/Sub
As per the Frequency the job has to start every 15 mins once. But the job is not working as expected. It runs only when we click on "Run NOW" button.
Can someone help explain how the scheduler works in GCP and how the timezones works here.
Here you can find detailed information on Configuring Cron Job Schedules with the unix-cron format.
The 1st asterisk represents the minute
The 2th asterisk represents the hour
The 3th asterisk represents the day of month
The 4th asterisk represents the month
The 5th asterisk represents the day of the week
For step values, you correctly used the slash, executing every N steps.
For your case - running the job every 15 minutes, the configuration would be: “*/15 * * * *”
You can select the time zone for evaluating the schedules either by using the dropdown on the GCP Console Create a job screen or the gcloud --time-zone flag when you create the job. The default time-zone is Etc/UTC.
Having an issue with AWS Instance schedule.
Im trying to shutdown Instances at 2am and start them up at 8am Mon-Fri.
When i tried to do this with one Period, i get the following error:
ERROR : Begin time 08:00:00 must be earlier than end time in 02:00:00
Is there any way around this? Help would be great.
Based on AWS documentation:
The begintime and endtime fields define when the Instance Scheduler will start and stop instances. If you specify a start time only, the instance must be stopped manually. Similarly, if you specify a stop time only, the instance must be started manually. If you don’t specify either time, the solution uses the days of the week, days of the month, or months rules to start and stop instances.
One way to solve your problem is, define two periods to a schedule. One for start and another for Stop (or) Use 3rd part solutions like INVOKE Cloud. Disclaimer:- I am co-founder
The solution is to have two periods. One period begintime is 00:00 and endtime 02:00. The other period begintime 08:00 and endtime 23:59. Add them to the same schedule in the periods StringSet.
Note:
The scheduler will not try to stop the instance at 23:59 and try to start it at 00:00 because by default the scheduler "checks the state of each instance at five minute intervals which means the solution will perform start and stop actions every five minutes" as seen in the facts. The scheduler skips over the first period endtime as the second period's begintime has already begun.
In my Django project, users are allowed to register to a free trial, but if they do not complete a purchase within 15 days, their accounts are locked out until they do complete the purchase. After 13 days (ie within 48 hours or expiry) I wish to send an email the registered user reminding him/her to purchase.
Currently, I have a cron job set up to run daily and check all trial accounts if the registration date and current date are 2 days apart and if so, I send an email.
I was wondering if there is a more elegant solution to do this?
If you don't want to mess with your cron file you should check out Celery, an asynchronous task queue written in Python. It was originally created with Django in mind but has since been broken out into a separate package. What you want to do then is set up a Celerybeat schedule like this:
CELERYBEAT_SCHEDULE = {
"purchase-reminder": {
"task": "accounts.tasks.remind",
"schedule": timedelta(hours=24),
},
}
This will call the task (read: function) accounts.tasks.remind every 24 hours.