I'm doing incremental data load from relatioal db to dynamically created flat file. Suppose if there are no new records in source the mapping not creating target file. I need a empty Target file if there are no records fetched From source
You can create a cmd task which will kick off on a condition.
in cmd task just put this command
touch /location/empty_file.txt
Link main session to this command task. Double click on link and add below condition to link.
$yourMainSessionName.SrcSuccessRows = 0
So, this command task will activate only when your main session pulls 0 rows.
Related
I have multiple source sending incremental data and there are no metadata columns at record level. How can I ensure that Airflow is processing data in the order of receipt. I may end-up processing the file in out-of-sync order.
Does airflow have inbuilt methods/way to handle the files in the order of receival. ?
Airflow version used :2.4.3
You can use boto to retrieve the last modified timestamp from files in your S3 bucket within a PythonOperator.
This question has an answer that shows how to pull the last modified timestamp. Then you can sort the keys by the timestamp, process the files in that order and move the files to an achieve folder or bucket so only new files are processed with every DAG run.
As a general note if you have any control over your sources I would prefer trying to add a timestamp at the record level, this seems like an easier option.
I am trying to run an Informatica workflow to check database table and write fetched data to a flat file on the server.
source: database table
target: flat file
transformations: non
this wf runs fine when "run on demand" but I need to run this continuously so I tried with INFA scheduler to run it every 5 minutes. When the scheduler is enabled the workflow continuously fails. Kindly help with any ideas to run this on scheduler.
oops... sorted, this was my mistake. I have not checked in the Target I created for flat file. my bad. Thanks all
Hello everyone and thanks for taking the time to read this question!
What I am trying to do in Informatica is populate and email a file based on a sql query. So right now I have
DB source table ----> SQ w/ query -----> flat file target (emails flat file to me every time) in a single session workflow.
What I would like to do is ONLY email the file if it has records. I also don't want to the workflow to fail if there are no records and no email is sent. Just go on it's merry way and wait for the next run.
Thank You!
Should be simple.
Create two sessions/mappings like this -
session 1 - Read using your SQL and write data into a dummy file.
session 2 - Depending on target success row of session1 this will run. Link condition should be - if Session1.TgtSuccessRows>0 then run session2. Add your mail id on session2's success e mail. You can use any dummy mapping here.
Please note you can use email task instead of session2.
Koushik
Is there a way to purge the log in Sitecore such that logs are written immediately. It's for production debugging.
Also strolling through log files, there are number of log files e.g. log.date.text and log.date.time.txt. Which one is the latest i.e. with our without time.
You can use next module for production server if you have remote access there :
https://marketplace.sitecore.net/Modules/S/Sitecore_Log_Analyzer.aspx
Other option is to use this module:
https://marketplace.sitecore.net/Modules/S/Sitecore_ScriptLogger.aspx
The log with no timestamp in the file name is the first on for that day.
A new log file is created each time the application pool restarts.
If you haven't changed any of the default log4net settings then the initial log file will be in the format log.yyyyMMdd.txt, each subsequent restart will cause a new file to be generated with the following format log.yyyyMMdd.HHmmss.txt.
The latest log file for the day will be the file with the latest timestamp.
I am setting up our project to integrate with a shipping platform called Endicia which has the ability to insert new rows into our database when a package is shipped.
How can I detect from python when a new row has been inserted?
My solution for now would be to query the DB every 30 seconds or so for new rows... is there another solution to send a signal from postgres to python?
You'd set up a custom command that is run by the manage.py file.
You'd put it in `yourapp/management/commands/' folder. Make sure to add an init.py file to both the management and commands folder or the command won't work. Then you create the code for the custom command.
Then, see this related question about running a shell script when changes are made to a postgres database. The answer there was to use PL/sh. You'll need to figure that part out on your own, but basically however you do it, the end result is that the script should call something like /path/to/app/manage.py command_name