I am looking for information on looping in Informatica. Specifically, I need to check if a source table has been loaded, if it has, move to next step, if not wait X minutes and check the status table again. I would prefer direction to a place I can learn this on my own, but I need to confirm this is even possible as I have not found anything on my google searches.
You can use a simple shell script to do this wait and watch capability.
#/bin/sh
# call it as script_name.sh
# it will wait for 10 min and check again for data, in total it will wait for 2hours. change them if you want to
# Source is assumed as oracle. change it as per your source.
interval=600
loop_count=10
counter=0
while true
do
$counter=`expr $counter + 1 `
db_value=`sqlplus -s user/pass#local_SID <<EOF
set heading off
set feedback off
SELECT count(*) FROM my_source_table;
exit
EOF`;
if [ $db_value -gt 0 ]; then
echo "Data Found."
exit 0
else
if [ $counter -eq $loop_count ]
then
echo "No data found in source after 2hours"
exit 1
else
sleep $interval
fi
fi
done
And add this shell script(in a CMD task) to the beginning of the workflow.
Then use informatica link condition as if status= 0, proceed else email that wait time is over.
You can refer to the pic below. This will send a mail if wait time is over and still data is not there in source.
In general, looping is not supported in Informatica PowerCenter.
One way is to use scripts, as discussed by Koushik.
Another way to do that is to have a Continuously Running Workflow with a timer. This is configurable on Scheduler tab of your workflow:
Such configuration makes the workflow start again right after it succeeds. Over and over again.
Workflow would look like:
Start -> s_check_source -> Decision -> timer
|-> s_do_other_stuff -> timer
This way it will check source. Then if it has not been loaded trigger the timer. Thet it will succeed and get triggered again.
If source turns out to be loaded, it will trigger the other session, complete and probably you'd need another timer here to wait till next day. Or basically till whenever you'd like the workflow to be triggered again.
Related
I want to use Step function Map, and have this logic that for each item in the input array
Lambda: checks a status of something:
CHOICE: true -> move on | false -> wait X hours and check for status again.
Does this mean if we set a max concurrency of 10 & have all 10 concurrently running maps in the wait stage because the status of something is false for all 10, my step function is essentially stuck and won't be able to continue on until one or more status of something comes back true?
It's probably the case that you want to avoid using Max Concurrency with Wait state in it, but I just want to confirm this is the case.
My alternative approach is to simply stage it to be processed again later and not block it.
Thanks.
I have a subroutine to check if a disk is mounted,
I would like to know how do I make this subroutine always run every 5 seconds.
thanks in advance!
on checkMyDiskIsMounted()
tell application "Finder"
activate
if exists disk "myDisk" then
--do anything
else
--do anything
end if
end tell
end checkMyDiskIsMounted
Using things like AppleScript's delay command, a shell utility such as sleep, or even a tight repeat loop should all be avoided, as those tend to block the user interface while they are running.
A repeating timer could be used to periodically poll, but instead of wasting time continually checking for something that may or may not happen, NSWorkspace can be used, as it provides notifications for exactly this kind of thing (amongst others). The way this works is your application registers for the particular notifications that it is interested in, and the specified handler is called when (if) the event occurs.
Note that the following script includes statements so that it can be run from the Script Editor as an example - observers are added to the application instance, and will stick around until they are removed or the application is quit:
use AppleScript version "2.4" -- Yosemite (10.10) or later
use framework "AppKit"
use scripting additions
on run -- or whatever initialization handler
# set up notifications
tell current application's NSWorkspace's sharedWorkspace's notificationCenter
its addObserver:me selector:"volumeMounted:" |name|:(current application's NSWorkspaceDidMountNotification) object:(missing value)
its addObserver:me selector:"volumeUnmounted:" |name|:(current application's NSWorkspaceDidUnmountNotification) object:(missing value)
end tell
end run
on volumeMounted:aNotification -- do something on mount
set volumeName to (NSWorkspaceVolumeLocalizedNameKey of aNotification's userInfo) as text
display notification "The volume " & quoted form of volumeName & " was mounted." with title "Volume mounted" sound name "Hero" -- or whatever
end volumeMounted:
on volumeUnmounted:aNotification -- do something on unmount
set volumeName to (NSWorkspaceVolumeLocalizedNameKey of aNotification's userInfo) as text
display notification "The volume " & quoted form of volumeName & " was unmounted." with title "Volume unmounted" sound name "Funk" -- or whatever
end volumeUnmounted:
Four options:
A repeat loop with delay as suggested by matt
repeat
-- code
delay 5
end repeat
A (stay open) applet with idle handler
on run
-- do intialiations
end run
on idle
-- code
return 5
end idle
An AppleScriptObjC notification like suggested by red_menace
A launchd agent observing the /Volumes folder
In favor of option 3 and 4 which inexpensively notify about a change the first two options which poll periodically are discouraged.
I am retrieving a list of files in a folder with tens of thousands of files. I've let the script execute for a few hours but it never seems to progress after this part. Is there a way to diagnose what's going on or the progress of the list? It used to take a few minutes to load the list, so I am not sure why it's all of a sudden hanging now.
Code
function EchoDump($echoOut = 0)
{
if($echoOut)
echo str_repeat("<!-- Agent Smith -->", 1000);
else
return str_repeat("<!-- Agent Smith -->", 1000);
}
$sftp_connection = new Net_SFTP($website ,2222);
$login_result = $sftp_connection->login($credentials->login(), $credentials->password());
echo "<pre>Login Status: ".( $login_result ? "Success" : "Failure") ."</pre>";
if($sftp_connection->chdir($photosFolder))
echo "<pre>Changed to $photosFolder</pre>";
else
{
echo "<pre>Failed to change to $photosFolder</pre>";
}
echo "<pre>Downloading List.</pre>";
EchoDump(1);
$rawlist = $sftp_connection->rawlist();
echo "<pre>List downloaded.</pre>";
EchoDump(1);
Output
Login Status: Success
Changed to /vspfiles/photos
Downloading List.
And then it will just sit with Downloading List as the last output forever.
Real time logging might help provide some insight. You can enable that by adding define('NET_SSH2_LOGGING', 3); to the top of your script. Once you got that maybe post it on pastebin.org for dissection.
That said, I will say that rawlist() doesn't return directory info on demand. It'll return everything. It's not the most efficient of designs but back in phpseclib 1.0 days Iterators didn't exist. phpseclib works with PHP4 so it had to work with what it had. The current master branch requires PHP 5.6 and is under active development so maybe this behavior will change but we'll see.
I need guidence on implementing WHILE loop with Kettle/PDI. The scenario is
(1) I have some (may be thousand or thousands of thousand) data in a table, to be validated with a remote server.
(2) Read them and loopup to the remote server; I use Modified Java Script for this as remote server lookup validation is defined in external Java JAR file (I can use "Change number of copies to start... option on Modified java script and set to 5 or 10)
(3) Update the result on database table. There will be 50 to 60% connection failure cases each session.
(4) Repeat Step 1 to step 3 till all gets updated to success
(5) Stop looping on Nth cycle; this is to avoid very long or infinite looping, N value may be 5 or 10.
How to design such a WhILE loop in Pentaho Kettle?
Have you seen this link? It gives a pretty well detailed explanation of how to implement a while loop.
You need a parent job with a sub-transformation for doing a check on the condition which will return a variable to the job on whether to abort or to continue.
I need to detect when a computer is idle for a certain time period. My definition of idleness is:
No users logged in, either by remote methods or on the local machine
X server inactivity, with no movement of mouse or key presses
TTY keyboard inactivity (hopefully)
Since the majority of distros have now moved to logind, I should be able to use its DBUS interface to find out if users are logged in, and also to monitor logins/logouts. I have used xautolock to detect X idleness before, and I could continue using that, but xscreensaver is also available. Preferably however I want to move away from any specific dependencies like the screensaver due to different desktop environments using different components.
Ideally, I would also be able to base idleness on TTY keyboard inactivity, however this isn't my biggest concern. According to this answer, I should be able to directly query the /dev/input/* interfaces, however I have no clue how to go about this.
My previous attempts at making such a monitor have used Bash, due to the ease of changing a plain text script file, howver I am happy using C++ in case more advanced methods are required to accomplish this.
From a purely shell standpoint (since you tagged this bash), you can get really close to what you want.
#!/bin/sh
users_are_logged_in() {
who |grep -q .
return $?
}
x_is_blanked() {
local DISPLAY=:0
if xscreensaver-command -time |grep -q 'screen blanked'; then
return 0 # we found a blanked xscreensaver: return true
fi
# no blanked xscreensaver. Look for DPMS modes
xset -q |awk '
/DPMS is Enabled/ { dpms = 1 } # DPMS is enabled
/Monitor is On$/ { monitor = 1 } # The monitor is on
END { if(dpms && !monitor) { exit 0 } else { exit 1 } }'
return $? # true when DPMS is enabled and the monitor is not on
}
nobody_here() {
! users_are_logged_in && x_is_blanked
return $?
}
if nobody_here; then
sleep 2m
if nobody_here; then
# ...
fi
fi
This assumes that a user can log in in two minutes and that otherwise, there is no TTY keyboard activity.
You should verify that the who |grep works on your system (i.e. no headers). I had originally grepped for / but then it won't work on FreeBSD. If who has headers, maybe try [ $(who |grep -c .) -gt 1 ] which will tell you that the number of lines that who outputs is more than one.
I share your worry about the screensaver part; xscreensaver likely isn't running in the login manager (any other form of X would involve a user logged in, which who would detect), e.g. GDM uses gnome-screensaver, whose syntax would be slightly different. The DPMS part may be good enough, giving a far larger buffer for graphical logins than the two minutes for console login.
Using return $? in the last line of a function is redundant. I used it to clarify that we're actually using the return value from the previous line. nobody_here short circuits, so if no users are logged in, there is no need to run the more expensive check for the status of X.
Side note: Be careful about using the term "idle" as it more typically refers to resource (hardware, that is) consumption (e.g. CPU load). See the uptime command for load averages for the most common way of determining system (resource) idleness. (This is why I named my function nobody_here instead of e.g. is_idle)