Cannot assign instance name to concurrent workflow in Informatica - informatica

In Informatica, I can start a workflow but cannot get it to recognize my instance name in the session log and Workflow Monitor.
The workflow starts but in the session log it displays this:
Workflow wf_Tenter image description hereemp started with run id [22350], run instance name [], run type [Concurrent Run with Un[enter image description here][1]ique Instance Name]
Instance name is blank.
My command is:
pmcmd startworkflow -sv <service> -d <domain> -u <user> -p <password> -f <folder> -rin INST1 -paramfile <full param file path name> wf_Temp
I have edited the workflow and selected the checkbox Configure Current Execution. Inside Configure Concurrent Execution button, I have created three instances: INST1, INST2, INST3, but without any associated parameter files. All parameter files are blank.
I understand, I think, that in order to start a workflow with PMCMD I must pass in one of the configured instance names (i.e. INST1, INST2, INST3, etc.)
If I execute the PMCMD command from Putty a second time to see the second instance run, I receive a message that workflow is still running and I have to wait? Why? I have checked the Concurrent Workflow box in the workflow.
ERROR: Workflow [wf_Temp]: Could not start execution of this workflow because the current run on this Integration Service has not completed yet.
Disconnecting from Integration Service
So, I think I'm close, but am missing something. The workflow runs with the parameter file I pass in PMCMD but the instance name seems to be ignored.
Further. Do, I have to pre-configure instance names in the Workflow manager? Is the PMCMD instance and parameter file parameters enough? It doesn't seem quite so dynamic if Instances have to be pre-defined in the workflows.
Thanks.
#MacieJG
Here's the screenshots from Putty when I run the command. You can see the instance name DALLAS is being passed through the PMCMD OK. No combination ever gets the Instance name. I did not include the pics of your suggested Test 1, but results were same.. still no instance.

Here's my complete test as requested in a comment above. I tried my best to put everything you may need here, but if I missed anything, just let me know. So here goes...
I've created a very simple workflow to run with instance name. It uses a timer to wait and a command tast to write the instance name to a file:
The concurrent execution has been set up in the most simple way:
Now, I've prepared the followig batch to run the workflow (just user & password removed):
SET "PMCMD=C:\Informatica\9.5.1\clients\PowerCenterClient\CommandLineUtilities\PC\server\bin\pmcmd"
%PMCMD% startworkflow -sv Dev_IS -d Domain_vic-vpc -u ####### -p ####### -f Dev01 -rin GLASGOW wf_Instance_Test
%PMCMD% startworkflow -sv Dev_IS -d Domain_vic-vpc -u ####### -p ####### -f Dev01 -rin FRANKFURT wf_Instance_Test
%PMCMD% startworkflow -sv Dev_IS -d Domain_vic-vpc -u ####### -p ####### -f Dev01 -rin GLASGOW wf_Instance_Test
It runs three instances, two of them with same name, just to test it. I run the batch the following way to capture the output:
pmStartTestWF.bat > c:\MG\pmStartTestWF.log
Once I execute it, here what I see in workflow monitor:
Just as expected, three instances executed and properly displayed. File output looks fine as well:
The output of pmcmd can be found here. Full definition of my test workflow is available here.
I really hope this will help you somehow. Feel free to let me know if you'd find anything missing here. Good luck!

You don't need to pre-configure instance names in workflow. Passing the instance name in pmcmd along with parameter filename is enough.

try this: pmcmd startworkflow -sv (service) -d (domain) -u (user) -p (password) -f (folder) -paramfile (full param file path name) -rin INST1 wf_Temp

To be precise: when you configure Concurrent Execution, you can specify if you:
allow concurrent run with same instance name
allow concurrent run only with unique instance name
In addition to that you may, but don't have to, indicate which instance should use which parameter file, so it won't be need to mention it while executing. But that's a separate feature.
Now, if you've chosen the first one, you will be able to invoke the WF multiple times with the very same command. If you've chosen the second one and try this, you will get the 'WF is already running' error.
The trouble is that your example seems correct at first glance. As per the log message:
Workflow wf_Temp started with run id [22350], run instance name [], run type [Concurrent Run with Unique Instance Name]
So you're allowing unique instances only. It seems that the instance name has not been used. First execution does not set the instance name, so similar second execution won't use it either and will get rejected as this is the same instance name (i.e. None).
You may try to change the setting to Allow concurrent run with same instance name, this shall allow the secon execution, but does not solve the main issue. For some reason the instance name does not get passed.
Please verify your command against the docs referenced below. Try to match the order perhaps. Please share some more info if it still fails.
Looking at the docs:
pmcmd StartWorkflow
<<-service|-sv> service [<-domain|-d> domain] [<-timeout|-t> timeout]>
<<-user|-u> username|<-uservar|-uv> userEnvVar>
<<-password|-p> password|<-passwordvar|-pv> passwordEnvVar>
[<<-usersecuritydomain|-usd> usersecuritydomain|<-usersecuritydomainvar|-usdv>
userSecuritydomainEnvVar>]
[<-folder|-f> folder]
[<-startfrom> taskInstancePath]
[<-recovery|-norecovery>]
[<-paramfile> paramfile]
[<-localparamfile|-lpf> localparamfile]
[<-osprofile|-o> OSUser]
[-wait|-nowait]
[<-runinsname|-rin> runInsName]
workflow

Related

Allow a bash script to run at boot in AWS Centos 7 instance

I need to create AWS CentOS 7 instance images for a customer, and need it to automatically send the ip and instance id to our AWS server every time the instance boots. For example, this is the very basic test version of the script I need to run:
#!/bin/bash
$serverIP=""
curl "https://$serverIP"/myphp.php?id='sentid'&ip='sentip'"
If the script is run directly, it works fine and is received by the server and processed there. But I can't get it to run at boot. I cannot put the script in the "User Data" directly due to security concerns as the customer can then see it easily, it needs to be in a script in the filesystem of the image.
I've tried several things that work fine on a physical Linux server, but not on AWS. I know profile.d runs every time someone logs in but over-sending like that is fine.
/etc/profile.d/myscript.sh
This stops the AWS instance from booting. Even just
#!/bin/bash/
echo "hello world"
prevents it from booting. The instance starts, but when you go to ssh into it you get 'Network Error: connection timed out', which is the standard error if you put a wrong ip in, or upset it by leaving a service like httpd enabled.
However, a blank bash script with just #!/bin/bash will allow the instance to start. Removing the script via user data usually makes it boot, sometimes it just dies.
The first thing I tried was crontab. I did:
crontab -e
#reboot /var/ook/myscript.sh
systemctl enable crond.service
But the instance wouldn't start. So I put "systemctl disable crond.service" in the User Data and one booted, but another still stayed dead. Myscript.sh was just another echo "doob" >> file which worked fine when run directly.
I tried putting in /etc/systemd/system/my-startup.service:
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/var/ook/writedood.sh
[Install]
WantedBy=multi-user.target
then:
systemctl enable my-startup.service
But this did nothing. My script "writedood.sh" was just echo "doob" >> ./file.txt ensuring file.txt was chmod 777. At least it didn't prevent the instance from starting.
To give context, an instance won't start if httpd is left enabled on shutdown, but will if you disable it in User Data.
I wanted to have a go at putting something in init.d but I'm not sure how to simply tell it to run a script once in the background, and given the plethora of success I've had so far with the instance not restarting, I'm not holding out much hope that that would work.
Thanks in advance!
EDIT::: I realised that sometimes AWS EC2 Instances Console is causing the problem where I can't ssh in after stopping and starting. It blanks the public ipv4 address when I click stop, but when I start, it puts the old address up and hangs. If I refresh the page, or uncheck/check the instance; the ip changes to the new address. This has caused much consternation.
Crontab worked if I placed the scripts and output file in different folders. It's very finicky; any errors, such as it not being able to write to the output file, and the instance won't start. I put startscript.sh in /usr/local/src, and output.out to /tmp/ to ensure there were no permissions problems, and now the instance starts and runs the script on boot.
I then realised that sometimes AWS EC2 Instances Console is causing the problem where I can't ssh in after stopping and starting. It blanks the public ipv4 address when I click stop, but when I start, it puts the old address up and hangs. If I refresh the page, or uncheck/check the instance; the ip changes to the new address. This has caused much consternation.

AWS-RunBashScript errors/warnings with Python

I have many EC2 instances that retain Celery jobs for processing. To efficiently start the overall task of completing the queue, I have tested AWS-RunBashScript in AWS' SSM with a BASH script that calls a Python script. For example, for a single instance this begins with sh start_celery.sh.
When I run the command in SSM, this is the following output (compare to other output below, after reading on):
/home/ec2-user/dh2o-py/venv/local/lib/python2.7/dist-packages/celery/utils/imports.py:167:
UserWarning: Cannot load celery.commands extension u'flower.command:FlowerCommand':
ImportError('No module named compat',)
namespace, class_name, exc))
/home/ec2-user/dh2o-py/tasks/task_harness.py:49: YAMLLoadWarning: calling yaml.load() without
Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
task_configs = yaml.load(conf)
Running a worker with superuser privileges when the worker accepts messages serialized with pickle is a very bad idea!
If you really want to continue then you have to set the C_FORCE_ROOT
environment variable (but please think about this before you do).
User information: uid=0 euid=0 gid=0 egid=0
failed to run commands: exit status 1
Note that only warnings are thrown. When I SSH to the same instance and run the same command (i.e. sh start_celery.sh), the following (same) output results BUT the process runs:
I have verified that the process does NOT run when doing this via SSM, and I have no idea why. As a work-around, I tried running the sh start_celery.sh command with bootstrapping in user data for each EC2, but that failed too.
So, why does SSM fail to actually run the process that I succeed in doing by actually via SSH to each instance running identical commands? The details below relate to machine and Python configuration:

How to Trigger a Workflow B start right after Workflow A completed?

I am new to Informatica Power Center.
My task is to trigger/start Workflow B right after when Workflow A just completed using infacmd command.
Suggestion is after all session in workflow a add a command task with "infacmd.sh startworkflow" to start the workflow b with all the options.
I've tried some guides but the version was too old. I'm using Informatica 10.1.1.
Thank you.
Fro the command task you can use the following command.
pmcmd startworkflow -sv $INFA_SERVICE -d $INFA_DOMAIN -u $INFRAREPO_USERID -p $INFRAREPO_PASSWD -f $INFA_FOLDER -wait $INFA_WORKFLOW
Replace the variable according to your domain/folder/workflow name etc.
Otherwise, you can create a shell script from where you have to call the workflow using the above command and call the shell script from your last session 'Post Session success command'
Consider creating a Command task that will touch a file and having Workflow B started together with Workflow A, with a Wait task that will wait for a file, and delete the file as a last step.
This way you don't need to invoke the pmcmd with hardcoded username and password.

chef-client failing as node_name not present in client.rb

I followed a tutorial here to bootstrap and register a node to chef server. The instance is in autoscaling group which is why I opted this method for bootstrapping.
Scenario is - I am using client.rb, validation.pem, trusted_certs from s3 to newly launched instance via userdata.
client.rb
log_location STDOUT
chef_server_url "https://chef.myserver.org/organizations/org"
validation_client_name "org-validator"
# Using default node name (fqdn)
trusted_certs_dir "/etc/chef/trusted_certs"
After downloading required files following command get executed to run chef-client with $INSTANCE_ID as node_name.
chef-client -N $INSTANCE_ID -j /etc/chef/first-boot.json
The initial bootstrapping is successful and the node gets registered to chef-server with instance-id as node-name but when running subsequent chef-client, it fails with the error:
ERROR: 401 "Unauthorized"
This is due to the node_name not present in client.rb.
How can I make the entry of node_name in client.rb during the very first chef-client run?
This is generally handled in the userdata script or config, a la echo "node_name '$HOSTNAME'" >>/etc/chef/client.rb. The specifics can vary depending on your naming scheme, sometimes you'll make some string edits to $HOSTNAME or use a different name entirely. This isn't strictly required, but without a name in the config file, Chef uses whatever the current FQDN of the system is, and it sounds like something in the initial Chef run changes the FQDN. Another option is to just not do that.

How to check whether my user data passing to EC2 instance is working

While creating a new AWS EC2 instance using the EC2 command line API, I passed some user data to the new instance.
How can I know whether that user data executed or not?
You can verify using the following steps:
SSH on launch EC2 instance.
Check the log of your user data script in:
/var/log/cloud-init.log and
/var/log/cloud-init-output.log
You can see all logs of your user data script, and it will also create the /etc/cloud folder.
Just for reference, you can check if the user data executed by taking a look at the system log from the EC2 console. Right click on your instance -
In the new interface: Monitor and Troubleshoot > Get System Log
In the old interface: Instance Settings > Get System log
This should open a modal window with the system logs
It might also be useful for you to see what the userdata looks like when it's being executed during the bootstrapping of the instance. This is especially true if you are passing in environmental variables or flags from the CloudFormation template. You can see how the UserData is being executed in two different ways:
1. From within the instance:
# Get instance ID
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
# Print user data
sudo cat /var/lib/cloud/instances/$INSTANCE_ID/user-data.txt
2. From outside the instance
Note: this will only work if you have configured the UserData shell in such a way that it will output the commands it runs.
For bash, you can do this like as follows:
"#!/bin/bash\n",
"set -x\n",
Right click on the EC2 instance from the EC2 console -> Monitor and Troubleshoot -> Get system log. Download the log file and look for something a section that looks like this:
ip-172-31-76-56 login: 2021/10/25 17:13:47Z: Amazon SSM Agent v3.0.529.0 is running
2021/10/25 17:13:47Z: OsProductName: Ubuntu
2021/10/25 17:13:47Z: OsVersion: 20.04
[ 45.636562] cloud-init[856]: Cloud-init v. 21.2-3...
[ 47.749983] cloud-init[896]: + echo hello world
this is what you would see if the UserData was configured like this:
"#!/bin/bash\n",
"set -x\n",
"echo hello world"
Debugging user data scripts on Amazon EC2 is a bit awkward indeed, as there is usually no way to actively hook into the process, so one ideally would like to gain Real time access to user-data script output as summarized in Eric Hammond's article Logging user-data Script Output on EC2 Instances:
The recent Ubuntu AMIs still send user-data script to the console
output, so you can view it remotely, but it is no longer available in
syslog on the instance. The console output is only updated a few
minutes after the instance boots, reboots, or terminates, which forces
you to wait to see the output of the user-data script as well as not
capturing output that might come out after the snapshot.
Depending on your setup you might want to ship the logs to a remote logging facility like Loggly right away, but getting this installed early enough can obviously be kind of a chicken/egg problem (though it works great if the AMI happens to be configured like so already).
Enable logging for your user data
Eric Hammond, in "Logging user-data Script Output on EC2 Instances (2010, Hammond)", suggests:
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
Take care to put a space between the two > > characters at the beginning of the statement.
Here’s a complete user-data script as an example:
#!/bin/bash -ex
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
echo BEGIN
date '+%Y-%m-%d %H:%M:%S'
echo END
Put this in userdata
touch /tmp/file2.txt
Once the instance is up you can check whether the file is created or not. Based on this you can tell if the userdata is executed or not.
Have your user data create a file in your ec2's /tmp directory to see if it works:
bob.txt:
#!/bin/sh
echo 'Woot!' > /home/ec2-user/user-script-output.txt
Then launch with:
ec2-run-instances -f bob.txt -t t1.micro -g ServerPolicy ami-05cf5c6d -v