We have requirement to load the data from web-service into the target database using Informatica. The web-service will be initiated by the source application whenever there is a change in source side. From Informatica side, we have to trigger the loading job whenever we receive the web-service instead of scheduling/batch jobs. using informatica powercenter
Related
What is the best way to replicate data from Oracle Goldengate On premise to AWS (SQL or NOSQL)?
I was just checking this for azure,
My company is looking for solutions of moving data to the cloud
Minimal impact for on-prem legacy/3rd party systems.
No oracle db instances on the cloud side.
Minimum "hops" for the data between the source and destination.
Paas over IaaS solutions.
Out of the box features over native code and in-house development.
oracle server 12c or above
some custom filtering solution
some custom transformations
** filtering can be done in goldengate, in nifi, azure mapping, ksqldb
solutions are divided into:
If solution is alolwed to touch.read the logfile of the oracle server
you can use azure ADF, azure synapse, K2view, apache nifi, Orcle CDC adapter for BigData (check versions) to directly move data to the cloud buffered by kafka however the info inside the kafka will be in special-schema json format.
If you must use GG Trail file as input to your sync/etl paradigm you can
use a custom data provider that would translate the trailfile into a flowfile for nifi (you need to write it, see this 2 star project on github for a direction
use github project with gg for bigdata and kafka over kafkaconect to also get translated SQL dml and ddl statements which would make the solution much more readable
other solutions are corner cases, but i hope this gives you what you needed
In my company's case we have Oracle as a source db and Snowflake as a target db. We've built the following processing sequence:
On-premise OGG Extract works with on-premise Oracle DB.
Datapump sends trails to another host
On this host we have OGG for Big data Replicat that processes trails and then sends result as json to AWS S3 bucket.
Since Snowflake DB can handle JSON as a source of data and works with S3 bucket it loads jsons into staging tables where further processing takes place.
You can read more about this approach here: https://www.snowflake.com/blog/continuous-data-replication-into-snowflake-with-oracle-goldengate/
I have a requirement where I need to run azure data factory pipeline from copying data from On-Premise database to Azure SQL Server Virtual Machine database(using VM data server due to limitations with azure SQL database and Azure SQL managed instance).
Hence I have created two self-hosted integration Runtime, 1 for on-prem VM database server, and another for azure VM data server.
But when I validate or run the pipeline, getting below error message
"source and sink in the copy activity must be connected via the same self-hosted integration runtime."
Can some please suggest the possible solution if any...
Basically it means that you cannot have 2 self-hosted integration runtimes in a single copy activity. What you are trying to do is copy from an onpremise source to another onpremise sink, but the IRs are different.
You will need to use 2 copy activities and an intermediate storage. For example:
1st copy: take the data from the source sql to a blob storage as a csv.
2nd copy: grab that csv and insert its data into sink database.
This way, you will be using only 1 IR for each copy activity.
Hope this helped!
We have requirement to load the data from web-service into the target database using Informatica. The web-service will be initiated by the source application whenever there is a change in source side. From Informatica side, we have to trigger the loading job whenever we receive the web-service instead of scheduling/batch jobs.
Please let me know if you have any option to achieve this using power exchange.
You could make use of HTTP transformation to load the data from web-service.
There is a demo on this in Informatica marketplace. Download the file there to get the complete implementation steps - https://marketplace.informatica.com/solutions/mapping_web_service_using_http_transformation
And with respect to triggering the work flows adhoc, may be you can make use of file watchers. Whenever there is a web service request, you can arrange to have a file transferred to your source location that indicates a new request. I am not sure if this is possible in your case. It would be great if you could provide more details. However, there is another demo here explaining implementation of file wather to auto trigger your workflows that could help -
https://marketplace.informatica.com/solutions/mapping_email_on_non_occurrence_of_event
I know Informatica Cloud has a JSON target connector and I know you can build a JSON source in B2B DT Studio for PowerCenter, but don't see in the version 9.0 documentation that PowerCenter can write directly to a JSON target. Is it a feature in later or future releases of PowerCenter?
Yes there is a JSON Connector available in Informatica Marketplace. You can avail the 30 day trail of that plug-in and can check whether it works for you or not. These functionalities are provided by this connector:
•Create, update, and merge new JSON files.
•Load data into JSON arrays.
But to use this you need subscription to Informatica Cloud Integration Edition.
Please go to this link for further information: Informatica JSON Connector Link
Is there any way to call a web service in a ssis package in order to insert some data into a table within SQL Server? How? any sample or guidance please?
I assume by your question, you are referring to using a web service as a destination for a data flow? I inherited a series of packages that integrate with our MS CRM site. As designed, these packages are a horrible fit for the SSIS paradigm but that's my burden to bear...
These packages generally fit the form of Source (OLE DB or Flat File) fed to a Script Task (destination). I don't know that providing all the code of a particular task would be enlightening. It's simply invokes the web service for each row sent into it. RBAR is not what SSIS or set based languages are made for but you can certainly do it.
The Script transformation will have a web reference (ours is named CrmSdk) to the service.
Declare an instance of the service as a member of ScriptMain.
Instantiate that service in your script, passing credentials as needed. Most likely in your PreExecute method
Make calls to the web service in your Input0_ProcessInputRow method using the Row.Column1 notation. Do be aware of nulls and how the web service handles them. Our code uses service.CompanyName = Row.CompanyName_IsNull ? string.Empty : Row.CompanyName;
If your intention is to use a web service at the Control Flow level, be aware that the default Task has a hard coded 5 minute timeout. Not sure if that's still the case, but in the 2005 package I was dealing with, we had to use a straight Script Task to communicate with our webservice (it was cleansing millions of rows of address data in batch fashion) to bypass the timeout issue. Reference to timeout property