I know Informatica Cloud has a JSON target connector and I know you can build a JSON source in B2B DT Studio for PowerCenter, but don't see in the version 9.0 documentation that PowerCenter can write directly to a JSON target. Is it a feature in later or future releases of PowerCenter?
Yes there is a JSON Connector available in Informatica Marketplace. You can avail the 30 day trail of that plug-in and can check whether it works for you or not. These functionalities are provided by this connector:
•Create, update, and merge new JSON files.
•Load data into JSON arrays.
But to use this you need subscription to Informatica Cloud Integration Edition.
Please go to this link for further information: Informatica JSON Connector Link
Related
What is the best way to replicate data from Oracle Goldengate On premise to AWS (SQL or NOSQL)?
I was just checking this for azure,
My company is looking for solutions of moving data to the cloud
Minimal impact for on-prem legacy/3rd party systems.
No oracle db instances on the cloud side.
Minimum "hops" for the data between the source and destination.
Paas over IaaS solutions.
Out of the box features over native code and in-house development.
oracle server 12c or above
some custom filtering solution
some custom transformations
** filtering can be done in goldengate, in nifi, azure mapping, ksqldb
solutions are divided into:
If solution is alolwed to touch.read the logfile of the oracle server
you can use azure ADF, azure synapse, K2view, apache nifi, Orcle CDC adapter for BigData (check versions) to directly move data to the cloud buffered by kafka however the info inside the kafka will be in special-schema json format.
If you must use GG Trail file as input to your sync/etl paradigm you can
use a custom data provider that would translate the trailfile into a flowfile for nifi (you need to write it, see this 2 star project on github for a direction
use github project with gg for bigdata and kafka over kafkaconect to also get translated SQL dml and ddl statements which would make the solution much more readable
other solutions are corner cases, but i hope this gives you what you needed
In my company's case we have Oracle as a source db and Snowflake as a target db. We've built the following processing sequence:
On-premise OGG Extract works with on-premise Oracle DB.
Datapump sends trails to another host
On this host we have OGG for Big data Replicat that processes trails and then sends result as json to AWS S3 bucket.
Since Snowflake DB can handle JSON as a source of data and works with S3 bucket it loads jsons into staging tables where further processing takes place.
You can read more about this approach here: https://www.snowflake.com/blog/continuous-data-replication-into-snowflake-with-oracle-goldengate/
I have a bigquery stored procedures which will run on some GCS object and do magic out of it. The procedures work perfect manually but I want to call the procedure from Nifi. I have worked with HANA and know that I need JDBC driver to connect and perform query.
Either I can use the executeprocess processor or I could use executeSQL processor. I dont know to be honest
I am not sure how to achieve that in Nifi with bigquery stored procedures. Could anyone help me on this?
Thanks in advance!!
Updated with new error if someone could help
Option1: Executeprocess
The closest thing to "execute manually" is installing the Google Cloud SDK and execute within 'executeprocess' this:
bq query 'CALL STORED_PROCEDURE(ARGS)'
or
bq query 'SELECT STORED_PROCEDURE(ARGS)'
Option 2: ExecuteSQL
If you want to use ExecuteSQL with Nifi to call the stored procedure, you'll the BigQuery JDBC Driver.
Both 'select' and 'call' methods will work with BigQuery.
Which option is better?
I believe ExecuteSQL is easier than Executeprocess.
Why? because you need to install the GCloud SDK on all systems that might run executecommand, and you must pass the google cloud credentials to them.
That means sharing the job is not easy.
Plus, this might involve administrator rights in all the machines.
In the ExecuteSQL case you'll need to:
1 - Copy the jdbc driver to the lib directory inside your Nifi installation
2 - Connect to BigQuery using pre-generated access/refresh tokens - see JDBC Driver for Google BigQuery Install and Configuration guide - that's Oauth type 2.
The good part is that when you export the flow, the credentials are embedded on it: no need to mess with credentials.json files etc (this could be also bad from a security standpoint).
Distributing jdbc jars is easier than installing the GCloud SDK: just drop a file on the lib folder. If you need it in more than one node, you can scp/sftp it, or distribute it with Ambari.
I need to design and display a compute engine snapshot report for different projects in the cloud in data-studio. For this, I am trying to use the below Google Compute Engine snapshot-api for retrieving data.
https://compute.googleapis.com/compute/v1/projects/my-project/global/snapshots
The data may change everyday depending on the snapshots created from the disks. So the report should display all the updated data.
Can this rest-api be called directly from Google data-studio?
Alternatively, what is the best/simplest way to display the response in data-studio?
You can use a Community Connector in Data Studio to directly pull the data from the API.
Currently, their is no way to connect GCP Compute Engine (GCE) resource data or use the REST API in Data Studio. The only products that are available on connecting data from GCP are the following:
BigQuery
Cloud Spanner
Cloud SQL for MySQL
Google Cloud Storage
MySQL
PostgreSQL
A possible way to design and display a Compute Engine Snapshot Report for different projects in the Cloud in Data Studio is by creating a Google App Script (to call the snapshot REST API) with a Google Sheet, and then import the data into the sheet on Data Studio.
Additionally, if you have any questions in regards to Data Studio, I would suggest reviewing the following documents below:
Data Studio Help Center
Data Studio Help Community
EDIT: My apologies, it seems that their is a way to show snapshot API response data in Data Studio by using a Community Connector to directly pull the data from the API.
I am new learner in Informatica cloud data integration. Currently I am trying to convert SSIS ETL to Informatica.
While conversion, at one point I need to call a SQL Server stored procedure inside Informatica data integration which mainly update some data in tables. I tried many things but not getting success.
Can anyone have any idea how we can call a SQL Server stored procedure using informatica cloud data integration?
Please use the SQL transformation.
refer to the following link : network.informatica.com/videos/1213
Accept this as the right answer if it helped you. This will help people in future
You need to use a pre- or post processing script and call it from the native system as part of an integration. If this is for Application Integration then call as a command (and enable commands).
HTH
Scott S Nelson
I have a talend job designed to do an ETL task. I am using the talend open studio for data integration. I have gone through the beginners, and components manuals(of TOS 5.6), but cannot find a way to design a job that could be exported as a web service such that it could be called with parameters in the request, and then getting a collection of strings as the response. I found out that in the version 5.1 there were components like tRestRequest, and tRESTResponse that were used to achieve what I want, but for versions above 5.1 I have found nothing helpful.
The web service and REST components are now the part of the Talend ESB. In Talend DI only the consumer components are remained. You can download ESB, and build standalone job or set up an ESB server and deploy your jobs there.
The ESB is available here: http://www.talend.com/download/talend-open-studio?qt-product_tos_download=2#qt-product_tos_download