I want to copy the data into one excel file, while the mapping has been running in informatica.
I used Informatica last year, now am on SSIS. If I remember correctly, you can setup a separate connection for Excel target destination. There after you pretty much drag all fields from your source to the target destination (in this case Excel).
As usual develop a mapping with source and target.
But make target as Flat File while creating target.
Then run the mapping u will get
Related
I wanted to get the flat files as source without any file structure specified using IICS(Informatica Intelligent cloud services). Flat files names can be anything, and structure also will change any. And also need to create table dynamically based on flat file and insert into table.
There's a number of options here. You can use a fully parameterized mapping niside a taskflow that will start on file listener, prepare the parameters and statements to be executed as part of the pre-SQL statement on your Target.
Inside the mapping you define Source and Target as parameterized - and that's briefly it!
I need to read a text file (JSON format) and load it into a database table in Informatica Developer. For only one text file and one database table, that is easy.
But now I have N different text files, hence N different database tables and their corresponding data processor transformations. The transformation logic inside the mappings is the same. Besides creating N sets of mappings and workflows for each set of text files, is it possible to create just one generalized mapping and workflow to cater for all text files? I would appreciate it if any one of you could give me a general direction for me to explore further.
I have a Dataprep flow configured. The Dataset is a GCS folder (all files from it). Target is BigQuery table.
Since data is coming from multiple files, I want to have filename as of the columns in the resulting data.
Is that possible?
UPDATE: There's now a source metadata reference called $filepath—which, as you would expect, stores the local path to the file in Cloud Storage (starting at the top-level bucket). You can use this in formulas or add it to a new formula column and then do anything you want in additional recipe steps. (If your data source sample was created before this feature, you'll need to generate a new sample in order to see it in the interface)
Full notes for these metadata fields are available here: https://cloud.google.com/dataprep/docs/html/Source-Metadata-References_136155148
Original Answer
This is not currently possible out of the box. IF you're manually merging datasets with UNION, you could first process them to add a column with the source so that it's then present in the combined output.
If you're bulk-ingesting files, that doesn't help—but there is an open feature request open that you can comment on and/or follow for updates:
https://issuetracker.google.com/issues/74386476
I'm uploading data from zipped flat files to redshift using copy command, I would like to understand if there is any way to validate that the column order of the files is correct? (for example, if fields are all varchar then the data could be uploaded to the wrong columns).
In the copy command documentation it shows that you can specify the column order, but not for flat files, but I was wondering if there are any other approaches that would allow me to check how the columns have been supplied (for example, uploading only the header row into a dummy table to check, but that doesn't seem a possibility).
You can't really do this inside Redshift. COPY doesn't provide any options to only load a specific number of rows or perform any validation.
Your best option would be to do this in the tool where you schedule the loads. You can get the first line from a compressed file easily enough (zcat < file.z|head -1) but for a file on S3 you may have to download the whole thing first.
FWIW, the process generating the load file should be fully automated in such a way that the column order can't change. If these files are being manually prepared you're asking for all sorts of trouble.
I have a workflow which writes data from a table into a flatfile. It works just fine, but I want to insert a blank line inbetween each records. How can this be achieved ? Any pointer ?
Here, you can create 2 target instances. One with the proper data and in other instance pass blank line. Set Merge Type as "Concurrent Merge" in session properties.
Multiple possibilities -
You can prepare appropriate dataset into a relational table, and afterwards, dump data from that into a flat file. For preparation of that data set, you can insert blank rows into that relational target.
Send a blank line to a separate target file (based on some business condition using a router or something similar), after that you can use merge files option (in session config) to get that data into a single file.