I want to conditionally union 2 csv files using the UNION transformation.
Depending on the input parameter, the mapping would do any of the following:
Union A and B
Only A goes through the union transformation. B is ignored
Only B goes through the union transformation. A is ignored
If i understand requirement correctly,
For condition 1, only TableA goes to target
For condition 2, only TableB goes to target
You can do this using FILTERS and UNION. Pls refer to below diagram.
After reading form source you can attach two filters to filter data as per requirement. And then union them together before writing into target.
Filter A will have condition like - IIF(Condition1=True, True,false)
Filter B will have condition like - IIF(Condition2=True, True,false)
Pls note, i used IIF so you can write your own complex expression if you want. Whole mapping should look like this -
Source Qualifier A --> FilterA -->
| UNION --> Target
Source Qualifier B --> FilterB -->
Related
I have a requirement to filter (flatfile) only those records who has the colA values as 1,2,3,4,5,6 and also ColB as 'N'. The records that satisfy this condition from the source file should process to target.
Earlier it was said to check for only one value from colA. So therefore i applied
IIF(COLA='1' AND COLB'N',TRUE)
How to filter with multiple values for the same column? I am new to informatica power center.
There are two ways you can achieve this in an expression : using OR logical operator or using IN function.
With OR
IIF((COLA='1' OR COLA='2' OR COLA='3' OR COLA='4' OR COLA='5) AND COLB='N',TRUE)
Parenthesis are essential to group conditions on COLA.
With IN
IIF(IN(COLA,'1','2,'3','4','5') AND COLB='N',TRUE)
I find this one easier to read.
When there are multiple filters, they're evaluated by using the AND logical operator. That means all conditions must be TRUE at the same time.
I understand this when the filters are like:
AMOUNT>100, CATEGORY='Sales'
However, when one or all of the filters are given by FILTER formula, I am unable to visualise how the AND logic works (and what does all conditions must be true mean because a condition [FILTER] itself is a table). Please can you give an example.
All of the filters will be part of an expanded table and ultimately each condition is a set of rules for specific columns on this expanded table.
Can someone please explain me how to implement the following logic in informatica. but Not with source qualifier with other transformations inside the mapping.
SUM(WIN_30_DUR) OVER(PARTITION BY AGENT_MASTER_ID ORDER BY ROW_DT ROWS BETWEEN 30 PRECEDING AND 1 PRECEDING)
Basically this is sql(oracle) level requirement but i want at informatica level.
Use the Aggregator to calculate sums groupped by AGENT_MASTER_ID and self-join it on AGENT_MASTER_ID. Make sure to have the data sorted and use 'sorted input' property for aggregator - it will also be mandatory for a self join.
I've imported a datset into Rapidminer 5 and one of the columns that was supposed to be nominal or polynomial was set as a numeric. My data set has over 500 attributes so I don't really want to have to reimport my data every time I realize I've made a mistake like this. Is there some way to either automate the import process so that it saves the column types I set each time or can I go back and edit my already imported data set attribute types?
add this operator to your process, after you load the data:
Data Transformation > Type Conversion > Numerical to Polynomial
on the operator, select
attribute type filter = single
attribute = [name of your attribute]
here you go: http://i.stack.imgur.com/ov5yn.png
Select "Numerical to Polynomial"
Then change "attribute filter type" to 'subset' Then select attributes that you want to change.
One more suggestion, you better store this output in your local repository so you dont need the conversion everytime you need the data. So, you will have both original and duplicate in your basket. :)
Happy Data Mining...
apply the 'set role function':
It's listed under operators -> data tranformations -> Name and role modification -> Set role
We are planning to build a dynamic data import tool. Basically taking information on one end in a specified format (access, excel, csv) and upload it into an web service.
The situation is that we do not know the export field names, so the application will need to be able to see the wsdl definition and map to the valid entries in the other end.
In the import section we can define most of the fields, but usually they have a few that are custom. Which I see no problem with that.
I just wonder if there is a design pattern that will fit this type of application or help with the development of it.
I am not sure where the complexity is in your application, so I will just give an example of how I have used patterns for importing data of different formats. I created a factory which takes file format as argument and returns a parser for particular file format. Then I use the builder pattern. The parser is provided with a builder which the parser calls as it is parsing the file to construct desired data objects in application.
// In this example file format describes a house (complex data object)
AbstractReader reader = factory.createReader("name of file format");
AbstractBuilder builder = new HouseBuilder(list_of_houses);
reader.import(text_stream, builder);
// now the list_of_houses should contain an extra house
// as defined in the text_stream
I would say the Adaptor Pattern, as you are "adapting" the data from a file to an object, like the SqlDataDataAdapter does it from a Sql table to a DataTable
have a different Adaptor for each file type/format? example SqlDataAdptor, MySqlDataAdapter, they handle the same commands but different datasources, to achive the same output DataTable
Adaptor pattern
HTH
Bones
Probably Bridge could fit, since you have to deal with different file formats.
And Façade to simplify the usage. Handle my reply with care, I'm just learning design patterns :)
You will probably also need Abstract Factory and Command patterns.
If the data doesn't match the input format you will probably need to transform it somehow.
That's where the command pattern come in. Because the formats are dynamic, you will need to base the commands you generate off of the input. That's where Abstract factory is useful.
Our situation is that we need to import parametric shapes from competitors files. The layout of their screen and data fields are similar but different enough so that there is a conversion process. In addition we have over a half dozen competitor and maintenance would be a nightmare if done through code only. Since most of them use tables to store their parameters for their shapes we wrote a general purpose collection of objects to convert X into Y.
In my CAD/CAM application the file import is a Command. However the conversion magic is done by a Ruleset via the following steps.
Import the data into a table. The field names are pulled in as well depending on the format.
We pass the table to a RuleSet. I will explain the structure the ruleset in a minute.
The Ruleset transform the data into a new set of objects (or tables) which we retrieve
We pass the result to the rest of the software.
A RuleSet is comprise of set of Rules. A Rule can contain another Rule. A rule has a CONDITION that it tests, and a MAP TABLE.
The MAP TABLE maps the incoming field with a field (or property) in the result. There are can be one mapping or a multitude. The mapping doesn't have to involve just poking the input value into a output field. We have a syntax for calculation and string concatenation as well.
This syntax is also used in the Condition and can incorporate multiple files like ([INFIELD1] & "-" & [INFIELD2])="A-B" or [DIM1] + [DIM2] > 10. Anything between the brackets is substituted with a incoming field.
Rules can contain other Rules. The way this works is that in order for a sub Rule mapping to apply both it's condition and those of it's parent (or parents) have to be true. If a subRule has a mapping that conflicts with a parent's mapping then the subRule Mapping applies.
If two Rules on the same level have condition that are true and have conflicting mapping then the rule with the higher index (or lower on the list if you are looking at tree view) will have it's mapping apply.
Nested Rules is equivalent to ANDs while rules on the same level are equivalent of ORs.
The result is a mapping table that is applied to the incoming data to transform it to the needed output.
It is amicable to be being displayed in a UI. Namely a Treeview showing the rules hierarchy and a side panel showing the mapping table and conditions of the rule. Just as importantly you can create wizards that automate common rule structures.