I am having an issue with SAS DI.
In certain columns, I would like the empty strings to appear as empty, instead of them being displayed as "NULL".
While they some times appear as empty, once I later run them in SQL, they come up as "NULL". I have attempted to use NUMCHAR variable in SAS DI, but I am not quite sure how to implement it as I am new to this language.
EDIT:
I will try to be more specific. In short, I have a workflow (Job) in SAS DI, where I have a table, I am using the table loader to choose some of the variables (i.e. A, B and C). The aforementioned variables have both values and empty values. On the occasion where an empty value appears, the table entry of that value is saved, although when I load it it shows as "NULL". What I would like to do, is change the workflow (either visually or using a code) to tell SAS DI to show values of columns A and B (but not C) as only empty (rather than "NULL")
Related
SOLVED USING A DIFFERENT APPROACH (see at the end)
I am trying to combine some queries into one by using the Table.Combine() function.
If I explicitly write the name of each query (e. g., Table.Combine({#"Name of query 1", #"Name of query 2"})) and then apply the changes, everything works fine.
However, since I want to make it dynamic, instead of writing a list of names, I pass the function a list of tables generated in a previous step:
So after I get this table, the next step is: = Table.Combine(PreviousStep[Value]). Note that Value is the name of the column that contains the tables. Apparently, by doing so this column of a table containing tables is converted to a list containing tables. This works fine (I can preview the resultset) until I hit that Apply changes button. When I do it, this message pops up:
I had a look at these threads: https://community.powerbi.com/t5/Desktop/We-cannot-convert-the-value-null-to-type-Table/td-p/391064, https://community.powerbi.com/t5/Desktop/We-cannot-convert-the-value-null-to-type-table/m-p/346056, but it didn't work. I've tried other approaches as well.
Further information:
Power BI Desktop version: 2.106.582.0 64-bit (June 2022)
Data source: combining existing queries that come from a single Excel file.
Steps followed to get that list of tables that I pass the Table.Combine() function:
let
Origen = #sections[Section1],
#"Convertido en tabla" = Record.ToTable(Origen),
#"Errores quitados" = Table.RemoveRowsWithErrors(#"Convertido en tabla", {"Value"}),
Personalizado1 = Table.SelectRows(#"Errores quitados", each Text.StartsWith([Name], "COMPRAS Y GASTOS")),
Personalizado2 = Table.Combine(Personalizado1[Value])
in
Personalizado2
I access all the queries I have (with the #sections keyword), convert it to a table, remove possible errors, filter to get the queries I want (the ones starting by "COMPRAS Y GASTOS") and then try to combine the queries).
A DIFFERENT APPROACH
What I wanted to do was merge tables that came from an Excel file, each of them referring to a year (2019, 2020, 2021, 2022). But I also wanted the combined table to update when new sheets were added on Excel (2023, 2024...).
I've tried many different approaches, like generating a dynamic list (from 2019 until the current year)... but for some reason none of them worked, even though the code apparently is correct.
So my new approach has been to create a sufficient amount of Excel sheets for the coming years (that are now empty, but when the new year comes the information will be filled in there), to create the queries referring to those sheets (they return empty tables) and merging those existing (but empty) tables with the ones from 2019-2022. This way, when data from 2023 is filled in in the sheet, the query is updated and it works.
It's a shame I couldn't actually solve the original problem I had, but this approach works.
I have an existing ETL solution built-in SAS Data Integration, where one of the columns in initially set to have all null values. I want to populate that column with actual data. The original column in that table was set to receive numeric values with specific format and in format. After I am changing the code (that is the easy part), I notice that the column doesn't accept character values (I did not get an error, I just noticed the column still having all NULL values).
Can anyone help ?
So you have a table that is defined in Data Integration studio (1) and created by running the job (2) a long time ago with a numeric column. Let us call that table THE_TABLE that field the_field and the job, The_Job, that loads data into THE_TABLE
You must be aware of the fundamental difference
defining a THE_TABLE in DI studio, which creates a description of the table in meta data
creating THE_TABLE by running The_Job, which creates a file in a folder with data
If The_Job really creates THE_TABLE from scratch each time (which is typical for ETL jobs), it is sufficient to do change THE_TABLE and The_Job in DI studio. Your edits will only change the meta data, but the next time you run The_job, THE_TABLE wil be created with the the right structure.
However, if The_Job updates THE_TABLE or appends to it, your edits will not change the structure of THE_TABLE and your job will not be fit for the structure of the file THE_TABLE like it still exists in the folder, so you must convert THE_TABLE before running The_Job.
This can be done with a simple program like
data THE_TABLE;
set THE_TABLE (drop=the_field); /* forget about the numeric field */
attrib the_field length=$200 format=$200.; /* and create the character field */
run;
The correct attrib statement might well be in the code generated for The_Job somewhere.
Mind that in a typical setup with a development, test and production environment, you will need that program once in each environment.
I have two tables in different databases. In a table A is the data, in the other table B are information for incremental load of the data from the first table. I want to load from table B and store the date of the last successful load from table A in a mapping variable $$LOAD_DATE. To achieve this, I read a date from table B and use the SETVARIABLE() function in a expression to set the $$LOAD_DATE variable. The port in which I do this is marked as output and writes into a dummy flat file. I only read on row of this source!
Then I use this $$LOAD_DATE variable in the Source Filter of the Source Qualifier of table A to only load new records which are younger than the date stored in the $$LOAD_DATE variable.
My problem is that I am not able to set the $$LOAD_DATE variable correctly. It is always the date 1753-1-1-00.00.00, which is the default value for mapping variables of the type date/time.
How do I solve this? How can I store a date in that variable and use it later in a Source Qualifiers source filter? Is it even possible?
EDIT: Table A has too much records to read them all and filter them later. This would be to expensive, so they have to be filtered at source filter level.
Yes, it's possible.
In the first map you have to initialize the variable, like this:
In first session configuration you have to define the Post-session on success variable assignment:
The second map (with your table A) will get the variable after this configuration of the session in Pre-session variable assignment:
It will work.
It is not possible to set a mapping variable and use it's value somewhere else in the same run, because, the variable is actually set when the session completes.
If you really want to implement it using mapping variables you have to create two mappings, one for setting the mapping variable and another for actual incremental load. You can pass a mapping variable value from one session to another in a workflow using a workflow variable. https://stackoverflow.com/a/26849639/2626813
Other solutions could be to use a lookup on B and a filter after that.
You can also write some scripts to query table B and modify the parameter file with the latest $LOAD_DATE value prior to executing the mapping.
Since we're having two different DBs, use two sessions. Get values in the first one and pass the parameters to the second one.
while writing code we can either use select statement or select field list or find method on table for fetching the records.
I wonder which of the statement helps in better performance
It really depends on what you actually need.
find() methods must return the whole table buffer, that means, all of the columns are projected into the buffer returned by it, so you have the complete record selected. But sometimes you only need a single column, or just a few. In such cases it can be a waste to select the whole record, since you won't use the columns selected anyway.
So if you're dealing with a table that has lots of columns and you only need a few of them, consider writing a specific select statement for that, listing the columns you need.
Also, keep in mind that select statements that only project a few columns should not be made public. That means that you should NOT extract such statements into a method, because imagine the surprise of someone consuming that method and trying to figure out why column X was empty...
You can look at the find() method on the table and find out the same 'select'-statement there.
It can be the same 'select; statement as your own an the performance will be the same in this case.
And it can be different select statement then your own and the performance will be depend on indexes on the table, select statement, collected statistics and so on.
But there is no magic here. All of them is just select statement - no matter which method do you use.
I'm trying to migrate a meta data from an excel spreadsheet to a SP 2013 document library. The columns are managed meta data columns with pre defined terms matching the data in the excel spreadsheet.
However I cannot copy and paste data from excel via Quick Edit in the doucment library without getting the following error "The data returned from the tagging UI was not formatted correctly"
This happens even when I remove all formatting or paste to notepad first.
Are there any simple solutions to this issue?
http://i.imgur.com/1bqpMPA.jpg
Thanks,
Any metadata fields are in fact foreign keys, as it were, to a dynamic, hidden table (or 'list', whatever you want to call it) within SharePoint. To paste a value into a metadata column, you need to know your element's guid (as in, within the term set) and then append that to each metadata element you're pasting in as a <name>|<guid> pair.
Getting the GUID for an element within your term set
Browse to [site-root]/TaxonomyHiddenList/AllItems.aspx and create a new view (or edit the default one) to display the field 'IdForTerm'.
Where you have a term 'apple', your IdForTerm may look like '1288beaf-82e0-4d81-b9de-ad5ad8382938'. Take a note of the guid for each term which appears within your input data.
Edit your input to correctly reference each term
Let's say you're importing your data from an Excel spreadsheet. Or from a CSV. It doesn't really matter. What you need to do is, basically, a find and replace down each managed metadata column, replacing 'term' with 'term|guid'. So our example from earlier, with the apple, would become 'apple|1288beaf-82e0-4d81-b9de-ad5ad8382938'.
Finally, assuming your view is set up in exactly the same order as your input data, you should be able to 'edit list' from within the browser, hit the leftmost side of your first input row (to select the entire row) and CTRL+V all of your data at the same time.
Note there appears to be a limit to the number of entries you can make at the same time. It appears to sit at around 5,000 elements.
Adding on to #rmacd's answer, you can also get the GUID for a given MMS term by first manually entering the value(s) you need in a Quick Edit cell, then copy and paste the same value(s) from SharePoint to Excel. The pasted value will appear with the full term|guid that you need to complete the bulk copy/paste.