Custom Function Using Column As Parameter - powerbi

I have a custom function created in the Power Query Editor that accepts two parameters. One parameter should be dynamic based upon the value in a column. When tested with static parameters, the function works correctly as far as using a static parameters goes. However, if the table column is supplied (to provide the dynamic value needed), it produces a cyclical error. The function is invoked using "Invoke Custom Function". The first parameter selects a field, while the second parameter is set statically.
The Invoke Custom Function that produces a correct result:
= Table.AddColumn(#"Rounded Up", "qryProjectedDate", each qryProjectedDate(5, -1))
The Invoke Custom Function that produces an error:
= Table.AddColumn(#"Rounded Up", "qryProjectedDate", each qryProjectedDate([RemainingBudgetDays_RndUp], -1))
The qryProjectedDate:
(TotalLoops as number, Loop as number) =>
let
CurrentLoop = Loop + 1,
output =
if Calendar[IsWorkDay]{CurrentLoop} = 1
then
(if CurrentLoop = TotalLoops - 1
then Calendar[Date]{CurrentLoop}
else #qryProjectedDate(TotalLoops, CurrentLoop))
else #qryProjectedDate(TotalLoops + 1, CurrentLoop)
in output as date
This function produces a stack overflow error and a cyclical error messages.
Further testing of just inputting the column value as a parameter and returning that same value produces the value that I would expect. Therefore, I believe my issue is in how I have written the power query M function.

if function(5,-1) works then using function([column1],-1) will also work if column1 contains a numeric five
Are you sure the column is named RemainingBudgetDays_RndUp ? Spelling and capitalization of the column name matter. Try temporarily renaming that column and look at the output to see what powerquery thinks that column is named
Once that is set, is the column formatted as numeric or alpha? If not numeric, change the column type or use Number.From() in your function
Try calling your function using a table with a single row, and the contents of the RemainingBudgetDays_RndUp column equal to a numeric five. Does it work? Then the data needs cleaning on some of the rows
Are there any nulls, alpha characters, out of bound, or other data in the RemainingBudgetDays_RndUp column that will break your function? To guard against that, try adding a try and otherwise null before and after strategic points in the function

Related

Can I generate 2 numbers from a single What-if parameter in Power BI?

I'm using a what-if parameter to filter a table. I'm trying to generate 2 number from this what-if parameter, as the visual "slicer" supports "between" selection and other multi-selection options.
When I change from single value selection to multi-number selection, I can't get any value on Parameter Value. When I change back to single value selection, I can get the parameter value, as seen on the pictures.
Is it possible to use the What-if parameter to generate two (or more) numbers with this multi-selection?
PS. I really need to use parameters like this, like a "between slicer", as the top and bottom numbers will be used to filter a couple of different columns with rather complex rules.
Yes! That's possible!
You can notice that when you create a What-if parameter, the Parameter Value measure is created with the code like:
Parameter Value = SELECTEDVALUE('Parameter'[Parameter])
The SELECTEDVALUE function returns a single value from a list, that's why no value is returned when you select multiple ones.
Knowing that, you can create measures to get the MAX and MIN value of those listed on the parameter, like this:
Top parameter = MAX('Parameter'[Parameter])
and
Bottom parameter = MIN('Parameter'[Parameter])

ColdFusion - POI Excel Workbook Format as Number

I'm creating an Excel file using ColdFusion and POI Workbook. The file is created great, however I have cells that contain numeric values and need to be stored as numbers for SUM and other functions to work properly.
I want to call code after the data has been added for the excel sheet that will basically loop over the data, and if the value is numeric, set the data format as numeric. That way when I open the Excel file, I can perform a SUM or whatever on that column of data.
Below is what I've tried so far, and it does "work" because all the cells have the #EBEBEB background color, however they still have the warning that says "Number stored as text" as seen below. How can I fix this?
// Create our dataFormat object
df = poiWorkbook.createDataFormat();
// Create our new style
formatNumber = poiWorkbook.createCellStyle();
formatNumber.setFillPattern( formatEvenRowRightAlignStyle.SOLID_FOREGROUND );
formatNumber.setAlignment( formatNumber.ALIGN_RIGHT );
XSSFColor = createObject("java", "org.apache.poi.xssf.usermodel.XSSFColor");
formatNumber.setFillForegroundColor( XSSFColor.init(Color.decode("##EBEBEB")) );
formatNumber.setDataFormat(df.getFormat("0.00"));
// Loop over the data and apply the format
for (x=0;x<rowCount;x++) {
for (y=0;y<colCount;y++) {
poiSheet.getRow(x).getCell(y).setCellStyle(formatNumber);
}
}
(Promoted from comments ...)
call code after the data has been added
Any reason why? While it is possible, it is simpler, and less error prone, to format while populating the sheet. Assuming the query column data type is numeric (not varchar or something), you should be able to call SpreadsheetFormatColumn for that column, after adding the query data.
Having said that, the fact that you are getting "Number stored as text" warnings in the first place makes me wonder about the data type of your query column. IIRC, later versions of CF should detect numeric columns when SpreadsheetAddRows is used, and apply a numeric format to those columns automatically. Is it possible your query column is being formatted as a string instead of some numeric type? As that would explain the results. The solution is to cast the column to a numeric type inside the db query.

Coldfusion Query broken column structure

A coldfusion query's column can be referenced like a 2D array from what I know of my past experience. Sometimes though I get this issue.
create query from spreadsheet
put column names into an array
only get first element when trying to access the row
have to use a workaround
//imports is a query object after this function
var imports = convertSpreadsheetWOHeaders(spreadsheet, 1);
//this is used to give name values in the json struct, not to access the query columns
var jsonHeaders = ListToArray("number,constructiontype,description,site_address,parcel,permit_date,note_Applicant,note_Contractor,valuation,note_Bld_Fees,note_Other_Fees");
//this gives me ["col_1","col_2","col_3",,,etc]. used to access query columns
var columnHeaders = imports.getColumnNames();
writeDump(imports[columnHeaders[1]]);
writeDump(imports);
I am left with just the first element in column one. And I get of course:
Message: You have attempted to dereference a scalar variable of type class java.lang.String as a structure with members.
When trying to do this:
structInsert(jsonStruct,jsonHeaders[j],imports[columnHeaders[j]][i]);
However, this works:
writeDump(ListToArray(ArrayToList(imports[columnHeaders[1]],'|'),'|',true));
I first do a dump of imports["col_1"] and i get only the first element.
Then I do a dump of ListToArray(ArrayToList(imports["col_1"])) like you see in the above image and it gives me the whole column.
Why can I not access the column correctly in the first place?
The Real Problem:
I was originally trying to access the jsonHeaders list as an array without doing ListToArray() on it. After that my function worked.
BUT. This next part is helpful.
When attempting to access a query object, doing queryObject["columnName"] is considered a shortcut of queryObject["columnName"][1] so coldfusion will just give you the first element.
But when I said ListToArray(ArrayToList()) coldfusion sees that ArrayToList MUST take in an array so an exception is made and the column is returned as an array.
To get a column back as an array to work with you can do a couple things
ListToArray(ArrayToList(query["column"]));
ListToArray(valueList(query.column));
valueArray(query, "column");

HBase scanning through shell command and mapreduce gives two different rersult

I have HBase table and it has more than billion records. When I query scan the HBase table with certain ValueFilter I get 41820 records, but it took more than 35 mins to give the result but when I used mapreduce program to scan the same HBase table, I got the count with in 2 mins but gave me 41035 recods. I don't know.
Here is the shell command I use :
scan 'permhistory', { COLUMNS => 'h:e_source', FILTER => "ValueFilter( =, 'binaryprefix:AC_B2B' )" }
result : 41820
Here is the Scan object in mapreduce :
Scan scan = new Scan();
scan.setCaching(2000);
scan.setCacheBlocks(false);
scan.addFamily(Bytes.toBytes("h"));
scan.addColumn(Bytes.toBytes("h"), Bytes.toBytes("e_source"));
SingleColumnValueFilter filter = new SingleColumnValueFilter(Bytes.toBytes("h"),
Bytes.toBytes("e_source"),CompareOp.EQUAL,Bytes.toBytes("AC_B2B"));
filter.setLatestVersionOnly(false);
scan.setFilter(filter);
Any idea? This is my first post on here. Experts out there, would you please help me out? I am kind of stuck on automating our system
In the mapreduce you are using this constructor
public SingleColumnValueFilter(byte[] family,
byte[] qualifier,
CompareFilter.CompareOp compareOp,
byte[] value)
It means you instantiate a Filter using default comparator, but in the hbase shell you're using
"ValueFilter( =, 'binaryprefix:AC_B2B' )"
A binaryprefix comparator, so you should try this instead
SingleColumnValueFilter filter = new SingleColumnValueFilter(Bytes.toBytes("h"),
Bytes.toBytes("e_source"),
CompareOp.EQUAL,
new BinaryPrefixComparator(Bytes.toBytes("AC_B2B")));
Moreover, in the hbase shell you are using ValueFilter and in the mapreduce you are using SingleColumnValueFilter. For your reference:
SingleColumnValueFilter
This filter is used to filter cells based on value. It takes a
CompareFilter.CompareOp operator (equal, greater, not equal, etc), and
either a byte [] value or a ByteArrayComparable. If we have a byte []
value then we just do a lexicographic compare. For example, if passed
value is 'b' and cell has 'a' and the compare operator is LESS, then
we will filter out this cell (return true). If this is not sufficient
(eg you want to deserialize a long and then compare it to a fixed long
value), then you can pass in your own comparator instead.
You must also specify a family and qualifier. Only the value of this
column will be tested. When using this filter on a Scan with specified
inputs, the column to be tested should also be added as input
(otherwise the filter will regard the column as missing).
To prevent the entire row from being emitted if the column is not
found on a row, use setFilterIfMissing(boolean). Otherwise, if the
column is found, the entire row will be emitted only if the value
passes. If the value fails, the row will be filtered out.
ValueFilter
This filter is used to filter based on column value. It takes an
operator (equal, greater, not equal, etc) and a byte [] comparator for
the cell value.
In this case, since you're specifically set the column to be scanned, it will act the same way. The ValueFilter will filter all column, and the SingleColumnValueFilter will only filter a specific column and omit the row altogether if it doesn't pass the filter.

Is there a way to get each row's value from a database into an array?

Say I have a query like the one below. What would be the best way to put each value into an array if I don't know how many results there will be? Normally I would do this with a loop, but I have no idea how many results there are. Would I need run another query to count the results first?
<CFQUERY name="alllocations" DATASOURCE="#DS#">
SELECT locationID
FROM tblProjectLocations
WHERE projectID = '#ProjectName#'
</CFQUERY>
Depending on what you want to do with the array, you can just refer to the column directly for most array operations, eg:
i = arrayLen(alllocations["locationID"]);
Using that notation will work for most array operations.
Note that this doesn't "create an array", it's simply a matter that a query columns - a coldfusion.sql.QueryColumn object is close enough to a CFML array for CF to be able to convert it to one when an array is needed. Hence the column can be passed to an array function.
What one cannot do is this:
myArray = q["locationID"];
This is because by default CF will treat q["locationID"] as a string if it can, and the string value is what's in the first row of the locationID column in the q query. It's only when an array is actually required will CF convert it to an array instead. This is basically how loose-typing works.
So if you just need to pass your query column to some function that expects an array, you can use the syntax above. If you want to actually put the column into a variable, then you will need to do something like this:
myArray = listToArray(valueList(q.localtionID));
NB: make sure you use <cfqueryparam> on your filter values instead of hard-coding them into your SQL statement.
myquery.column.toArray() is also a good undocumented choice.
Since you're only retrieving 1 field value from the query, you could use ValueList() to convert the query results into a comma-delimited list of locationIds, then use listToArray() to change that list into an array.
If you were retrieving multiple field values from the query, then you'd want to loop through the query, copy all the field values from the given row into a struct, and then add that struct to an array using arrayAppend().
(If you're not familiar with these functions, you can look them up in the Adobe docs or on cfquickdocs.com).