I am querying DEVOPS to create reports from the source data in POWER BI. I can do this no problem. I'd like to be able to restrict the data that is collected from the ODATA feed by date(CreatedDate). This is to minimise the amount of churn client side.
I can manually enter a date for CreatedDate ge [date] - this works fine.
I want to use a parameter instead called "ReportingDateFrom" as there are a few projects I want to query and collate without editing each query individually.
I've tried many different combinations of the below. I'm currently experiencing error as below.
What am I doing wrong?
Error
Expression.Error: We cannot apply operator & to types Date and Text.
Details:
Operator=&
Left=01/12/2022
Right=' and State ne 'Removed' and State eq 'Closed'
Example Code
let
Source = OData.Feed("https://analytics.dev.azure.com/zzz/xxx/_odata/v4.0-preview/WorkItems?"
&"$filter=(WorkItemType eq 'User Story' or WorkItemType eq 'Bug' or WorkItemType eq 'Spike' or WorkItemType eq 'Show and tell') and CreatedDate ge '" , DateTime.Date(ReportingDateFrom) & "' and State ne 'Removed' and State eq 'Closed' "
&" and Descendants/any()"
&"& $select=WorkItemId,Title,WorkItemType,State,Custom_CostCode,ChangedOn,Children"
&"& $expand=AssignedTo($select=UserName),Iteration($select=IterationPath,StartDate,EndDate),ChangedOn($select=Date),Children($select=WorkItemId,Title,WorkItemType,State,CompletedWork;$expand=AssignedTo($select=UserName)),"
&"Descendants("
&"$apply=filter(WorkItemType eq 'Task' or WorkItemType eq 'Defect')"
&"/aggregate(CompletedWork with sum as CompletedWork)"
&")",
null, [Implementation="2.0",OmitValues = ODataOmitValues.Nulls,ODataVersion = 4]),
#"Replaced Errors" = Table.ReplaceErrorValues(Source, {{"AssignedTo", "Unassigned"}})
in
#"Replaced Errors"
You need to convert your date to text before concatenating. Something like:
Text.From( DateTime.Date(ReportingDateFrom))
Related
I have a lot of tables where in each of them there is an int filter. I want to simply change that value from editing the parameter here
and reference it from each query of each table like this
Is there any way I can do this?
I've tried searching into Power BI docs but I have not found anything yet
Yes. Either inject the parameter value into the query through string manipulation, like this or use the parameter in a filter in a subsequent step and let "query folding" do its job. I don't know if query folding works when using SQL against Kusto/ADX, so you may need to use KQL instead.
This query demonstrates both approaches, the ago condition is injected into the query text because it can't be folded around the original query, but the filter is added through folding, and will be evaluated by Kusto:
let
KQL = "
SomeTable
| where TIMESTAMP > ago(" & Text.From(WindowDays) & "d)
",
Data = AzureDataExplorer.Contents("yourcluster.kusto.windows.net", "yourdb", KQL, [CaseInsensitive=true]),
#"Filtered Rows" = Table.SelectRows(Data, each [SomeColumn] = "SomeValue")
in
#"Filtered Rows"
And if you don't further filter or select rows the query sent to Kusto will be
SomeTable
| where TIMESTAMP > ago(2d)
| where SomeColumn = "SomeValue"
But normally your report will add additional filters and select particular columns.
I am new to Power BI. I am using an SQL stored procedure to get the data.
execute ED_DS_TRANS_DETAIL01 #DateFrom ='2022-09-20', #DateTo = '2022-09-20'
It does give me the required data but I want it to be incremental. For example, today is the 20th and tomorrow is the 21st. So I want to set up the power bi in such a way that it gets the 21st data and then the next data 22nd and so on. Also, it doesn't replace the previous date data and places the next data underneath the previous one.
I have tried the given solution
let
Source = (Query as text) => let
Source = Sql.Database("IP", "DB" , [Query=Query, CreateNavigationProperties=false])
in
Source
in
Source
let
tdy = Date.From(DateTime.LocalNow()),
yest = Date.AddDays(tdy , - 1),
sQuery = Table.FromRecords({
[sQuery = "execute ED_DS_TRANS_DETAIL01 #DateFrom ='" & Date.ToText(yest,[Format="yyyy-MM-dd"]) & "', #DateTo = '" & Date.ToText(tdy,[Format="yyyy-MM-dd"]) & "'"
]}),
#"Invoked Custom Function" = Table.AddColumn(sQuery, "Query2", each #"Fnc Query"([sQuery]))
in
#"Invoked Custom Function"
GUI
When I click "OK" button I am getting Details: "Microsoft SQL: Incorrect syntax near '='."
You can do it at the Power Query side...
First create the parameters of dBaseIP and dBase in case you may need to change it later...
Then create a custom function which will run the SQL query "New Query" → Other Sources → "Blank Query".
let
Source = (Query as text) => let
Source = Sql.Database(dBaseIP, dBase , [Query=Query, CreateNavigationProperties=false])
in
Source
in
Source
At last, create the below function which will update your query as per today's date...
let
tdy = Date.From(DateTime.LocalNow()),
yest = Date.AddDays(tdy , - 1),
sQuery = Table.FromRecords({
[sQuery = "execute ED_DS_TRANS_DETAIL01 #DateFrom ='" & Date.ToText(yest,[Format="yyyy-MM-dd"]) & "', #DateTo = '" & Date.ToText(tdy,[Format="yyyy-MM-dd"]) & "'"
]}),
#"Invoked Custom Function" = Table.AddColumn(sQuery, "Query2", each #"Fnc Query"([sQuery]))
in
#"Invoked Custom Function"
Check this sample file... Change the database IP and name and then try to run it...
I have one table in snowflake, I am performing bulk load using.
one of the columns in table is date, but in the source table which is on sql server is having null values in date column.
The flow of data is as :
sql_server-->S3 buckets -->snowflake_table
I am able to perform the sqoop job in EMR , but not able to load the data into snowflake table, as it is not accepting null values in the date column.
The error is :
Date '' is not recognized File 'schema_name/table_name/file1', line 2, character 18 Row 2,
column "table_name"["column_name":5] If you would like to continue loading when an error is
encountered, use other values such as 'SKIP_FILE' or 'CONTINUE' for the ON_ERROR option.
can anyone help, where I am missing
Using below command you can able to see the values from stage file:
select t.$1, t.$2 from #mystage1 (file_format => myformat) t;
Based on the data you can change your copy command as below:
COPY INTO my_table(col1, col2, col3) from (select $1, $2, try_to_date($3) from #mystage1)
file_format=(type = csv FIELD_DELIMITER = '\u00EA' SKIP_HEADER = 1 NULL_IF = ('') ERROR_ON_COLUMN_COUNT_MISMATCH = false EMPTY_FIELD_AS_NULL = TRUE)
on_error='continue'
The error shows that the dates are not arriving as nulls. Rather, they're arriving as blank strings. You can address this a few different ways.
The cleanest way is to use the TRY_TO_DATE function on your COPY INTO statement for that column. This function will return database null when trying to convert a blank string into a date:
https://docs.snowflake.com/en/sql-reference/functions/try_to_date.html#try-to-date
I am just getting my hands on PowerBI, i am trying to get the date column which is in text to display Month and Year and i am not able to. Any directions. The column type is text which has value like this "2010-09-09T14:46:41+0000" i am trying to display "2010 September".
Tested the below in Power Query for Excel, but hopefully it should still work for you in Power BI. You could create a simple function like:
ParseYearAndMonth = (someText as text) as text => DateTimeZone.ToText(DateTimeZone.From(someText), "yyyy MMMM")
(In the function above, we convert the text to type DateTimeZone, and then just the year and month components of the DateTimeZone back to text. However, there is no error handling implemented.)
Below is an example of invoking/using the function.
let
someText = "2010-09-09T14:46:41+0000",
ParseYearAndMonth = (textToParse as text) as text => DateTimeZone.ToText(DateTimeZone.From(textToParse), "yyyy MMMM"),
someAttempt = ParseYearAndMonth(someText)
in
someAttempt
Edit:
Your query might look something like this (although I can't be sure without testing):
let
Source = Facebook.Graph("graph.facebook.com/v2.8/Me/posts"),
#"Removed Other Columns" = Table.SelectColumns(Source,{"message", "created_time", "id", "story"}),
ParseYearAndMonth = (someText as text) as text => DateTimeZone.ToText(DateTimeZone.From(someText), "yyyy MMMM"),
invokedFunction = Table.AddColumn(#"Removed Other Columns", "yearAndMonth", each ParseYearAndMonth([created_time]), type text)
in
invokedFunction
I am importing data into Power BI desktop from an XML file. If the field value is empty, e.g.,
<Address></Address> then PBI assigns that column type "Table" and it is not possible to change the type into text. I am getting an error:
Expression.Error: We cannot convert a value of type Table to type Text.
Details:
Value=Table
Type=Type
It looks like if the first encountered value is empty, then all following rows of that column in the same "page" (data is being read in chunks a.k.a "pages" by the data connector) fetch are assigned type "Table", though not always.
For some columns, it even not possible to change the column data type (they have "table" icon)
Any ideas on how to change the column type to text?
Edit1: I have noticed that clicking "Table" actually shows there is a text value in that column. Maybe there is some invisible setting for Power BI to overcome this blank string value problem?
Edit2: it looks like I have the same problem as described here https://community.powerbi.com/t5/Desktop/XML-Import-Table-Type-in-Column/td-p/97512. There is no solution to this simple problem?
I found this https://community.powerbi.com/t5/Issues/Power-Query-should-not-wrap-simple-xml-elements-in-a-table/idi-p/125525. It looks it is "by design" and no solution exists. It's a pitty for wasting time...
I have found someone's solution:
let
ConvertTableField = (aTable as table, fieldName as text) =>
let
//alternative: https://community.powerbi.com/t5/Desktop/Expand-value-from-table/td-p/214838
//#"Expanded g_v" = Table.TransformColumns(#"Expanded ts_info", {{"g_v", each if _ is table then Table.FirstValue(_, "") else _}})
#"Lists from table" =
Table.TransformColumns(
aTable,
{
{
fieldName,
each
if _ is table then
Table.ToList(_)
else {_}
}
}
),
#"Expanded List" =
Table.ExpandListColumn(
#"Lists from table",
fieldName
)
in
#"Expanded List"
in
ConvertTableField
I haven't still figured out how to run this function for all the columns of the table. M language is the most difficult of the programming languages I know more or less...
I am not sure about the XML usage, but as far as table type to text conversion goes - this worked for me. Maybe it's an overkill.
let
Source = (Table1 as table) =>
let
Section = Record.ToTable(#sections[Section1]),
Compare =
Table.SelectRows(
Section,
each [Value] = Table1
){0},
GetName = Record.Field(Compare, "Name")
in
GetName
in
Source