I need to convert a query on excel to a Insert on SQL, I wont to create a function that will work on any query, and Invoke it, so he need to be quite reusable.
let
SQL = (TableX as table) => let
// Get Columns Name abd use on the columns INSERT
SOURCE = TableX,
Names = Table.ColumnNames(SOURCE),
ListN = Table.FromList(Names),
TransposeN = Table.Transpose(ListN),
CombineN = Table.CombineColumns(TransposeN,Table.ColumnNames(TransposeN),Combiner.CombineTextByDelimiter(",", QuoteStyle.None),"M"),
ColunmsAsText = List.Single(CombineN[M]),
// ColumnsasText is all the Columns Name
// This part I cant do I need to format all columns acording to the type and Combine They
COLUNMS = (ListN as text) => let
TYPE = some.function.to.get.type(SOURCE[ListN]),
if TYPE = date then FIELDS = FIELDS & "'" & Date.ToText([ListN], "yyyy-MM-dd") & "'," else
if TYPE = text then FIELDS = FIELDS & "'" & [ListN] & "'," else
if TYPE = number then FIELDS = FIELDS & [ListN] & "," else FIELDS = FIELDS & "#error,"
in
FIELDS
INSERT = "INSERT INTO " & "TABLENAME" & " (" & ColunmsAsText & ") VALUES (" & FIELDS & ");",
NEWCOLUMNADD = Table.AddColumn(SOURCE, "SQL", each INSERT)
in
NEWCOLUMNADD
in
SQL
The problem is the FIELDS variable where, I need to compose a string of all columns, and the function "some.function.to.get.type", and the output 'll be something like this, for each row:
INSERT INTO (Column_name1, Column_name2, Column_name3) value (15, '2017-05-20', 'Text hear');
Tanks for the help.
You want Value.Type
Value.Type("abc") = type text
Related
I'm trying to relabel a bunch of Power BI reports and I am doing that with this script.
(Source) =>
let
res = List.Accumulate(
Text.ToList(Source),
[result="", index=0, source=Source],
fnAccumulator
),
IsUpper = (txt) => txt <> "" and txt = Text.Upper(txt) and txt <> Text.Lower(txt),
fnAccumulator =
(state as record, current as text) as record =>
let
prevCharacter =
if state[index]=0 then
""
else
Text.At(state[source], state[index] - 1),
prevCharacter2 =
if state[index]<=1 then
""
else
Text.At(state[source], state[index] - 2),
nextCharacter =
if state[index] = Text.Length(state[source]) - 1 then
""
else
Text.At(state[source], state[index] + 1),
aggregatedResult =
if state[index]=0 or current = "" then
current
else
if IsUpper(current) and
(not IsUpper(prevCharacter) or
(IsUpper(prevCharacter2) and not IsUpper(nextCharacter))) then
state[result] & " " & current
else
state[result] & current,
resultRecord =
if aggregatedResult = null then
null
else
[result = aggregatedResult, index = state[index]+1, source = state[source]]
in
resultRecord
in
res[result]
The problem is after running this on every table and applying the changes all the relationships in the report end up nuked.
Before
After
My thought process is that because my script isn't taking into account SID's and renaming those when they should stay the same this is breaking the relationships. Would appreciate either an alternative way of mass renaming these columns or if there is a way to add an exclusion statement into my script that will ignore any and all SIDs
Good afternoon all,
I'm trying to call all of the results within an API that has:
6640 total records
100 records per page
67 pages of results (total records / records per page)
This is an ever growing list so I've used variables to create the above values.
I can obviously use the $Skip ODATA expression to get any one of the 67 pages by adding the expression to the end of the URL like so (which would skip the first 100, therefore returning the 2nd page:
https://psa.pulseway.com/api/servicedesk/tickets/?$Skip=100
What I'm trying to do though is to create a custom function that will loop through each of the 67 calls, changing the $Skip value by an increment of 100 each time.
I thought I'd accomplished the goal with the below code:
let
Token = "Token",
BaseURL = "https://psa.pulseway.com/api/",
Path = "servicedesk/tickets/",
RecordsPerPage = 100,
CountTickets = Json.Document(Web.Contents(BaseURL,[Headers = [Authorization="Bearer " & Token],RelativePath = Path & "count"])),
TotalRecords = CountTickets[TotalRecords],
GetJson = (Url) =>
let Options = [Headers=[ #"Authorization" = "Bearer " & Token ]],
RawData = Web.Contents(Url, Options),
Json = Json.Document(RawData)
in Json,
GetPage = (Index) =>
let Skip = "$Skip=" & Text.From(Index * RecordsPerPage),
URL = BaseURL & Path & "?" & Skip,
Json = GetJson(URL)
in Json,
TotalPages = Number.RoundUp(TotalRecords / RecordsPerPage),
PageIndicies = {0.. TotalPages - 1},
Pages = List.Transform(PageIndicies, each GetPage(_))
in
Pages
I got all happy when it successfully made the 67 API calls and combined the results into a list for me to load in to a Power Query table, however what I'm actually seeing is the first 100 records repeated 67 times.
That tells me that my GetPage custom function which handles the $Skip value isn't changing and is stuck on the first one. To make sure the Skip index was generating them properly I duplicated the query and changed the code to load in the $Skip values and see what they are, expecting them all to be $Skip=0, what I see though is the correct $Skip values as below:
Image showing correct Skip values
It seems everything is working as it should be, only I'm only getting the first page 67 times.
I've made a couple of posts on other community site around this issue before but I realise the problem I was (poorly) describing was far too broad to get any meaningful assistance. I think now I've gotten to the point where I understand what my own code is doing and have really zoomed in to the problem - I just don't know how to fix it when I'm at the final hurdle...
Any help/advice would be massively appreciated. Thank you.
Edit: Updated following #RicardoDiaz answer.
let
// Define base parameters
Filter = "",
Path = "servicedesk/tickets/",
URL = "https://psa.pulseway.com/api/",
Token = "Token",
Limit = "100",
// Build the table based on record start and any filters
GetEntityRaw = (Filter as any, RecordStart as text, Path as text) =>
let
Options = [Headers=[ #"Authorization" = "Bearer " & Token ]],
URLbase = URL & Path & "?bearer=" & Token & "&start=" & RecordStart & "&limit=" & Text.From(Limit),
URLentity = if Filter <> null then URLbase & Filter else URLbase,
Source = Json.Document(Web.Contents(URLentity, Options)),
Result = Source[Result],
toTable = Table.FromList(Result, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
in
toTable,
// Recursively call the build table function
GetEntity = (optional RecordStart as text) as table =>
let
result = GetEntityRaw(Filter, RecordStart, Path),
nextStart = Text.From(Number.From(RecordStart) + Limit),
nextTable = Table.Combine({result, #GetEntity(nextStart)}),
check = try nextTable otherwise result
in
check,
resultTable = GetEntity("0")
in
resultTable
As I couldn't test your code, it's kind of hard to provide you a concrete answer.
Said that, please review the generic code I use to connect to an api and see if you can find where yours is not working
EDIT: Changed api_record_limit type to number (removed the quotation marks)
let
// Define base parameters
api_url_filter = "",
api_entity = "servicedesk/tickets/",
api_url = "https://psa.pulseway.com/api/",
api_token = "Token",
api_record_limit = 500,
// Build the table based on record start and any filters
fx_api_get_entity_raw = (api_url_filter as any, api_record_start as text, api_entity as text) =>
let
api_url_base = api_url & api_entity & "?api_token=" & api_token & "&start=" & api_record_start & "&limit=" & Text.From(api_record_limit),
api_url_entity = if api_url_filter <> null then api_url_base & api_url_filter else api_url_base,
Source = Json.Document(Web.Contents(api_url_entity)),
data = Source[data],
toTable = Table.FromList(data, Splitter.SplitByNothing(), null, null, ExtraValues.Error)
in
toTable,
// Recursively call the build table function
fxGetEntity = (optional api_record_start as text) as table =>
let
result = fx_api_get_entity_raw(api_url_filter, api_record_start, api_entity),
nextStart = Text.From(Number.From(api_record_start) + api_record_limit),
nextTable = Table.Combine({result, #fxGetEntity(nextStart)}),
check = try nextTable otherwise result
in
check,
resultTable = fxGetEntity("0"),
expandColumn = Table.ExpandRecordColumn(
resultTable,
"Column1",
Record.FieldNames(resultTable{0}[Column1]),
List.Transform(Record.FieldNames(resultTable{0}[Column1]), each _)
)
in
expandColumn
QUESTION TO OP:
Regarding this line:
Result = Source[Result],
Does the json return a field called result instead of data?
I have a list for example:
A
B
C
D
Now I want to create a query in power query that will create a text like this:
"The letters are: " A B C D
I already have this query:
let
xx= KEYListfunction(),
list = Text.Split(xx, ","),
text = "",
loop = List.Accumulate(list, 0, (state, current) => text = text & " " & current ))
in
loop
But the result only says "FALSE"
Are you looking for this?
let
list = {"A".."D"},
text = "The letters are: " & Text.Combine(list, ", ")
in
text
In Power BI Desktop i have a table from an excel file and i want to split a row based on a division between the value of a specific column and a default number.
In more details lets assume tha we have a table like this :
if the default value we want to devide column Amount is 50,then the desirable result would be something like that :
Do you have any idea how can i implement that in Power query editor or with dax?
Thanks
Tested this in Power Query for Excel, but hopefully should work for you in Power BI too. If you create a function like:
divisionToList = (numberToDivide as number, numberToDivideBy as number) as list =>
let
divisionResult = numberToDivide / numberToDivideBy,
isResultValid = (divisionResult >= 0) and (Number.Mod(divisionResult, 1) = 0),
errorIfInvalid = Error.Record("Cannot create a list with " & Text.From(divisionResult) & " items", Number.ToText(numberToDivide) & " / " & Number.ToText(numberToDivideBy) & " = " & Text.From(divisionResult), null),
listOrError = if isResultValid then List.Repeat({divisionResult}, divisionResult) else error errorIfInvalid
in listOrError,
It should divide two numbers and return a list of length d in which each element is d (d is the result of the division). This list can then, in the context of a table, be expanded into new rows.
There is some basic error handling in the function for cases where the division yields a problematic number (since you can't have a list with, for example, 5.1 elements or -1 elements). You can change/remove this handling if necessary.
I think this code below takes me from your first image to your second image -- and hopefully will give you some idea on how to go about achieving this.
let
mockData = Table.FromColumns({{200, 400}, {"A", "B"}}, type table [Amount = number, Description = text]),
defaultValue = 50, // Not sure what logic is required for arriving at this figure, so have simply assigned it.
divisionToList = (numberToDivide as number, numberToDivideBy as number) as list =>
let
divisionResult = numberToDivide / numberToDivideBy,
isResultValid = (divisionResult >= 0) and (Number.Mod(divisionResult, 1) = 0),
errorIfInvalid = Error.Record("Cannot create a list with " & Text.From(divisionResult) & " items", Number.ToText(numberToDivide) & " / " & Number.ToText(numberToDivideBy) & " = " & Text.From(divisionResult), null),
listOrError = if isResultValid then List.Repeat({divisionResult}, divisionResult) else error errorIfInvalid
in listOrError,
invokeFunction = Table.TransformColumns(mockData, {{"Amount", each divisionToList(_, defaultValue), type list}}),
expanded = Table.ExpandListColumn(invokeFunction, "Amount")
in
expanded
In Siddhi query, I am importing two stream S1 and S2. If I receive in S1 stream I will insert in event table T1, and when I receive in S2 I will update in the T1 table based on the id, and also I will send the updated values from the table into Output stream O1.
As a part of the requirement, I need to get the content which table T1, which is inserted before 5 min(ie, if a record resides more than 5 min) and send to another output stream O2.
#name('S1')
from S1
select id, srcId, 'null' as msgId, 'INP' as status
insert into StatusTable;
#name('S2')
from S2#window.time(1min) as g join StatusTable[t.status == 'INP'] as t
on ( g.srcId == t.id)
select t.id as id, g.msgId as msgId, 'CMP' as status
update StatusTable on TradeStatusTable.id == id;
#name('Publish')
from S2 as g join StatusTable[t.status == 'CMP'] as t on ( g.srcId == t.id and t.status == 'CMP')
select t.id as id, t.msgId as msgId, t.status as status
insert into O1;
How to add a query in this existing query to fetch the records from TradeStatus table, which receides more than 5 minutes. Since the table cannot be used alone, I need to join it with a stream, how to do this scenario?
String WebAttackSuccess = "" +
"#info(name = 'found_host_charged1') "+
"from ATDEventStream[ rid == 10190001 ]#window.timeBatch(10 sec) as a1 "+
"join ATDEventStream[ rid == 10180004 ]#window.time(10 sec) as a2 on a2.src_ip == a1.src_ip and a2.dst_ip == a1.dst_ip " +
" select UUID() as uuid,1007 as cid,a1.sensor_id as sensor_id,a1.interface_id as interface_id,a1.other_id as other_id,count(a1.uuid) as event_num,min(a1.timestamp) as first_seen,max(a2.timestamp) as last_seen,'' as IOC,a1.dst_ip as victim,a1.src_ip as attacker,a1.uuid as NDE4,sample:sample(a2.uuid) as Sample_NDE4 " +
" insert into found_host_charged1;"+
""+
"#info(name = 'found_host_charged2') "+
"from every a1 = found_host_charged1 " +
"-> a2 = ATDEventStream[dns_answers != ''] "+
"within 5 min "+
"select UUID() as uuid,1008 as cid,a2.sensor_id as sensor_id,a2.interface_id as interface_id,a2.other_id as other_id,count(a2.uuid) as event_num,a1.first_seen as first_seen,max(a2.timestamp) as last_seen,a2.dns_answers as IOC,a2.dst_ip as victim,a2.src_ip as attacker,a1.uuid as NDE5,sample:sample(a2.uuid) as Sample_NDE5 " +
"insert into found_host_charged2; ";
This is part of my work,i use two stream,maybe you can get the data from StatusTable in your second stream.If not yet resolved,you can change StatusTable to S1.