This is my column name and value below.
ASSET_NAME - mats&service
Below is the dax code i use to get data from table and create a link to filter the my page, but it wont work when value(mats&service) in column(ASSET_NAME) has & in it, the link gets terminated at & like this
https:/app.powerbi.com/?filter/somerandomtextand%20andthesemats
and the filter wont work, can someone help me escape this & so that value is accepted and filtered in my page,
RawData =
MAX ('Raw data'[Raw Data])&"?filter=Table/ASSET_NAME eq"&" '"
& MAX ( CurrentData[ASSET_NAME] )&"' and Table/LEVEL1 eq"&" '"
& MAX ( CurrentData[LEVEL])&"' and Table/DOS_NO eq"&" ' "
& MAX ( CurrentData[RULE_NO] )&" ' "
The official documentation has a section how to handle special characters in the values. The & character should be replaced with %26, so instead of mats&service value, try with mats%26service.
When concatenating the strings to construct the URL, use SUBSTITUTE DAX function (or multiple nested or separate calls to it) to replace the special characters in the values, e.g. like this:
Measure =
var AssetName1 = MAX(CurrentData[ASSET_NAME])
var AssetName2 = SUBSTITUTE(AssetName1, "%", "%25")
var AssetName3 = SUBSTITUTE(AssetName2, "+", "%2B")
var AssetName4 = SUBSTITUTE(AssetName3, "&", "%26")
RETURN "?filter=Table/ASSET_NAME eq '" & AssetName4 & "'"
Another option is to add a custom column with "safe" values, which are URL encoded using Uri.EscapeDataString M function:
ASSET_NAME_ENCODED = Uri.EscapeDataString([ASSET_NAME])
and use this column when constructing the URL.
I'm running a program to help document what is contained in our 30+year old database. During the course of this process, I am getting the following error message:
Attempted READ of record ID larger than file/table maximum record ID size of 255 characters.
My program is working like this:
LOOP WHILE I <= NUM.FILES
RECORD = ""
FILENAME = FILE.LIST<I>
ERROR = ""
DEBUG.RECORD = ""
HAVE.LOOKED = 0
OPEN 'DICT ':FILENAME TO D.FILE THEN
OPEN FILENAME TO T.FILE THEN
STATEMENT = "SSELECT ONLY DICT ":FILENAME:' BY FIELD.NO WITH FIELD.NO >= 0 AND WITH FIELD.NO <= 900 AND WITH FIELD # ".]"'
DEBUG = ""
PRINT FILENAME
EXECUTE STATEMENT RETURNING DEBUG
LOOP WHILE READNEXT FIELDNAME DO
READ FIELD.RECORD FROM D.FILE, FIELDNAME THEN
IF LEN(FIELDNAME) > BIGGEST.KEY.LEN THEN
BIGGEST.KEY = FIELDNAME
BIGGEST.KEY.LEN = LEN(FIELDNAME)
BIGGEST.KEY.FILE = "DICT ": FILENAME
PRINT FILENAME:" ":LEN(FIELDNAME):" ":FIELDNAME
END
USE.COUNT = ""
USE.LIST = ""
USE.COUNT.STATEMENT = "SELECT ":FILENAME:" WITH ":FIELDNAME:' # ""'
DEBUGS = ""
EXECUTE USE.COUNT.STATEMENT RTNLIST USE.LIST RETURNING DEBUGS
ROW = ""
ROW<1,1> = FIELD.RECORD<2> ; *Attribute Number
ROW<1,2> = FIELDNAME ; *Field Name
ROW<1,3> = FIELD.RECORD<1> ; *Field Type
ROW<1,4> = FIELD.RECORD<10> ; *Field Size
ROW<1,5> = FIELD.RECORD<12> ; *Is Multivalued: "" = no, "Y" = Multivalued, "###" = specific multivalue
ROW<1,6> = FIELD.RECORD<13> ; *Is Subvalued: "" = no, "Y" = Subvalued, "###" = specific subvalue
ROW<1,7> = FIELD.RECORD<7> ; *Automatic data output conversion
ROW<1,8> = FIELD.RECORD<8> ; *Correlative field definition
ROW<1,9> = FIELD.RECORD<11> ; *Field description
ROW<1,10> = #SELECTED ; *Number of records that don't have this field blank
RECORD<-1> = ROW
IF ROW<1,10> < 1 THEN
READ UNUSED.FIELDS FROM CHUCK.WORK, "FILE.DEBUG.UNUSED.FIELDS" ELSE
UNUSED.FIELDS = ""
END
UNUSED.FIELDS<-1> = FILENAME:VM:ROW
WRITE UNUSED.FIELDS ON CHUCK.WORK, "FILE.DEBUG.UNUSED.FIELDS"
END
IF FIELD.RECORD<2> = 0 AND #SELECTED > 0 AND HAVE.LOOKED = 0 THEN
LOOP WHILE READNEXT KEY FROM USE.LIST DO
IF LEN(KEY) > BIGGEST.KEY.LEN THEN
BIGGEST.KEY = KEY
BIGGEST.KEY.LEN = LEN(KEY)
BIGGEST.KEY.FILE = FILENAME
PRINT FILENAME:" ":LEN(KEY):" ":KEY
END
REPEAT
HAVE.LOOKED = 1
END
END
REPEAT
END ELSE
ERROR<-1> = "Failed to open file '":FILENAME:"'"
END
END ELSE
ERROR<-1> = "Failed to open file DICT '":FILENAME:"'"
END
WRITE RECORD ON CHUCK.WORK, "FILE.":FILENAME
WRITE DEBUG.RECORD ON CHUCK.WORK, "FILE.DEBUG.":FILENAME
READ CHUCK.LOG FROM CHUCK.WORK, "CHUCK.LOG" ELSE
CHUCK.LOG = ""
END
CHUCK.LOG<-1> = "FILE '":FILENAME:"' had ":DCOUNT(RECORD,AM):" fields"
IF ERROR THEN
CHUCK.LOG<-1> = ERROR
ERRORS<-1> = ERROR
END
WRITE CHUCK.LOG ON CHUCK.WORK,"CHUCK.LOG"
CLEARSELECT
I = I + 1
REPEAT
When I look at the database directly, I can't find any record IDs or keys with more than 35 characters in the file which is causing problems, and nothing longer than 70 characters in the entire database. Can anyone help identify why these records are getting flagged in this process but aren't discoverable directly?
Below is a program I wrote to specifically find the problematic records, but it can't find the culprit
OPEN "CHUCK.WORK" TO CHUCK.WORK ELSE
PRINT "UNABLE TO OPEN CHUCK.WORK"
RETURN
END
READ FILENAME FROM CHUCK.WORK, "LISTME" ELSE
PRINT "UNABLE TO READ LISTME"
RETURN
END
NUM.FILES = DCOUNT(FILENAME,AM)
FOR I = 1 TO NUM.FILES
OPEN FILENAME<I> TO T.FILE ELSE
PRINT "UNABLE TO OPEN ":FILENAME<I>
RETURN
END
EXECUTE 'SELECT ':FILENAME<I>
LOOP WHILE READNEXT KEY DO
IF LEN(KEY) > 20 THEN
PRINT FILENAME<I>:" ":LEN(KEY):" ":KEY
END
REPEAT
NEXT I
UPDATE
One of my coworkers identified the source of the problem, even though we haven't identified how to fix the problem:
one of our files has a multivalued field which is a key used in a correlative. For some reason, Universe is trying to read the entire attribute instead of the individual multivalue as the key, which causes the long record IDs. Anyone able to see whether I am doing something wrong in my code or if there is some setting in the database that we need to look at?
When you see this error it has nothing to do with the size of keys in file, it is simply that the #ID you are trying to READ is longer than 255 chars. When I have seen it usually show me what line in the source code it happened on. If you put this right before you that line you should be able to track it down.
IF LEN(THIS.ID) GT 255 THEN
DEBUG
END
Edit. Apparently the error in this case is does not reference a line number. I was not sure if this was omitted for clarity or was some difference in the UniVerse flavor, but I now believe its absence is a hint that the error message is coming from the shell and not that the interpreter.
OPEN '','VOC' TO FILE.VOC ELSE STOP "CANNOT OPEN FILE VOC"
STMT = "SELECT VAL WITH ":STR("A",256):" EQ 0"
EXECUTE STMT RTNLIST USE.LIST RETURNING DEBUGS
CRT "**************************************"
READ TEST FROM FILE.VOC,STR("A",256) ELSE NULL
END
Which on my system outputs this.
>RUN TEST.SC TEST.LONG.ID
Attempted READ of record ID larger than file/table maximum
record ID size of 255 characters.
RetrieVe: syntax error. Unexpected sentence without filename. Token was "".
Scanned command was SELECT 'VAL' WITH 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' EQ '0'
**************************************
Program "TEST.LONG.ID": Line 5, Attempted READ of record ID larger than file/t
able maximum
record ID size of 255 characters.
>
The first looks like your error message and would point to one of the dynamic SELECT statements you are building. #ID is synonymous with with record key and to carry the analogy a little further, it appears that you are trying to unlock your bike with one of those comically large "Keys to the City".
I have a native Oracle query running in an Excel workbook, and I'm passing the user-supplied values from a table into the queries WHERE clause.
I wrote a quick function in M that I think adds single quotes to a string that's passed in
(x) =>
let
string_format = "'" & x & "'"
in
string_format
I apply this function to a column and then transform the column to a list but any strings with embedded commas are surrounded by double quotes
text_tbl3 = Table.TransformColumns(text_tbl2,{{"Org_Names", string_format}})
text_ls = Table.ToList(text_tbl3)
It's difficult to see, but TD,AMERITRADE is surrounded by double and single quotes like this : "'TD, AMERITRADE'". I want it to read 'TD,AMERITRADE' , so it has the same formatting as the other cells, but I cannot figure out what causes the additional double quotes.
Quoting text
quick function in M that I think adds single quotes to a string that's passed in
Your function is correct. & is the text concatenation operator.
Because you're using a single expression, you could simplify it by removing the inner let..in expression. (If you don't open the advanced editor you won't see the outer let..in expression).
quote_text = (string as text) => "'" & string & "'"
Note: Your screenshot has extra quotes
Your inputs were:
CHASE
CITI
"TD, AMERITRADE"
Which is why you end up with:
'CHASE'
'CITI'
'"TD, AMERITRADE"'
Your cell probably has quotes on "TD, AMERITRADE" but not on the others.
Getting a comma separated list as a single string
Text.Combine(list, separator=", ") will create a string like a CSV file.
let
list_names = table3[Company],
// equivalent to: list_names = {"CHASE", "CITI", "TD, AMERITRADE"},
without_quotes = Text.Combine(list_names, ", "),
list_quoted = List.Transform(
list_names,
quote_text
),
with_quotes = Text.Combine(list_quoted, ", "),
results = [
list_names = list_names,
list_quoted = list_quoted,
string_without_quotes = without_quotes,
string_with_quotes = with_quotes,
without_equal_to = "string = ""CHASE, CITI, TD, AMERITRADE""",
with_equal_to = "string = ""'CHASE', 'CITI', 'TD, AMERITRADE'"""
]
in
results
How do we use that string in a native query?
My query uses SQL but the method is the same for Oracle.
raw_sql_query is your raw query. It uses the parameter #Company
sql_parameters is a Record type that declares all parameters you are using. Here we use your string with the qoute_text function.
Value.NativeQuery inserts the parameters for you.
let
company = "TD, AMERITRADE",
raw_sql_query = "
select * from Table
where Company = #Company
",
sql_parameters = [
Company = quote_text( company )
],
source = Sql.Database(
"localhost",
"Adventure Works"
),
results = Value.NativeQuery(
source,
raw_sql_query,
sql_parameters
)
in
results
How do we test whether the string function is quoting correctly?
First create a new blank query. We call quote_text() to verify the output.
I used a Record named results so you can label and view every value on a single screen.
manual_quote uses the string concatenation operator to quote strings
quote_string( sample_string ) inserts variables into a text template. Both return the exact same string.
Text.Format becomes cleaner the more complicated your template becomes. This function is simple enough it's not necessary.
Your original function
This is what your function in the advanced editor looks like:
let
quote_text = (x) =>
let
string_format = "'" & x & "'"
in
string_format
in
quote_text
You may remove the inner let
let
quote_text_simple = (string as text) =>
"'" & string & "'"
in
quote_text_simple
How you can use optional arguments and string Templates
let
// a custom function to Surround a string with single quotes.
// A optional second argument lets you specify a different character
quote_string = (input_string as text, optional character as text) =>
let
character = if character = null then "'" else character,
template = "#[quote]#[string]#[quote]",
quoted_string = Text.Format(
template,
[
quote = character,
string = input_string
]
)
in
quoted_string
in
quote_string
I would like to send my users directly to a specific log group and filter but I need to be able to generate the proper URL format. For example, this URL
https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/
%252Fmy%252Flog%252Fgroup%252Fgoes%252Fhere/log-events/$3FfilterPattern$3D$255Bincoming_ip$252C$2Buser_name$252C$2Buser_ip$2B$252C$2Btimestamp$252C$2Brequest$2B$2521$253D$2B$2522GET$2B$252Fhealth_checks$252Fall$2B*$2522$252C$2Bstatus_code$2B$253D$2B5*$2B$257C$257C$2Bstatus_code$2B$253D$2B429$252C$2Bbytes$252C$2Burl$252C$2Buser_agent$255D$26start$3D-172800000
will take you to a log group named /my/log/group/goes/here and filter messages with this pattern for the past 2 days:
[incoming_ip, user_name, user_ip , timestamp, request != "GET /health_checks/all *", status_code = 5* || status_code = 429, bytes, url, user_agent]
I can decode part of the URL but I don't know what some of the other characters should be (see below), but this doesn't really look like any standard HTML encoding to me. Does anyone know a encoder/decoder for this URL format?
%252F == /
$252C == ,
$255B == [
$255D == ]
$253D == =
$2521 == !
$2522 == "
$252F == _
$257C == |
$2B == +
$26 == &
$3D == =
$3F == ?
First of all I'd like to thank other guys for the clues. Further goes the complete explanation how Log Insights links are constructed.
Overall it's just weirdly encoded conjunction of an object structure that works like that:
Part after ?queryDetail= is object representation and {} are represented by ~()
Object is walked down to primitive values and the latter are transformed as following:
encodeURIComponent(value) so that all special characters are transformed to %xx
replace(/%/g, "*") so that this encoding is not affected by top level ones
if value type is string - it is prefixed with unmatched single quote
To illustrate:
"Hello world" -> "Hello%20world" -> "Hello*20world" -> "'Hello*20world"
Arrays of transformed primitives are joined using ~ and as well put inside ~() construct
Then, after primitives transformation is done - object is joined using "~".
After that string is escape()d (note that not encodeURIComponent() is called as it doesn't transform ~ in JS).
After that ?queryDetail= is added.
And finally this string us encodeURIComponent()ed and as a cherry on top - % is replaced with $.
Let's see how it works in practice. Say these are our query parameters:
const expression = `fields #timestamp, #message
| filter #message not like 'example'
| sort #timestamp asc
| limit 100`;
const logGroups = ["/application/sample1", "/application/sample2"];
const queryParameters = {
end: 0,
start: -3600,
timeType: "RELATIVE",
unit: "seconds",
editorString: expression,
isLiveTrail: false,
source: logGroups,
};
Firstly primitives are transformed:
const expression = "'fields*20*40timestamp*2C*20*40message*0A*20*20*20*20*7C*20filter*20*40message*20not*20like*20'example'*0A*20*20*20*20*7C*20sort*20*40timestamp*20asc*0A*20*20*20*20*7C*20limit*20100";
const logGroups = ["'*2Fapplication*2Fsample1", "'*2Fapplication*2Fsample2"];
const queryParameters = {
end: 0,
start: -3600,
timeType: "'RELATIVE",
unit: "'seconds",
editorString: expression,
isLiveTrail: false,
source: logGroups,
};
Then, object is joined using ~ so we have object representation string:
const objectString = "~(end~0~start~-3600~timeType~'RELATIVE~unit~'seconds~editorString~'fields*20*40timestamp*2C*20*40message*0A*20*20*20*20*7C*20filter*20*40message*20not*20like*20'example'*0A*20*20*20*20*7C*20sort*20*40timestamp*20asc*0A*20*20*20*20*7C*20limit*20100~isLiveTrail~false~source~(~'*2Fapplication*2Fsample1~'*2Fapplication*2Fsample2))"
Now we escape() it:
const escapedObject = "%7E%28end%7E0%7Estart%7E-3600%7EtimeType%7E%27RELATIVE%7Eunit%7E%27seconds%7EeditorString%7E%27fields*20*40timestamp*2C*20*40message*0A*20*20*20*20*7C*20filter*20*40message*20not*20like*20%27example%27*0A*20*20*20*20*7C*20sort*20*40timestamp*20asc*0A*20*20*20*20*7C*20limit*20100%7EisLiveTrail%7Efalse%7Esource%7E%28%7E%27*2Fapplication*2Fsample1%7E%27*2Fapplication*2Fsample2%29%29"
Now we append ?queryDetail= prefix:
const withQueryDetail = "?queryDetail=%7E%28end%7E0%7Estart%7E-3600%7EtimeType%7E%27RELATIVE%7Eunit%7E%27seconds%7EeditorString%7E%27fields*20*40timestamp*2C*20*40message*0A*20*20*20*20*7C*20filter*20*40message*20not*20like*20%27example%27*0A*20*20*20*20*7C*20sort*20*40timestamp*20asc*0A*20*20*20*20*7C*20limit*20100%7EisLiveTrail%7Efalse%7Esource%7E%28%7E%27*2Fapplication*2Fsample1%7E%27*2Fapplication*2Fsample2%29%29"
Finally we URLencode it and replace % with $ and vois la:
const result = "$3FqueryDetail$3D$257E$2528end$257E0$257Estart$257E-3600$257EtimeType$257E$2527RELATIVE$257Eunit$257E$2527seconds$257EeditorString$257E$2527fields*20*40timestamp*2C*20*40message*0A*20*20*20*20*7C*20filter*20*40message*20not*20like*20$2527example$2527*0A*20*20*20*20*7C*20sort*20*40timestamp*20asc*0A*20*20*20*20*7C*20limit*20100$257EisLiveTrail$257Efalse$257Esource$257E$2528$257E$2527*2Fapplication*2Fsample1$257E$2527*2Fapplication*2Fsample2$2529$2529"
And putting it all together:
function getInsightsUrl(queryDefinitionId, start, end, expression, sourceGroup, timeType = 'ABSOLUTE', region = 'eu-west-1') {
const p = m => escape(m);
const s = m => escape(m).replace(/%/gi, '*');
const queryDetail
= p('~(')
+ p("end~'")
+ s(end.toUTC().toISO()) // converted using Luxon
+ p("~start~'")
+ s(start.toUTC().toISO()) // converted using Luxon
// Or use UTC instead of Local
+ p(`~timeType~'${timeType}~tz~'Local~editorString~'`)
+ s(expression)
+ p('~isLiveTail~false~queryId~\'')
+ s(queryDefinitionId)
+ p("~source~(~'") + s(sourceGroup) + p(')')
+ p(')');
return `https://${region}.console.aws.amazon.com/cloudwatch/home?region=${region}#logsV2:logs-insights${escape(`?queryDetail=${queryDetail}`).replace(/%/gi, '$')}`;
}
Of course reverse operation can be performed as well.
That's all folks. Have fun, take care and try to avoid doing such a weird stuff yourselves. :)
I had to do a similar thing to generate a back link to the logs for a lambda and did the following hackish thing to create the link:
const link = `https://${process.env.AWS_REGION}.console.aws.amazon.com/cloudwatch/home?region=${process.env.AWS_REGION}#logsV2:log-groups/log-group/${process.env.AWS_LAMBDA_LOG_GROUP_NAME.replace(/\//g, '$252F')}/log-events/${process.env.AWS_LAMBDA_LOG_STREAM_NAME.replace('$', '$2524').replace('[', '$255B').replace(']', '$255D').replace(/\//g, '$252F')}`
A colleague of mine figured out that the encoding is nothing special. It is the standard URI percent encoding but applied twice (2x). In javascript you can use the encodeURIComponent function to test this out:
let inp = 'https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/'
console.log(encodeURIComponent(inp))
console.log(encodeURIComponent(encodeURIComponent(inp)))
This piece of javascript produces the expected output on the second encoding stage:
https%3A%2F%2Fconsole.aws.amazon.com%2Fcloudwatch%2Fhome%3Fregion%3Dus-east-1%23logsV2%3Alog-groups%2Flog-group%2F
https%253A%252F%252Fconsole.aws.amazon.com%252Fcloudwatch%252Fhome%253Fregion%253Dus-east-1%2523logsV2%253Alog-groups%252Flog-group%252F
Caution
At least some bits use the double encoding, not the whole link though. Otherwise all special characters would occupy 4 characters after double encoding, but some still occupy only 2 characters. Hope this helps anyway ;)
My complete Javascript solution based on #isaias-b answer, which also adds a timestamp filter on the logs:
const logBaseUrl = 'https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group';
const encode = text => encodeURIComponent(text).replace(/%/g, '$');
const awsEncode = text => encodeURIComponent(encodeURIComponent(text)).replace(/%/g, '$');
const encodeTimestamp = timestamp => encode('?start=') + awsEncode(new Date(timestamp).toJSON());
const awsLambdaLogBaseUrl = `${logBaseUrl}/${awsEncode('/aws/lambda/')}`;
const logStreamUrl = (logGroup, logStream, timestamp) =>
`${awsLambdaLogBaseUrl}${logGroup}/log-events/${awsEncode(logStream)}${timestamp ? encodeTimestamp(timestamp) : ''}`;
I have created a bit of Ruby code that seems to satisfy the CloudWatch URL parser. I'm not sure why you have to double escape some things and then replace % with $ in others. I'm guessing there is some reason behind it but I couldn't figure out a nice way to do it, so I'm just brute forcing it. If you have something better, or know why they do this, please add a comment.
NOTE: The filter I tested with is kinda basic and I'm not sure what might need to change if you get really fancy with it.
# Basic URL that is the same across all requests
url = 'https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/'
# CloudWatch log group
log_group = '/aws/my/log/group'
# Either specify the instance you want to search or leave it out to search all instances
instance = '/log-events/i-xxxxxxxxxxxx'
OR
instance = '/log-events'
# The filter to apply.
filter = '[incoming_ip, user_name, user_ip , timestamp, request, status_code = 5*, bytes, url, user_agent]'
# Start time. There might be an End time as well but my queries haven't used
# that yet so I'm not sure how it's formatted. It should be pretty similar
# though.
hours = 48
start = "&start=-#{hours*60*60*1000}"
# This will get you the final URL
final = url + CGI.escape(CGI.escape(log_group)) + instance + '$3FfilterPattern$3D' + CGI.escape(CGI.escape(filter)).gsub('%','$') + CGI.escape(start).gsub('%','$')
A bit late but here is a python implementation
def get_cloud_watch_search_url(search, log_group, log_stream, region=None,):
"""Return a properly formatted url string for search cloud watch logs
search = "{$.message: "You are amazing"}
log_group = Is the group of message you want to search
log_stream = The stream of logs to search
"""
url = f'https://{region}.console.aws.amazon.com/cloudwatch/home?region={region}'
def aws_encode(value):
"""The heart of this is that AWS likes to quote things twice with some substitution"""
value = urllib.parse.quote_plus(value)
value = re.sub(r"\+", " ", value)
return re.sub(r"%", "$", urllib.parse.quote_plus(value))
bookmark = '#logsV2:log-groups'
bookmark += '/log-group/' + aws_encode(log_group)
bookmark += "/log-events/" + log_stream
bookmark += re.sub(r"%", "$", urllib.parse.quote("?filterPattern="))
bookmark += aws_encode(search)
return url + bookmark
This then allows you to quickly verify it.
>>> real = 'https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logsV2:log-groups/log-group/$252Fapp$252Fdjango/log-events/production$3FfilterPattern$3D$257B$2524.msg$253D$2522$2525s$2525s+messages+to+$2525s+pk$253D$2525d...$2522$257D'
>>> constructed = get_cloud_watch_search_url(None, search='{$.msg="%s%s messages to %s pk=%d..."}', log_group='/app/django', log_stream='production', region='us-west-2')
>>> real == constructed
True
I encountered this problem recently when I wanted to generate cloudwatch insights URL. Typescript version below:
export function getInsightsUrl(
start: Date,
end: Date,
query: string,
sourceGroup: string,
region = "us-east-1"
) {
const p = (m: string) => escape(m);
// encodes inner values
const s = (m: string) => escape(m).replace(/\%/gi, "*");
const queryDetail =
p(`~(end~'`) +
s(end.toISOString()) +
p(`~start~'`) +
s(start.toISOString()) +
p(`~timeType~'ABSOLUTE~tz~'UTC~editorString~'`) +
s(query) +
p(`~isLiveTail~false~queryId~'`) +
s(v4()) +
p(`~source~(~'`) +
s(sourceGroup) +
p(`))`);
return (
`https://console.aws.amazon.com/cloudwatch/home?region=${region}#logsV2:logs-insights` +
escape("?queryDetail=" + queryDetail).replace(/\%/gi, "$")
);
}
Github GIST
A Python solution based on #Pål Brattberg's answer:
cloudwatch_log_template = "https://{AWS_REGION}.console.aws.amazon.com/cloudwatch/home?region={AWS_REGION}#logsV2:log-groups/log-group/{LOG_GROUP_NAME}/log-events/{LOG_STREAM_NAME}"
log_url = cloudwatch_log_template.format(
AWS_REGION=AWS_REGION, LOG_GROUP_NAME=CLOUDWATCH_LOG_GROUP, LOG_STREAM_NAME=LOG_STREAM_NAME
)
Make sure to substitute illegal characters first (see OP) if you used any.
I encountered this problem recently when I wanted to generate cloudwatch insights URL. PHP version below:
<?php
function getInsightsUrl($region = 'ap-northeast-1') {
// https://stackoverflow.com/questions/67734825/why-is-laravels-carbon-toisostring-different-from-javascripts-toisostring
$start = now()->subMinutes(2)->format('Y-m-d\TH:i:s.v\Z');
$end = now()->addMinutes(2)->format('Y-m-d\TH:i:s.v\Z');
$filter = 'INFO';
$logStream = 'xxx_backend_web';
$sourceGroup = '/ecs/xxx_backend_prod';
// $sourceGroup = '/aws/ecs/xxx_backend~\'/ecs/xxx_backend_dev'; // multiple source group
$query =
"fields #timestamp, #message \n" .
"| sort #timestamp desc\n" .
"| filter #logStream like '$logStream'\n" .
"| filter #message like '$filter'\n" .
"| limit 20";
$queryDetail = urlencode(
("~(end~'") .
($end) .
("~start~'") .
($start) .
("~timeType~'ABSOLUTE~tz~'Local~editorString~'") .
($query) .
("~isLiveTail~false~queryId~'") .
("~source~(~'") .
($sourceGroup) .
("))")
);
$queryDetail = preg_replace('/\%/', '$', urlencode("?queryDetail=" . $queryDetail));
return
"https://console.aws.amazon.com/cloudwatch/home?region=${region}#logsV2:logs-insights"
. $queryDetail;
}
A coworker came up with the following JavaScript solution.
import JSURL from 'jsurl';
const QUERY = {
end: 0,
start: -3600,
timeType: 'RELATIVE',
unit: 'seconds',
editorString: "fields #timestamp, #message, #logStream, #log\n| sort #timestamp desc\n| limit 200\n| stats count() by bin(30s)",
source: ['/aws/lambda/simpleFn'],
};
function toLogsUrl(query) {
return `#logsV2:logs-insights?queryDetail=${JSURL.stringify(query)}`;
}
toLogsUrl(QUERY);
// #logsV2:logs-insights?queryDetail=~(end~0~start~-3600~timeType~'RELATIVE~unit~'seconds~editorString~'fields*20*40timestamp*2c*20*40message*2c*20*40logStream*2c*20*40log*0a*7c*20sort*20*40timestamp*20desc*0a*7c*20limit*20200*0a*7c*20stats*20count*28*29*20by*20bin*2830s*29~source~(~'*2faws*2flambda*2fsimpleFn))
I HAVE to elevate #WayneB's answer above bc it just works. No encoding required - just follow his template. I just confirmed it works for me. Here's what he said in one of the comments above:
"Apparently there is an easier link which does the encoding/replacement for you: https://console.aws.amazon.com/cloudwatch/home?region=${process.env.AWS_REGION}#logEventViewer:group=${logGroup};stream=${logStream}"
Thanks for this answer Wayne - just wish I saw it sooner!
Since Python contributions relate to log-groups, and not to log-insights, this is my contribution. I guess that I could have done better with the inner functions though, but it is a good starting point:
from datetime import datetime, timedelta
import re
from urllib.parse import quote
def get_aws_cloudwatch_log_insights(query_parameters, aws_region):
def quote_string(input_str):
return f"""{quote(input_str, safe="~()'*").replace('%', '*')}"""
def quote_list(input_list):
quoted_list = ""
for item in input_list:
if isinstance(item, str):
item = f"'{item}"
quoted_list += f"~{item}"
return f"({quoted_list})"
params = []
for key, value in query_parameters.items():
if key == "editorString":
value = "'" + quote(value)
value = value.replace('%', '*')
elif isinstance(value, str):
value = "'" + value
if isinstance(value, bool):
value = str(value).lower()
elif isinstance(value, list):
value = quote_list(value)
params += [key, str(value)]
object_string = quote_string("~(" + "~".join(params) + ")")
scaped_object = quote(object_string, safe="*").replace("~", "%7E")
with_query_detail = "?queryDetail=" + scaped_object
result = quote(with_query_detail, safe="*").replace("%", "$")
final_url = f"https://{aws_region}.console.aws.amazon.com/cloudwatch/home?region={aws_region}#logsV2:logs-insights{result}"
return final_url
Example:
aws_region = "eu-west-1"
query = """fields #timestamp, #message
| filter #message not like 'example'
| sort #timestamp asc
| limit 100"""
log_groups = ["/application/sample1", "/application/sample2"]
query_parameters = {
"end": datetime.utcnow().isoformat(timespec='milliseconds') + "Z",
"start": (datetime.utcnow() - timedelta(days=2)).isoformat(timespec='milliseconds') + "Z",
"timeType": "ABSOLUTE",
"unit": "seconds",
"editorString": query,
"isLiveTrail": False,
"source": log_groups,
}
print(get_aws_cloudwatch_log_insights(query_parameters, aws_region))
Yet another Python solution:
from urllib.parse import quote
def aws_quote(s):
return quote(quote(s, safe="")).replace("%", "$")
def aws_cloudwatch_url(region, log_group, log_stream):
return "/".join([
f"https://{region}.console.aws.amazon.com/cloudwatch/home?region={region}#logsV2:log-groups",
"log-group",
aws_quote(log_group),
"log-events",
aws_quote(log_stream),
])
aws_cloudwatch_url("ap-southeast-2", "/var/log/syslog", "process/pid=1")
https://ap-southeast-2.console.aws.amazon.com/cloudwatch/home?region=ap-southeast-2#logsV2:log-groups/log-group/$252Fvar$252Flog$252Fsyslog/log-events/process$252Fpid$253D1
I have this little component in ColdFusion 9:
component
displayname = "My Component"
accessors = "true"
{
property
name = "myProperty"
type = "string"
validate = "regex"
validateparams = "{ pattern = '(Eats)|(Shoots)|(Leaves)' }";
}
which works as expected:
<cfscript>
myComponentInstance = new myComponent();
myComponentInstance.setMyProperty('Eats');
// Property is correctly set
myComponentInstance.setMyProperty('Shoots');
// Property is correctly set
myComponentInstance.setMyProperty('Drinks');
// Error: The value does not match the regular expression pattern provided.
</cfscript>
But if I modify the validation regex to allow a value like with a comma (,) in it
validateparams = "{ pattern = '(Eats)|(Shoots)|(Leaves)|(Eats, Shoots & Leaves)' }"
then I get an error on the instance creation
<cfscript>
myComponentInstance = new myComponent();
/* Error while parsing the validateparam
'{ pattern = '(Eats)|(Shoots)|(Leaves)|(Eats, Shoots & Leaves)' }'
for property myProperty */
</cfscript>
It seems like ColdFusion can't process a regular expression with a comma, nor have I found a way of escaping it.
If I try to use a backslash (\), as a regex escaping character, it is then processed as a foreslash (/) by ColdFusion:
validateparams = "{ pattern = '(Eats)|(Shoots)|(Leaves)|(Eats\, Shoots & Leaves)' }"
<cfscript>
myComponentInstance = new myComponent();
/* Error while parsing the validateparam
'{ pattern = '(Eats)|(Shoots)|(Leaves)|(Eats/, Shoots & Leaves)' }'
for property myProperty */
</cfscript>
Other forms of escaping that I have tried, but to no avail, are:
validateparams = "{ pattern = '(Eats)|(Shoots)|(Leaves)|(Eats#chr(44)# Shoots & Leaves)' }"
validateparams = "{ pattern = '(Eats)|(Shoots)|(Leaves)|(Eats,, Shoots & Leaves)' }"
It's a bug in ColdFusion. Raise it as such: https://bugbase.adobe.com/. I can replicate it in CF 9.0.1. I'm working on a work-around... will get back to you if I come up with something.
NB: one can pare the repro validateparams string down to this: {pattern = ","}. I'm guessing Adobe are using the comma as a delim, and it never occurred to them it might be data (they're a bit like that with delimited strings).