Here is a simple MOF Model to Text-script:
[comment encoding = UTF-8 /]
[module test('http://www.eclipse.org/uml2/2.1.0/UML')/]
[template public test(element : Model)]
[comment #main /]
[file ('test.txt', false, 'UTF-8')]
start
[loop(element)/]
[loop(element)/]
end
[/file]
[/template]
[template public loop(element : Model)]
[for (var : Integer | Sequence{1..3})]
[var/]
[/for]
[/template]
It generates the following text:
start
1
2
3
1
2
3
end
How to remove extra new lines after 3? Thanks!
You can use separator in the for:
[template public loop(element : Model)]
[for (var : Integer | Sequence{1..3}) separator('\n')]
[var/][/for]
[/template]
or you can trim the result of template "loop":
[template public loop(element : Model) post(trim())]
[for (var : Integer | Sequence{1..3})]
[var/]
[/for]
[/template]
Does post(trim()) help you ?
Some examples I use:
[template public bodyOperation(g : Getter) ? post (trim())]
[template public typeKind(t : Type) post (replaceAll('\n', '').trim())]
It seems that I've found a hack:
[template public loop(element : Model)]
[for (var : Integer | Sequence{1..3})]
[if i>1]
[/if][var/][/for]
[/template]
But it's a little bit ugly...
Related
I am using AWS Athena to extract some statistics from CloudWatch logs. However, attempting to use the Presto regexp_extract() is generating empty result sets even though the rexexp looks good according to online regexp testers.
The source cloudwatch log sample is as follows:
2021-10-04 00:10:56.201 INFO 10711 --- [io-5000-exec-31] au.com.crecy.VP4CStatistics : {"atlassianLicense" : {"key" : "visio-publisher-for-confluence","version" : "1.1.5-AC","state" : "ENABLED","installedDate" : 1619695028000,"lastUpdated" : 1632692975000,"license" : {"active" : true,"type" : "COMMERCIAL","evaluation" : false,"supportEntitlementNumber" : "SEN-0123456789"},"valid" : true,"host" : {"product" : "Confluence","contacts" : [ ]},"links" : {"marketplace" : [{"href" : "https://marketplace.atlassian.com/plugins/visio-publisher-for-confluence"}],"self" : [{"href" : "https://acme.atlassian.net/wiki/rest/atlassian-connect/1/addons/visio-publisher-for-confluence"}]}},"viewAttachments" : [{"height" : "1000","width" : "100%","scrolling" : "no","frameBorder" : "hide","url" : "/download/attachments/574160906/Foo.html.zip?version=22&modificationDate=1632311039065&cacheVersion=1&api=v2","space" : "VM","page" : 574160906,"id" : "att568885320","frameBorderStyle" : "border:none;"}],"durations" : {"1" : {"method" : "ModelGenAtlassianConnectPlugin.loadHtmlAttachment","startTime" : 2542004145837271,"endTime" : 2542005346331840,"durationMillis" : 1200,"durationNanos" : 1200494569},"2" : {"method" : "AtlassianHostRestClientsHelper.getLicense","startTime" : 2542004145845740,"endTime" : 2542004523777555,"durationMillis" : 377,"durationNanos" : 377931815},"3" : {"method" : "AtlassianHostRestClientsHelper.processJwt","startTime" : 2542004523813282,"endTime" : 2542004525757229,"durationMillis" : 1,"durationNanos" : 1943947},"4" : {"method" : "AttachmentLoaderHelper.loadAttachment","startTime" : 2542004525774026,"endTime" : 2542005346321184,"durationMillis" : 820,"durationNanos" : 820547158},"5" : {"method" : "AtlassianHostRestClientsHelper.getAttachments","startTime" : 2542004525784513,"endTime" : 2542004796450920,"durationMillis" : 270,"durationNanos" : 270666407},"6" : {"method" : "AtlassianHostRestClientsHelper.getCompressedPageSource","startTime" : 2542004796503557,"endTime" : 2542005341655641,"durationMillis" : 545,"durationNanos" : 545152084},"7" : {"method" : "ChecksumHelper.checksumValid","startTime" : 2542005341695482,"endTime" : 2542005341889382,"durationMillis" : 0,"durationNanos" : 193900},"8" : {"method" : "UnzipCompressionHelper.unzipCompressedPageSource","startTime" : 2542005341899585,"endTime" : 2542005346303984,"durationMillis" : 4,"durationNanos" : 4404399},"9" : {"method" : "VP4CResponseHeaderFilter.doFilter","startTime" : 2542008074147431,"endTime" : 2542008074514454,"durationMillis" : 0,"durationNanos" : 367023}},"uncompressedSize" : 520631,"compressedSize" : 48836}
the AWS Athena / presto query is as follows:
select regexp_extract(message, '(au.com.crecy.VP4CStatistics : )({.*}$)', 2)
FROM "VP4C_Statistics_Catalog"."/aws/elasticbeanstalk/vp4c-prod/var/log/web.stdout.log"."all_log_streams"
where message LIKE '%visio-publisher-for-confluence%'
order by time desc
In short I want to extract the JSON payload at the end of the log message. The above query is generating empty result sets.
Thanks and regards,
Andrew
Note that {, }, and . are regex metacharacters, and probably need to be escaped via backslash.
SELECT REGEXP_EXTRACT(message, 'au\.com\.crecy\.VP4CStatistics : (\{.*\})$', 1)
FROM "VP4C_Statistics_Catalog"."/aws/elasticbeanstalk/vp4c-prod/var/log/web.stdout.log"."all_log_streams"
WHERE message LIKE '%visio-publisher-for-confluence%'
ORDER BY time DESC;
Ok, figured it out - the regexp wants a match for multiple spaces or tabs (even though the sample log appears to have a single space. The following pattern works:
select regexp_extract(message, '(au\.com\.crecy\.VP4CStatistics[ \t]*:[ \t]*)(\{.*\})', 2)
FROM "VP4C_Statistics_Catalog"."/aws/elasticbeanstalk/vp4c-prod/var/log/web.stdout.log"."all_log_streams"
where message LIKE '%visio-publisher-for-confluence%'
order by time desc
I have a String, which I try to parse into a array of PSCustom Object with sub expression.
The String looks like this :
date=2021-09-13 time=20:05:25 devname="chwitrfg01" devid="FG10E0TB20903187" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1631556325 srcip=192.168.10.226 srcname="192.168.10.226" srcport=54809 srcintf="port8" srcintfrole="dmz" dstip=8.8.4.4 dstname="dns.google" dstport=53 dstintf="wan1" dstintfrole="lan" poluuid="01533038-da7b-51eb-b854-8fd38a0deba3" sessionid=1472996904 proto=17 action="accept" policyid=278 policytype="policy" service="DNS" dstcountry="United States" srccountry="Reserved" trandisp="snat" transip=194.56.218.226 transport=54809 duration=180 sentbyte=245 rcvdbyte=144 sentpkt=2 rcvdpkt=1 shapersentname="default_class" shaperdropsentbyte=0 shaperrcvdname="default_class" shaperdroprcvdbyte=0 appcat="unscanned" dstdevtype="Unknown" dstdevcategory="None" masterdstmac="00:00:0c:07:ac:8d" dstmac="00:00:0c:07:ac:8d" dstserver=1
And I tried something like this, but I'm a total noob in regex and have no Idea how to solve this. Is there a easy way, to add each value to a property of the custom object?
$Pattern = #(
'(?<devname>\devname=w+)'
'(?<srcip>(srcip=?:[0-9]+\.){3}[0-9]+):(?<srcport>srcport=[0-9]+)'
'(?<dstip>(dstip=?:[0-9]+\.){3}[0-9]+):(?<dstport>dstport=[0-9]+)'
) -join '\s+'
$cmd |
ForEach-Object {
if ($_ -match $Pattern) {
$Matches.Remove(0)
[PsCustomObject]#{
srcip = $_.Groups['srcip'].Value
dstip = $_.Groups['dstip'].Value
dstport = $_.Groups['dstport'].Value
srcport = $_.Groups['srcport'].Value
fw = $_.Groups['devname'].Value
}
}
}| Select-Object -First 5
$cmd | Format-Table
The simplest way to do this that I know of us the ConvertFrom-StringData cmdlet. That cmdlet creates a hashtable of name/value pairs out of a set of name=value formatted things. What you would do is put each value on its own line to make a multi-ling string, then create a new custom object, and use that hashtable to define the properties.
$cmd -replace ' (\w+=)',"`n`$1"|
%{new-object psobject -prop (ConvertFrom-StringData $_)}
Or the shorter version in v3+ (thanks to #mklement0):
$cmd -replace ' (\w+=)',"`n`$1"|
%{[pscustomobject] (ConvertFrom-StringData $_)}
When I ran that against the string you provided I got back:
sessionid : 1472996904
action : "accept"
rcvdbyte : 144
vd : "root"
logid : "0000000013"
policyid : 278
duration : 180
proto : 17
dstname : "dns.google"
srcintf : "port8"
eventtime : 1631556325
appcat : "unscanned"
srcip : 192.168.10.226
dstip : 8.8.4.4
trandisp : "snat"
srcname : "192.168.10.226"
srcport : 54809
devid : "FG10E0TB20903187"
dstdevcategory : "None"
level : "notice"
sentbyte : 245
shaperdroprcvdbyte : 0
sentpkt : 2
masterdstmac : "00:00:0c:07:ac:8d"
shaperrcvdname : "default_class"
poluuid : "01533038-da7b-51eb-b854-8fd38a0deba3"
type : "traffic"
srcintfrole : "dmz"
subtype : "forward"
policytype : "policy"
dstport : 53
transip : 194.56.218.226
shapersentname : "default_class"
dstdevtype : "Unknown"
dstserver : 1
dstcountry : "United States"
dstintf : "wan1"
service : "DNS"
srccountry : "Reserved"
shaperdropsentbyte : 0
dstintfrole : "lan"
transport : 54809
date : 2021-09-13
rcvdpkt : 1
dstmac : "00:00:0c:07:ac:8d"
devname : "chwitrfg01"
time : 20:05:25
You could probably strip quotes out of it if that is desired.
I try to match a fail2ban-regex with a snort3 logfile in alert_json format.
example alert_json output in log-file:
{ "timestamp" : "21/03/22-12:23:56.370262", "seconds" : 1616412236, "action" : "allow", "class" : "none", "b64_data" : "lVAAFpTzAXEAAAAAoAJyELUuAAACBAW0BAIICikv9agAAAAAAQMDBw==", "dir" : "C2S", "dst_addr" : "6.7.8.9", "dst_ap" : "6.7.8.9:0", "eth_dst" : "00:11:22:33:44:55", "eth_len" : 102, "eth_src" : "11:11:22:33:44:55", "eth_type" : "0x800", "gid" : 1, "icmp_code" : 3, "icmp_id" : 0, "icmp_seq" : 0, "icmp_type" : 3, "iface" : "eth0", "ip_id" : 5814, "ip_len" : 68, "msg" : "ICMP Traffic Detected", "mpls" : 0, "pkt_gen" : "raw", "pkt_len" : 88, "pkt_num" : 2270045, "priority" : 0, "proto" : "ICMP", "rev" : 0, "rule" : "1:10000001:0", "service" : "unknown", "sid" : 10000001, "src_addr" : "1.2.3.4", "src_ap" : "1.2.3.4:0", "tos" : 192, "ttl" : 64, "vlan" : 0 }
my fail2ban-regex which didn't match:
^\{.*\"src_addr\"\ :\ \"<HOST>\".*\}$
i tryed this on regexr.com and it match.
i already found out there is maybe some problem with the timestamp but i didn't figured out which?
can somebody help here?
thanks
It'd probably depend on fail2ban version, for example latest fail2ban >= 0.10.6/0.11.2 does not require timestamp anymore (it would simulate "now"), so it shows to me the IP and current time (as I execute it):
$ fail2ban-regex -v /tmp/log '^\{.*\"src_addr\"\ :\ \"<HOST>\".*\}$'
...
Lines: 1 lines, 0 ignored, 1 matched, 0 missed
To specify own datepattern you have to set it in filter (or supply to fail2ban-regex with -d parameter), so this will work:
# either for timestamp tag:
$ fail2ban-regex -v -d ^\{\s*"timestamp"\s*:\s*"%y/%m/%d-%H:%M:%S\.%f" /tmp/log \"src_addr\"\ :\ \"<HOST>\"
# or for posix seconds (probably better because don't need conversion):
$ fail2ban-regex -v -d '"seconds"\s*:\s*{EPOCH}\s*,\s*' /tmp/log '\"src_addr\"\ :\ \"<HOST>\"'
Note that in fail2ban configs you must escape every % as %% due to python ini-configs substitution rules.
Also note that fail2ban cuts part of message matching date pattern out before it apply pref- or failregex.
Also note that your RE is a bit vulnerable, see https://github.com/fail2ban/fail2ban/issues/2932#issuecomment-777320874 for a better example.
I want to look for "\r" in a string field I have in mongo, and I fount this, which looks like it works good:
db.users.findOne({"username" : {$regex : ".*son.*"}});
the problem is that i want to look for "\r" and I can find it, which I know its there, so I just did:
db.users.findOne({"username" : {$regex : ".*\r.*"}});
and it dosent work, how can I fix this?
example document:
{
"personId" : 1,
"personName" : "john",
"address" : {
"city" : "Rue Neuve 2\\r\\rue Pré-du-Mar \\r ché 1 1003 Lausanne",
"street" : "",
"zipCode" : "",
"streetNumber" : ""
}
}
so my query is:
db.users.findOne({"address.city" : {$regex : ".*\r.*"}});
also tried:
db.users.findOne({"address.city" : {$regex : ".*\\r.*"}});
try
db.users.findOne({"username" : {$regex : ".*\\r.*"}});
I think your issue is that you have your .* backwards at the end. You are looking for a "2." literal followed by any characters as opposed to what you have at the beginning, .*, saying anything before the literal that isn't a carriage return. Try to change this to
db.users.findOne({"username" : {$regex : ".*\\r*."}});
Which says give me "\r" with any non carriage return characters before the literal and any non carriage return characters after the literal.
I found that the way to do it is:
db.users.findOne({"username" : {$regex : ".*\\\\.*"}});
Does anyone know of a way to only extract the id3 size information using exiftool. There is a robust documentation but I'm not seeing how to extract only the id3 info.
======== Test_file.mp3
ExifToolVersion : 10.78
FileName : First-time Bosses.mp3
Directory : .
FileSize : 33 MB
FileModifyDate : 2018:02:08 16:35:02-06:00
FileAccessDate : 2018:02:09 13:40:59-06:00
FileInodeChangeDate : 2018:02:09 13:36:59-06:00
FilePermissions : rw-rw-r--
FileType : MP3
FileTypeExtension : mp3
MIMEType : audio/mpeg
MPEGAudioVersion : 1
AudioLayer : 3
AudioBitrate : 128 kbps
SampleRate : 44100
ChannelMode : Joint Stereo
MSStereo : Off
IntensityStereo : Off
CopyrightFlag : False
OriginalMedia : False
Emphasis : None
ID3Size : 222797
EncodedBy : iTunes 9.1
Title : Test_file
Artist : Test_file
Album : Test_file
Genre : Test_file
PictureFormat : PNG
PictureType : Other
PictureDescription :
Picture : (Binary data 212414 bytes, use -b option
to extract)
Duration : 0:36:14 (approx)
I would use grep to find the line, and then awk to keep only the last word:
grep ID3Size Test_file.mp3 | awk 'NF>1{print $NF}'
If that was generated by some function foo, you wouldn't need to mention a source file:
foo | grep ID3Size | awk 'NF>1{print $NF}'