I am seeking to extract timestamps and ip addresses out of log entries containing a varying amount of information. The basic structure of a log entry is:
<timestamp>, <token_1>, <token_2>, ... ,<token_n>, <ip_address> <token_n+2>, <token_n+3>, ... ,<token_n+m>,-
The number of tokens n between the timestamp and ip address varies considerably.
I have been studying regular expressions and am able to grep timestamps as follows:
grep -o "[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}T[0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\}"
And ip addresses:
grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
But I have not been able to grep both patterns out of log entries which contain both. Every log entry contains a timestamp, but not every entry contains an ip address.
Input:
2021-04-02T09:06:44.248878+00:00,Creation Time,EVT,WinEVTX,[4624 / 0x1210] Source Name: Microsoft-Windows-Security-Auditing Message string: An account was successfully logged on.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-18\n\tAccount Name:\t\tREDACTED$\n\tAccount Domain:\t\tREDACTED\n\tLogon ID:\t\tREDACTED\n\nLogon Type:\t\t\t10\n\nNew Logon:\n\tSecurity ID:\t\tREDACTED\n\tAccount Name:\t\tREDACTED\n\tAccount Domain:\t\tREDACTED\n\tLogon ID:\t\REDACTED\n\tLogon GUID:\t\tREDACTED\n\nProcess Information:\n\tProcess ID:\t\tREDACTED\n\tProcess Name:\t\tC:\Windows\System32\winlogon.exe\n\nNetwork Information:\n\tWorkstation:\tREDACTED\n\tSource Network Address:\t255.255.255.255\n\tSource Port:\t\t0\n\nDetailed Authentication Information:\n\tLogon Process:\t\tUser32 \n\tAuthentication Package:\tNegotiate\n\tTransited Services:\t-\n\tPackage Name (NTLM only):\t-\n\tKey Length:\t\t0\n\nThis event is generated when a logon session is created. It is generated on the computer that was accessed.\n\nThe subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service or a local process such as Winlogon.exe or Services.exe.\n\nThe logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).\n\nThe New Logon fields indicate the account for whom the new logon was created i.e. the account that was logged on.\n\nThe network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.\n\nThe authentication information fields provide detailed information about this specific logon request.\n\t- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.\n\t- Transited services indicate which intermediate services have participated in this logon request.\n\t- Package name indicates which sub-protocol was used among the NTLM protocols.\n\t- Key length indicates the length of the generated session key. This will be 0 if no session key was requested. Strings: ['S-1-5-18' 'DEVICE_NAME$' 'NETWORK' 'REDACTED' 'REDACTED' 'USERNAME' 'WORKSTATION' 'REDACTED' '10' 'User32 ' 'Negotiate' 'REDACTED' '{REDACTED}' '-' '-' '0' 'REDACTED' 'C:\\Windows\\System32\\winlogon.exe' '255.255.255.255' '0' '%%1833'] Computer Name: REDACTED Record Number: 1068355 Event Level: 0,winevtx,OS:REDACTED,-
Desired Output:
2021-04-02T09:06:44, 255.255.255.255
$ sed -En 's/.*([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}).*[^0-9]([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}).*/\1, \2/p' file
2021-04-02T09:06:44, 255.255.255.255
Your regexps can be reduced by removing some of the explicit repetition though:
$ sed -En 's/.*([0-9]{4}(-[0-9]{2}){2}T([0-9]{2}:){2}[0-9]{2}).*[^0-9](([0-9]{1,3}\.){3}[0-9]{1,3}).*/\1, \4/p' file
2021-04-02T09:06:44, 255.255.255.255
It could be simpler still if all of the lines in your log file start with a timestamp:
$ sed -En 's/([^,.]+).*[^0-9](([0-9]{1,3}\.){3}[0-9]{1,3}).*/\1, \2/p' file
2021-04-02T09:06:44, 255.255.255.255
If you are looking for lines that contain both patterns, it may be easiest to do it two separate searches.
If you're searching your log file for lines that contain both "dog" and "cat", it's usually easiest to do this:
grep dog filename.txt | grep cat
The grep dog will find all lines in the file that match "dog", and then the grep cat will search all those lines for "cat".
You seem not to know the meaning of the "-o" switch.
Regular "grep" (without "-o") means: give the entire line where the pattern can be found. Adding "-o" means: only show the pattern.
Combining two "grep" in a logical AND-clause can be done using a pipe "|", so you can do this:
grep <pattern1> <filename> | grep <pattern2>
I want to add the info below into the file usr/local/nagios/etc/hosts.cfg but want to do it just below ##company in the hosts.cfg file. My setup script will contain the info that needs to be added
I have spend hours trying to get sed to just add a line into a file after a marker but to no avail
define host{
use linux-box
host_name $host_name
alias $alias
address $ip
parents $parent
notification_period 24x7
notification_interval 5
}
Previously I used
cat <> /path /filename
EOT
but now I need to do it in specif places in the file
Given the following file:
# some content
###company
If I run the following command:
sed -i 's/###company/&\ndefine host {\nuse host\nhost_name HOSTNAME/' file
Now, the contents of file are:
# some content
###company
define host {
use host
host_name HOSTNAME
Is this what you're looking for?
I am trying to monitor a web service with check_http plug-in of Nagios.
The url I am trying to monitor includes url parameter.
And it turns out that check_http plugin will ignore the url parameter when checking.
Herewith my configuration.
'check_http' command definition
define command{
command_name check_http
command_line $USER1$/check_http -I $HOSTADDRESS$ $ARG1$
}
define service{
use local-service ; Name of service template to use
host_name pear
service_description HTTP
check_command check_http!-u http://pear.com/total?eId=12345&env=abcde
notifications_enabled 0
}
Try replacing the value passed into -u with a relative path instead of the full URL.
In this example the hostname (-H) will be supplied by $HOSTADDRESS$ which is taken from the address field of the pear host definition.
The value passed into the -u parameter should be a relative path, for example: /total?eId=12345&env=abcde.
We'll add the -u to the check_http_with_args command definition so we don't have to pass it as part of our parameters in our service definition.
define host {
host_name pear
alias pear
address pear.com
use linux-server
contact_groups admins
notification_interval 0
notification_period 24x7
notifications_enabled 1
register 1
}
define command{
command_name check_http_with_args
command_line $USER1$/check_http -H $HOSTADDRESS$ -u $ARG1$
}
define service {
service_description pear_total_http
use generic-service
check_command check_http_with_args!/total?eId=12345&env=abcde
host_name pear
contact_groups admins
notification_interval 0
notification_period 24x7
notifications_enabled 1
flap_detection_enabled 1
register 1
}
In the end, the command that Nagios execute should get translated into something that looks like this:
/usr/local/nagios/libexec/check_http -H pear.com -u /total?eId=12345&env=abcde
You can try executing the above from the command line to make sure it works for you.
Note: Replace the path to check_http with the actual path corresponding to your installation location on your Nagios server.
Relevant parts of the check_http man pages we're referencing:
-H, --hostname=ADDRESS
Host name argument for servers using host headers (virtual host)
Append a port to include it in the header (eg: example.com:5000)
...
-u, --url=PATH
URL to GET or POST (default: /)
Source: https://www.monitoring-plugins.org/doc/man/check_http.html
EDIT:
To answer your questions from the comment, the -k or --header= will allow you to pass in headers.
-k, --header=STRING
Any other tags to be sent in http header. Use multiple times for additional headers
Source: https://www.monitoring-plugins.org/doc/man/check_http.html
So to specify an Accept header, I'd modify the following to look like:
define command{
command_name check_http_with_args
command_line $USER1$/check_http -H $HOSTADDRESS$ -u "$ARG1$" -k "$ARG2$"
}
define service {
service_description pear_total_http
use generic-service
check_command check_http_with_args!/total?eId=12345&env=abcde!Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
host_name pear
contact_groups admins
notification_interval 0
notification_period 24x7
notifications_enabled 1
flap_detection_enabled 1
register 1
}
...adding -k "$ARG2$" to the command_line of the command definition and adding the Accept: <MIME types> (for example Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8) to the check_command of the service definition.
Also, I wrapped $ARG1$ of -u "$ARG1$" part of the command_line in double quotation marks as I suspect the & in eId=12345&env=abcde is causing the shell to think the end of the command is terminated just before the &. Wrapping the parameter in double quotes should make it see the whole string as one whole argument.
I want to log varnish backend request which matches specified IP (for example 127.0.0.1).
So i have
"varnishlog -b -I BereqHeader:X-Forwarded-For: 127.0.0.1'"
Which actualy logs only the "BereqHeader:X-Forwarded-For:" part. I want to log full request, not only IP part.
That was first question, the second one is: how to disable loging empty request? I mean, if i have regex filter then i have a lot of request looking like this "* << BeReq >> 307454" and i obviously dont want to see them.
I have a solution. Log the data by
varnishlog -b -I BereqHeader:'X-Forwarded-For: 123.215.32.76' -i [other tags to log] > file.varnishlog
and then grep it by
cat file.varnishlog | grep -Pzo '* {3}<< BeReq {4}>>.\n- BereqHeader.+\n(-.\n)*'
which'll give us expected results.
I have a master virtual machine with a few websites hosted on it. When I deploy a copy of this, the settings are a standard value, and I often need to change them so they fit different databases. For this I would like to use a small c# program so I can use regular expressions to find and replace whatever property needed. For this I use fnr (http://findandreplace.codeplex.com/).
findandreplace.bat:
set DATABASE="dbname"
set DBHOST="my.host.name"
set DBPORT="1234"
set ENV="this.env"
ECHO CHANGE DATABASE
"fnr" --cl --dir "D:\Inetpub\wwwroot" --fileMask "web.config" --includeSubDirectories --useRegEx --find "<add key=\"DATABASE\" value=\".*\"\s*>" --replace "<add key=\"DATABASE\" value=\"%DATABASE%\">"
ECHO CHANGE HOST
"fnr" --cl --dir "D:\Inetpub\wwwroot" --fileMask "web.config" --includeSubDirectories --useRegEx --find "<add key=\"HOST\" value=\".*\"\s*>" --replace "<add key=\"HOST\" value=\"%DBHOST%\">"
ECHO CHANGE PORT
"fnr" --cl --dir "D:\Inetpub\wwwroot" --fileMask "web.config" --includeSubDirectories --useRegEx --find "<add key=\"ConnectionString\" value=\"Data Source.*>" --replace "<add key=\"ConnectionString\" value=\"Data Source=(DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = %%3%%)(port = %DBPORT%)) )(CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = %%2%%)));User ID=%%0%%;Password=%%1%%;Enlist=false\">"
set /p await= "any key to continue"
Windows command prompt gives me the following output in the end:
CHANGE PORT
The system cannot find the file specified.
"any key to continue"
I can see the first two goes through and if I check the files it should touch, the values are changed correctly. I just cannot understand why the change port doesnt work, and tells me it cannot find the file specified. One, it doesnt specify any single file, and two, it looks for the same files as the first two.
Also, if I try to run the regex in FNR (the find part, with same parameters as only web.config files and in the same folder+subfolders) it finds the files perfectly.
Anyone got an idea as to what I'm doing wrong?
Thanks in advance.