I am trying to parse an error log with regex. It will give me everything I want but now I want to omit the text "client", or any text that would be in that place. All I want from between the [] is the ip address.
^\[([^]]+)\]\s*\[([^]]+)\]\s*\[([^]]+)\]\s*([A-Za-z ]*)[:]\s*([\/a-z-]*)$
Here is a line from the log.
[Mon Aug 23 15:25:35 2010] [error] [client 80.154.42.54] File does not exist: /var/www/phpmy-admin
This should do it:
^\[([^]]+)\]\s*\[([^]]+)\]\s*\[[a-zA-Z ]*([0-9.]+)\]\s*([A-Za-z ]*)[:]\s*([\/a-z-]*)$
Working regex example:
http://regex101.com/r/uN3fO3
Matches: (Using your example data..)
1. `Mon Aug 23 15:25:35 2010`
2. `error`
3. `80.154.42.54`
4. `File does not exist`
5. `/var/www/phpmy-admin`
Related
Hello everyone i'm trying to config joomla with fail2ban so i created
the file /etc/fail2ban/filter.d/joomla-error.conf
and added the failregex as below:
failregex = [[]client <HOST>[]] user .* authentication failure.*
After I added this code into the jail.conf
[joomla-error]
enabled = true
port = http,https
filter = joomla-error
logpath = /var/log/httpd/domains/jayjezz.com.error.log
maxretry = 5
bantime = 30
the logpath is right but every time i try to reload fail2ban service i get
ERROR NOK: ("No 'host' group in '[[]client <HOST>[]] user .* authentication failure.*'",)
i think something is wrong with my regex, can someone provide me the right regex for
[Thu Sep 28 17:14:23.932811 2017] [:error] [pid 6673] [client 000.000.000.000:56806] user xxxxx authentication failure, referer: http://jayjezz.com/administrator/index.php
thank you
fixed this by adding a script to change file permissions inside joomla website. now when i cannot login under /administrator without launching the script first
I'm using fail2ban. For some reason Fail2Ban refuse to compile my regex. Here is my logs that I need to match:
root#server1:/etc/fail2ban/filter.d# tail /var/log/apache2/error.log
[Sun Apr 20 10:40:05 2014] [error] [client 75.144.181.151] user root: authentication failure for "/phpmyadmin/": Password Mismatch
[Sun Apr 20 10:40:16 2014] [error] [client 75.144.181.151] user root: authentication failure for "/phpmyadmin/": Password Mismatch
[Sun Apr 20 10:40:38 2014] [error] [client 75.144.181.151] user haker not found: /phpmyadmin/
[Sun Apr 20 10:40:44 2014] [error] [client 75.144.181.151] user pentest not found: /phpmyadmin/
and here is my fail2ban filter.d file:
root#server1:/etc/fail2ban/filter.d# cat /etc/fail2ban/filter.d/phpmyadmin.conf
[Definition]
failregex = [client <HOST>;] user .*; not found: \/phpmyadmin\/|[client <HOST>;] user root: authentication failure for "\/phpmyadmin\/":
ignoreregex =
here is my regex line from the file above:
[client <HOST>;] user .*; not found: \/phpmyadmin\/|[client <HOST>;] user root: authentication failure for "\/phpmyadmin\/":
Unfortunately fail2ban log file giving me error about regex: Unable to compile regular expression..
root#server1:/etc/fail2ban# tail /var/log/fail2ban.log
2014-04-20 10:47:06,788 fail2ban.filter : INFO Added logfile = /var/log/apache2/error.log
2014-04-20 10:47:06,789 fail2ban.filter : INFO Set maxRetry = 3
2014-04-20 10:47:06,789 fail2ban.filter : INFO Set findtime = 600
2014-04-20 10:47:06,790 fail2ban.actions: INFO Set banTime = 600
2014-04-20 10:47:06,790 fail2ban.filter : ERROR Unable to compile regular expression '[client (?:::f{4,6}:)?(?P<host>[\w\-.^_]+);] user .*; not found: \/phpmyadmin\/|[client (?:::f{4,6}:)?(?P<host>[\w\-.^_]+);] user root: authentication failure for "\/phpmyadmin\/":'
2014-04-20 10:47:06,794 fail2ban.jail : INFO Jail 'ssh' started
2014-04-20 10:47:06,799 fail2ban.jail : INFO Jail 'pureftpd' started
2014-04-20 10:47:06,805 fail2ban.jail : INFO Jail 'phpmyadmin' started
My regex http://regex101.com/r/kU7tX3. What is wrong with this? Any help is appreciated. Thank you.
I would have asked a question in comment but i cannot add a comment:
So trying my best to understand the requirement and giving an answer.
Requirement: I think you are looking to filter all lines containing
"authentication failure for "/phpmyadmin/""
You can do so by changing your regular expression to following:
failregex = .*authentication failure for "\/phpmyadmin\/"
You may have to escape "
Please add comments if this wasn't the correct understanding.....
I have installed Sphinx from source with pgsql and then installed thinking-sphinx gem(3.0.1) on my application (Rails 4.0.3). And I configured & generated the sphinx configurations. Then I added the indices on app/indices and later ran the index & start the sphinx via rake ts:index && ts:start, but I got the below error, let me know the way to resolve this.
rake ts:index
Generating configuration to /home/stc/presto/config/development.sphinx.conf
Sphinx 2.1.7-release (rel21-r4638)
Copyright (c) 2001-2014, Andrew Aksyonoff
Copyright (c) 2008-2014, Sphinx Technologies Inc (http://sphinxsearch.com)
using config file '/home/stc/config/development.sphinx.conf'...
FATAL: no indexes found in config file '/home/stc/config/development.sphinx.conf'
rake ts:start
In the log file I can see the below errors
[Wed Apr 2 10:40:49.834 2014] [14338] Child process 14339 has been forked
[Wed Apr 2 10:40:49.835 2014] [14339] listening on 127.0.0.1:9306
[Wed Apr 2 10:40:49.835 2014] [14339] WARNING: ERROR: index 'collection_core': RT indexes support prefixes and infixes with only dict=keywords - NOT SERVING
[Wed Apr 2 10:40:49.836 2014] [14339] WARNING: ERROR: index 'resource_core': RT indexes support prefixes and infixes with only dict=keywords - NOT SERVING
[Wed Apr 2 10:40:49.836 2014] [14339] WARNING: index 'collection': no such local index 'collection_core' - SKIPPING LOCAL INDEX
[Wed Apr 2 10:40:49.836 2014] [14339] WARNING: index 'collection': no valid local/remote indexes in distributed index - NOT SERVING
[Wed Apr 2 10:40:49.836 2014] [14339] WARNING: index 'resource': no such local index 'resource_core' - SKIPPING LOCAL INDEX
[Wed Apr 2 10:40:49.836 2014] [14339] WARNING: index 'resource': no valid local/remote indexes in distributed index - NOT SERVING
[Wed Apr 2 10:40:49.836 2014] [14339] FATAL: no valid indexes to serve
[Wed Apr 2 10:40:49.836 2014] [14338] Child process 14339 has been finished, exit code 1. Watchdog finishes also. Good bye!
Addressed this on the Thinking Sphinx Google group as well:
You seem to be using real-time indices - which is great - but that means you don’t need to use the ts:index task. The two main tasks that are useful are:
ts:generate - which adds/updates all documents in each real-time index.
ts:regenerate - which stops Sphinx, clears out existing index files, generates the configuration, starts Sphinx and runs ts:generate.
However, you’re also using either min_infix_len or min_prefix_len - and with Sphinx 2.1, the default dict setting doesn’t match with what is needed (as the logs detail). So, if you add dict: keywords to the appropriate environments in config/thinking_sphinx.yml and then run ts:regenerate, you should hopefully have a working Sphinx setup.
I need a regex for parsing Apache files
For example:
Here is a portion of a /var/log/httpd/error_log
[Sun Sep 02 03:34:01 2012] [notice] Digest: done
[Sun Sep 02 03:34:01 2012] [notice] Apache/2.2.15 (Unix) DAV/2 mod_ssl/2.2.15 OpenSSL/1.0.0- fips SVN/1.6.11 configured -- resuming normal operations
[Sun Sep 02 03:34:01 2012] [error] avahi_entry_group_add_service_strlst("localhost") failed: Invalid host name
[Sun Sep 02 08:01:14 2012] [error] [client 216.244.73.194] File does not exist: /var/www/html/manager
[Sun Sep 02 11:04:35 2012] [error] [client 58.218.199.250] File does not exist: /var/www/html/proxy
I want a regex that includes space as delimiter and excludes embedded space. And the apache error log format alternates between
[DAY MMM DD HH:MM:SS YYYY] [MSG_TYPE] DESCRIPTOR: MESSAGE
[DAY MMM DD HH:MM:SS YYYY] [MSG_TYPE] [SOURCE IP] ERROR: DETAIL
I created 2 Regexes, 1st one is
^(\[[\w:\s]+\]) (\[[\w]+\]) (\[[\w\d.\s]+\])?([\w\s/.(")-]+[\-:]) ([\w/\s]+)$
This one is simple and just match the contents as it is
I want something like the following Regex which I created
(?<=|\s)([\w:\S]+)
This one doesn't give me the desired output, it doesn't include embedded space. So I need a regex which groups each field, includes embedded space and uses space as delimiter. Pls Help me out with the logic!!!!
my code
void regexparser( CharBuffer cb)
{ try{
Pattern linePattern = Pattern.compile(".*\r?\n");
Pattern csvpat = Pattern.compile( "^\\[([\\w:\\s]+)\\] \\[([\\w]+)\\] (\\[([\\w\\d.\\s]+)\\])?([\\w\\s/.(\")-]+[\\-:]) ([\\w/\\s].+)",Pattern.CASE_INSENSITIVE | Pattern.UNICODE_CASE | Pattern.MULTILINE);
Matcher lm = linePattern.matcher(cb);
Matcher pm = null;
while(lm.find())
{ //System.out.print("1st loop");
CharSequence cs = lm.group();
if (pm==null)
pm = csvpat.matcher(cs);
else
pm.reset(cs);
while(pm.find())
{ // System.out.println("2nd loop");
//System.out.println(pm.groupCount());
//CharSequence ps = pm.group();
//System.out.print(ps);
if(pm.group(4)==null)
System.out.println(pm.group(1)+" "+pm.group(2)+" "+pm.group(5)+" "+pm.group(6));
else
System.out.println(pm.group(1)+" "+pm.group(2)+" "+pm.group(4)+" "+pm.group(5)+" "+pm.group(6));
I agree that this task should be done with an existing solution to parse Apache logs.
However, if you want to try something out for training purposes, maybe you want to start with this. Instead of parsing everything in one single huge regex, I do it in small steps that are much better readable:
Code
#!/usr/bin/env perl
use strict;
use warnings;
use DateTime::Format::Strptime;
use feature 'say';
# iterate log lines
while (defined(my $line = <DATA>)) {
chomp $line;
# prepare
my %data;
my $strp = DateTime::Format::Strptime->new(
pattern => '%a %b %d %H:%M:%S %Y',
);
# consume date/time
next unless $line =~ s/^\[(\w+ \w+ \d+ \d\d:\d\d:\d\d \d{4})\] //;
$data{date} = $strp->parse_datetime($1);
# consume message type
next unless $line =~ s/^\[(\w+)\] //;
$data{type} = $1;
# "[source ip]" alternative
if ($line =~ s/^\[(\w+) ([\d\.]+)\] //) {
#data{qw(source ip)} = ($1, $2);
# consume "error: detail"
next unless $line =~ s/([^:]+): (.*)//;
#data{qw(error detail)} = ($1, $2);
}
# "descriptor: message" alternative
elsif ($line =~ s/^([^:]+): (.*)//) {
#data{qw(descriptor message)} = ($1, $2);
}
# invalid
else {
next;
}
# something left: invalid
next if length $line;
# parsed ok: output
say "$_: $data{$_}" for keys %data;
say '-' x 40;
}
__DATA__
[Sun Sep 02 03:34:01 2012] [notice] Digest: done
[Sun Sep 02 03:34:01 2012] [notice] Apache/2.2.15 (Unix) DAV/2 mod_ssl/2.2.15 OpenSSL/1.0.0- fips SVN/1.6.11 configured -- resuming normal operations
[Sun Sep 02 03:34:01 2012] [error] avahi_entry_group_add_service_strlst("localhost") failed: Invalid host name
[Sun Sep 02 08:01:14 2012] [error] [client 216.244.73.194] File does not exist: /var/www/html/manager
[Sun Sep 02 11:04:35 2012] [error] [client 58.218.199.250] File does not exist: /var/www/html/proxy
Output
descriptor: Digest
date: 2012-09-02T03:34:01
type: notice
message: done
----------------------------------------
descriptor: avahi_entry_group_add_service_strlst("localhost") failed
date: 2012-09-02T03:34:01
type: error
message: Invalid host name
----------------------------------------
detail: /var/www/html/manager
source: client
ip: 216.244.73.194
date: 2012-09-02T08:01:14
error: File does not exist
type: error
----------------------------------------
detail: /var/www/html/proxy
source: client
ip: 58.218.199.250
date: 2012-09-02T11:04:35
error: File does not exist
type: error
----------------------------------------
Note that according to your format description, the second line is invalid and ignored by the program.
I can't seem to find the problem for the life of me. Very simply, I have a database object that I'm pulling from the database, incrementing it's "views" by one, and saving. My view display's the incremented value, but then my logs show that the value is incremented AGAIN.
g=Game.objects.filter(slug=slug).distinct()[0]
g.views += 1
g.save()
Here's my logs:
[Fri Oct 29 15:15:49 2010] [error] DEBUG:root:Updating plays
[Fri Oct 29 15:15:49 2010] [error] DEBUG:root:plays: 40
[Fri Oct 29 15:15:50 2010] [error] DEBUG:root:Updating plays
[Fri Oct 29 15:15:50 2010] [error] DEBUG:root:plays: 41
My view shows that it has 40 hits. This causes that my view increments by 2 every time I refresh:
[Fri Oct 29 15:20:19 2010] [error] DEBUG:root:Updating plays
[Fri Oct 29 15:20:19 2010] [error] DEBUG:root:plays: 42
[Fri Oct 29 15:20:19 2010] [error] DEBUG:root:Updating plays
[Fri Oct 29 15:20:19 2010] [error] DEBUG:root:plays: 43
Any hints as to what this could be?
EDIT:
Here's my view. I simplified it to the core elements (and it still behaves oddly).
def game(request, slug=None):
g=Game.objects.filter(slug=slug)[0]
if g:
comments=GameComment.objects.filter(item=g, parent__isnull=True)
g.plays+=1
g.save()
else:
comments=None
return render_to_response('goto/goto_game.html', {'g': g, 'comments':comments, 'cCount':len(comments) if comments else 0, 'newCCount':0}, request=request)
AH! Turns out it was a missing url and ajax issue. It wasn't the favicon, as that only hits the root, and not my specific view.
The problem came when I had an AJAX command like this:
$.post('{{url("_submitcomment")}}', data, function(data) { ...
and that 'url' could not reverse '_submitcomment' so returned an empty string. Calling
$.post('');
will hit your current URL and try to fetch everything!
(I'm using Jinja2, btw, so the url command probably looks different from them traditional Django template system.)
Thanks for the help and the tips!
In my case, each view was called twice, because my base template contained incorrect icon link in the <head> tag. It might look like this:
<link rel="icon" href="#">
So, after the page is loaded the browser sends a request to the current page again, which causes the second view call.
Remove the wrong <link> tag, and it should work.
Can you verify the second request is not for "/favicon.ico" ?