ElK stack AWS S3 log grok pattern - amazon-web-services

Can someone help me creating a grook pattern for this kind of log:
79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be mybucket [06/Feb/2014:00:00:38 +0000] 192.0.2.3 79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be 3E57427F3EXAMPLE REST.GET.VERSIONING - "GET /mybucket?versioning HTTP/1.1" 200 - 113 - 7 - "-" "S3Console/0.4" -
79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be mybucket [06/Feb/2014:00:00:38 +0000] 192.0.2.3 79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be 891CE47D2EXAMPLE REST.GET.LOGGING_STATUS - "GET /mybucket?logging HTTP/1.1" 200 - 242 - 11 - "-" "S3Console/0.4" -
79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be mybucket [06/Feb/2014:00:00:38 +0000] 192.0.2.3 79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be A1206F460EXAMPLE REST.GET.BUCKETPOLICY - "GET /mybucket?policy HTTP/1.1" 404 NoSuchBucketPolicy 297 - 38 - "-" "S3Console/0.4" -
79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be mybucket [06/Feb/2014:00:01:00 +0000] 192.0.2.3 79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be 7B4A0FABBEXAMPLE REST.GET.VERSIONING - "GET /mybucket?versioning HTTP/1.1" 200 - 113 - 33 - "-" "S3Console/0.4" -
79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be mybucket [06/Feb/2014:00:01:57 +0000] 192.0.2.3 79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be DD6CC733AEXAMPLE REST.PUT.OBJECT s3-dg.pdf "PUT /mybucket/s3-dg.pdf HTTP/1.1" 200 - - 4406583 41754 28 "-" "S3Console/0.4" -
79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be mybucket [06/Feb/2014:00:03:21 +0000] 192.0.2.3 79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be BC3C074D0EXAMPLE REST.GET.VERSIONING - "GET /mybucket?versioning HTTP/1.1" 200 - 113 - 28 - "-" "S3Console/0.4" -
I have to analize them but I don't really know the way to create a grook filter for those logs, and also get #timestamp from them, thanks a lot!

This grok debugger tool is also useful: http://grokdebug.herokuapp.com/

Use the grok pattern show below.
%{WORD:Bucket_Owner} %{WORD:bucket_name} %{DATA:timestamp} %{IP:Remote_IP} %{WORD:Requester} %{WORD:Request} %{DATA:Rest} %{WORD:HTTP_Status} \- %{NUMBER:Bytes_Sent} \- %{NUMBER:Object_Size} \- \"-" \"%{DATA:S3_console}" \-

Related

Error 4xx AWS Elastic Beanstalk - Severe integrity

Good afternoon people,
I created an environment in Elastic Beanstalk and uploaded a NODEjs application an api with express.
She's working fine, all right.
But the integrity of the environment is reported as serious, and this monitoring attempt appears in the logs.
----------------------------------------
/var/log/nginx/access.log
----------------------------------------
172.31.46.198 - - [03/Nov/2021:19:14:13 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.1.181 - - [03/Nov/2021:19:14:13 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.30.127 - - [03/Nov/2021:19:14:13 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.46.198 - - [03/Nov/2021:19:14:28 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.1.181 - - [03/Nov/2021:19:14:28 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.30.127 - - [03/Nov/2021:19:14:28 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.46.198 - - [03/Nov/2021:19:14:43 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.30.127 - - [03/Nov/2021:19:14:43 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.1.181 - - [03/Nov/2021:19:14:43 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.30.127 - - [03/Nov/2021:19:14:58 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.1.181 - - [03/Nov/2021:19:14:58 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.46.198 - - [03/Nov/2021:19:14:58 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
172.31.30.127 - - [03/Nov/2021:19:15:13 +0000] "GET / HTTP/1.1" 404 139 "-" "ELB-HealthChecker/2.0" "-"
Does anyone know how I can fix this, without turning off the monitoring?
Good night people,
I found the problem, I didn't have anything set in my API's root on "/", so EB tried to monitor the api state and took a 404.
I set up a HealthCheck on the root "/" and normalized the 404 errors and integrity issue in the environment.

Regex in fail2ban not matching

Should be a simple thing, but with regex nothing is simple.
My fail2ban filter for wordpress sites:
[Definition]
#failregex = <HOST>.*POST.*(wp-login\.php|xmlrpc\.php).* 200
#failregex = <HOST>.*POST.*(wp-login\.php|xmlrpc\.php).* 200[ 0-9]*
failregex = ^"<HOST> .* "POST .*wp-login.php
#failregex = <HOST>.*POST.*wp-login.php .*
#failregex = ^"<HOST> .* "POST .*(wp-login.php|xmlrpc.php) HTTP/.*" (200|401)
ignoreregex =
As you can see I have tested multiple things, but I just don't get a match. Odly I do get a match on regex101.
And this is my logfile (those entires should be found):
"hostname 172.70.34.43 - - [18/May/2021:05:58:22 +0000] "POST //wp-login.php HTTP/1.1" 200 3069"
"hostname 172.70.34.43 - - [18/May/2021:05:58:22 +0000] "POST //wp-login.php HTTP/1.1" 200 3069"
"hostname 172.70.34.43 - - [18/May/2021:05:58:21 +0000] "POST //wp-login.php HTTP/1.1" 200 3069"
The logfile could also contain entries like this:
"hostname 172.69.63.84 - - [19/May/2021:09:23:01 +0000] "GET /feed/ HTTP/1.1" 200 14872"
"hostname 172.69.63.84 - - [19/May/2021:09:23:00 +0000] "GET /feed HTTP/1.1" 301 0"
"hostname 162.158.91.10 - - [19/May/2021:09:23:01 +0000] "POST /wp-cron.php?doing_wp_cron=1621416181.1017169952392578125000 HTTP/1.1" 200 0"
"hostname 172.68.57.138 - - [19/May/2021:09:22:34 +0000] "GET /versand/ HTTP/1.1" 200 27456"
"hostname 172.68.110.69 - - [19/May/2021:09:22:34 +0000] "POST /wp-cron.php?doing_wp_cron=1621416154.5001699924468994140625 HTTP/1.1" 200 0"
"hostname 172.69.34.217 - - [19/May/2021:09:19:48 +0000] "GET / HTTP/1.1" 200 32986"
And I have tested with fail2ban-regex, but with no success. I have also tried to replace < HOST > with the actual hostname, but in this case fail2ban will not accept the regex.
Running tests
=============
Use failregex filter file : wordpress, basedir: /etc/fail2ban
Use log file : /home/runcloud/logs/tmp.log
Use encoding : UTF-8
Results
=======
Failregex: 0 total
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [3] Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-
Lines: 3 lines, 0 ignored, 0 matched, 3 missed
this regex match (in this example the first 3 lines)
"POST request on either wp-login.php or xmlrp.php" as rapsli wanted
"POST\b.+\b(wp-login|xmlrp)\.php
in
"hostname 172.70.34.43 - - [18/May/2021:05:58:22 +0000] "POST //wp-login.php HTTP/1.1" 200 3069"
"hostname 172.70.34.43 - - [18/May/2021:05:58:22 +0000] "POST //wp-login.php HTTP/1.1" 200 3069"
"hostname 172.70.34.43 - - [18/May/2021:05:58:21 +0000] "POST //wp-login.php HTTP/1.1" 200 3069"
"hostname 172.69.63.84 - - [19/May/2021:09:23:01 +0000] "GET /feed/ HTTP/1.1" 200 14872"
"hostname 172.69.63.84 - - [19/May/2021:09:23:00 +0000] "GET /feed HTTP/1.1" 301 0"
"hostname 162.158.91.10 - - [19/May/2021:09:23:01 +0000] "POST /wp-cron.php?doing_wp_cron=1621416181.1017169952392578125000 HTTP/1.1" 200 0"
"hostname 172.68.57.138 - - [19/May/2021:09:22:34 +0000] "GET /versand/ HTTP/1.1" 200 27456"
"hostname 172.68.110.69 - - [19/May/2021:09:22:34 +0000] "POST /wp-cron.php?doing_wp_cron=1621416154.5001699924468994140625 HTTP/1.1" 200 0"
"hostname 172.69.34.217 - - [19/May/2021:09:19:48 +0000] "GET / HTTP/1.1" 200 32986"
https://regexr.com/5t8e3
needs to stand for the place with the IP. So this regex should work with fail2ban
failregex = "[a-z]* <HOST>.*(wp-login\.php|xmlrpc.php).*

How i filter fluentD logs on kubernetes?

My kubernetes have liveness enable, and it log on application, like this:
kubectl logs -n example-namespace example-app node-app
::ffff:127.0.0.1 - - [17/Sep/2020:14:12:19 +0000] "GET /docs HTTP/1.1" 301 175
::ffff:127.0.0.1 - - [17/Sep/2020:14:13:19 +0000] "GET /docs/ HTTP/1.1" 200 3104
::192.168.0.1 - - [17/Sep/2020:14:13:19 +0000] "GET /home-page HTTP/1.1" 200 3104
::ffff:127.0.0.1 - - [17/Sep/2020:14:13:19 +0000] "GET /docs HTTP/1.1" 301 175
::ffff:127.0.0.1 - - [17/Sep/2020:14:13:22 +0000] "GET /docs/ HTTP/1.1" 200 3104
I Use fluentD to send logs to ClowdWatch.
My fluentD configuration:
https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluentd-quickstart.yaml
How can i filter, to fluentD only matches
::192.168.0.1 - - [17/Sep/2020:14:13:19 +0000] "GET /home-page HTTP/1.1" 200 3104
And ignore
::ffff:127.0.0.1 - - [17/Sep/2020:14:13:19 +0000] "GET /docs HTTP/1.1" 301 175
Thanks!
After some research, i found this solution:
<match kubernetes.var.log.containers.**_kube-system_**>
#type null
</match>
and this
<filter **>
#type grep
exclude1 log docs
</filter>
The reference:
https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/issues/91
https://docs.fluentd.org/filter/grep
EDIT
or add:
exclude_path ["/var/log/containers/cloudwatch-agent*", "/var/log/containers/fluentd*", "/var/log/containers/*istio*"]
this config ignore the source files with pattern istio.

How to use group in match method to extract data from Apache logs

I'm trying to extract the IP addresses and URLs from an Apache log file using grouping match.
The access.log file is:
10.0.0.3 - - [08/Jul/2019:10:26:41 +0000] "GET /hello.html HTTP/1.1" 404 444 "-" "curl/7.52.1"
10.0.0.3 - - [08/Jul/2019:10:26:41 +0000] "GET /hello.html HTTP/1.1" 404 444 "-" "curl/7.52.1"
10.0.0.3 - - [08/Jul/2019:10:26:41 +0000] "GET /secret.html HTTP/1.1" 200 282 "-" "curl/7.52.1"
And I'm using:
File.open("access.log").each do |line|
m = /(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})(.+\/)([a-zA-Z0-9]+.html)/.match(line)
puts m
end
puts m gives me this:
10.0.0.3 - - [08/Jul/2019:10:26:41 +0000] "GET /hello.html
puts m[1] and puts m[3] gives me an error:
1.rb:25:in block in <main>': undefined method []' for nil:NilClass (NoMethodError)
I expected "10.0.0.3" and "hello.html".
What am I doing wrong?
Your expression seems to be working just fine, maybe we'd just a bit modify that to:
re = /(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})(.+\/)([a-zA-Z0-9]+\.html)/s
str = '10.0.0.3 - - [08/Jul/2019:10:26:41 +0000] "GET /hello.html HTTP/1.1" 404 444 "-" "curl/7.52.1"
10.0.0.3 - - [08/Jul/2019:10:26:41 +0000] "GET /hello.html HTTP/1.1" 404 444 "-" "curl/7.52.1"
10.0.0.3 - - [08/Jul/2019:10:26:41 +0000] "GET /secret.html HTTP/1.1" 200 282 "-" "curl/7.52.1"'
str.scan(re) do |match|
puts match.to_s
end
Output
["10.0.0.3", " - - [08/Jul/2019:10:26:41 +0000] \"GET /", "hello.html"]
["10.0.0.3", " - - [08/Jul/2019:10:26:41 +0000] \"GET /", "hello.html"]
["10.0.0.3", " - - [08/Jul/2019:10:26:41 +0000] \"GET /", "secret.html"]
The expression is explained on the top right panel of this demo if you wish to explore/simplify/modify it.
I don't know how general it can be, but try:
File.open("apache.log").each do |line|
ip = line.split.first
path = line.split(/(?:GET|POST|PUT|PATCH) /).last.split(/ (?:HTTP|HTTPS)/).first
puts ("#{ip} - #{path}")
end
For the data sample it returns:
# 10.0.0.3 - /hello.html
# 10.0.0.3 - /hello.html
# 10.0.0.3 - /secret.html

AWS elastic beanstalk cannot return custom response code using resteasy?

I'm working on a web service using RESTEASY to set the response status code when get some exception.
First I tried resteasy exception mapper which works fine locally. The mapper code attached below. However, when I upload that WS into elastic beanstalk, that always return 500 (internal server error).
#Provider
public class LoadGridTileFailedExceptionMapper extends BaseExceptionMapper implements ExceptionMapper<LoadGridTileFailedException>
{
#Override
public Response toResponse(LoadGridTileFailedException e)
{
log(e.getMessage(), e);
return printMsg(e.getMessage(), DtmWebServiceReturnStatus.LOAD_GRID_TILE_FAILED_EXCEPTION_CODE);
}
}
Then I try just throw exception WebApplicationException(ex, DtmWebServiceReturnStatus.LOAD_GRID_TILE_FAILED_EXCEPTION_CODE) to get around exception mapping. The result is that I got a response status 498(LOAD_GRID_TILE_FAILED_EXCEPTION_CODE) wrapped in status code 500.
Apache Tomcat/7.0.27 - Error report HTTP Status 498 - type Status reportmessage description http.498Apache Tomcat/7.0.27
It seems that elastic beanstalk wrapped all exceptions throw out in the server side with status code 500?The question is how can I get around that feature and return the status code I set in response? Thank you.
UPDATE
Try more requests this morning and find something interesting:
Get the right return status in elastic beanstalk log snapshot
/var/log/tomcat7/localhost_access_log.txt
127.0.0.1 - - [09/Jan/2013:15:06:28 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:31 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:34 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:37 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:39 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:41 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:44 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:48 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:51 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:54 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
127.0.0.1 - - [09/Jan/2013:15:06:57 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22
/var/log/httpd/elasticbeanstalk-access_log
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:28 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:31 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:34 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:37 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:39 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:41 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:44 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:48 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:51 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:54 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
10.28.215.233 (65.167.11.254, 10.28.215.233) - - [09/Jan/2013:15:06:57 +0000] "GET /published/tile/003331330031 HTTP/1.1" 498 22 "-" "-"
However in client side, still got 500 :-(
printMsg method:
protected Response printMsg(String msg, int intStatus)
{
// Need this due to the Resteasy bug
ServiceDataCollector.processRequest(true);
ResponseBuilder builder = Response.status(intStatus);
builder.type("text/plain");
builder.entity("ERROR: " + msg);
Response rep = builder.build();
LOG.error(rep.getStatus() + ":" + rep.toString());
return rep;
}
Some one help me to work the problem out. I had the httpd deployed in my AMI before tomcat server at 80. So the load balancer will interact with httpd server, which change the status code from tomcat to 500. Disable that httpd server will solve the problem. Thx for everyone's help.