I'm using ModSecurity to catch a string from JSON response and log it.
Here is a sample HTTP Request and Response:
Request:
Request Method:POST
Status Code:200 OK
Accept:application/json, text/plain, */*
Content-Type:application/json;charset=UTF-8
Pragma:no-cache
X-Requested-With:XMLHttpRequest
{"data":{"username":"andrew","password":"AndreWTestingUser123dddd","remember":false,"authType":0}}
Response:
Connection:close
Content-Type:application/json
Date:Mon, 02 Feb 2015 10:39:15 GMT
Server:Apache
{"status":401,"messages":["Invalid password. (error code: 401)"],"createCookie":false}
This the rule i wrote:
SecRule STREAM_OUTPUT_BODY "#rx (?i)Invalid password"\
"id:9900022, \
phase:4 \
log, \
pass, \
chain, \
t:none, \
msg:'Invalid VT login detected', \
logdata:%{RESPONSE_BODY}"
SecRule RESPONSE_HEADERS:Content-Type "application/json" \
"log, chain"
SecRule REQUEST_URI "/api.php/auth/login" \
"log"
However ModSecurity is not loggin this request as expected.
Here is my ModSecurity Version Info:
ModSecurity for Apache/2.8.0 (http://www.modsecurity.org/) configured.
APR compiled version="1.3.9"; loaded version="1.3.9"
PCRE compiled version="7.8 "; loaded version="7.8 2008-09-05"
LIBXML compiled version="2.7.6"
Thanks in advance.
JSON inspection is not compiled into modsecurity by default. You need to re-compile your modsecurity installation with libyajl JSON library to enable JSON support
this link shows how to do this
https://gist.github.com/rpfilomeno/1140359f4bd360137a98
Have you enabled SecStreamOutBodyInspection directive? And I presume it needs SecResponseBodyAccess enabled too.
Both are disabled by default so you can't inspect response body details. Be warned there can be quite a performance hit for scanning output (which is why it's off by default).
Related
I noticed something weird when calling my service on a URL like this
GET https://myservice.com//someresource
--header 'Content-Type: application/json' \
--header 'x-api-key:<somekey>' \
--header 'Authorization: Bearer <sometoken>
When I do this, I get a response from AWS gateway that includes in the response body
< HTTP/2 403
< content-type: application/json
< content-length: 3222
< x-amzn-requestid: 348fab78-b84d-4af9-88v9-e1e6effc487b
< x-amzn-errortype: IncompleteSignatureException
< x-amz-apigw-id: dgYbMFbhliAFv8w=
<
* Connection #0 to host api-aws-tst.reprisk.com left intact
{"message":"<sometoken>"}
Calling https://myservice.com/someresource , without the extra slash, works ok. I know that // is not a correct path but I would like that the response message reflect the fact that the path is wrong instead of returning my token in a 403 response.
Is there any setting to configure this behaviour?
Is there any setting to configure this behavior?
Yes. You can set up integration response mappings to modify the responses of API Gateway. In your case you need to have an integration response mapping for HTTP 403.
These docs explain how the mapping works and how to set them:
Set up an integration response in API Gateway
Use a mapping template to override an API's request and response parameters and status codes
I have a server that has a Nginx VTS module installed on it, which outputs metrics in prometheus format.
When I try to actively check web.page.get via Zabbix I get the HTTP header and then the data in the format below:
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 24 Sep 2020 09:16:20 GMT
Content-Type: text/plain
Content-Length: 33769
Connection: close
Vary: Accept-Encoding
# HELP nginx_vts_info Nginx info
# TYPE nginx_vts_info gauge
nginx_vts_info{hostname="example",version="1.18.0"} 1
# HELP nginx_vts_start_time_seconds Nginx start time
# TYPE nginx_vts_start_time_seconds gauge
nginx_vts_start_time_seconds 1600367492.145
# snip output...
I wrote a regular expression that removes the header but only outputs the first line:
# \n\s?\n(.*)
# HELP nginx_vts_info Nginx info
How do I rewrite the expression so that the header is removed and the rest of the data is available?
Please try below regex
\n\s?\n([\s\S]*)
in regex . wont check newlines unless specific flags set. hence in your example, only the first line was returned. so rewriting it to include newlines as well will help.
My Angular4 app (running on http://127.0.0.1:4200 development server) is supposed to access a django REST backend on the web. The backend is under my control and is available only via HTTPS (running Apache that tunnels the request to a gunicorn server running on an internal port). Let's say that this is https://example.com/. For historical reasons, logging the user in is done using sessions, because I want the users to be able to also use Django's admin interface after they logged in. The workflow is as follows:
Users opens http://127.0.0.1:4200, I perform a GET request to https://example.com/REST/is_logged_in which returns a 403 when the user isn't logged in via sessions yet, 200 otherwise. In the former case, the user is redirected to https://example.com/login/, rendered by Django's template engine, allowing the user to log in. Once logged in, the user is redirected to http://127.0.0.1:4200
When clicking on some button in my Angular UI, a POST request is performed. This post request fails with 403, even though the preflight OPTIONS request explicitly lists POST as allowed actions.
Here is my CORS configuration in Django:
NG_APP_ABSOLUTE_URL = 'http://127.0.0.1:4200'
# adapt Django's to Angular's presumed XSRF cookie/header names
CSRF_COOKIE_NAME = "XSRF-TOKEN"
CSRF_HEADER_NAME = "HTTP_X_XSRF_TOKEN"
CORS_ORIGIN_WHITELIST = (
urlparse(NG_APP_ABSOLUTE_URL).netloc
)
CSRF_TRUSTED_ORIGINS = (
urlparse(NG_APP_ABSOLUTE_URL).netloc
)
CORS_ALLOW_HEADERS = default_headers + (
'x-xsrf-token',
)
CORS_ALLOW_CREDENTIALS = True
This is what Chrome reports for the (successful, 200) first REST GET request to check whether the user is logged in (after he successfully did) in the response:
Access-Control-Allow-Credentials:true
Access-Control-Allow-Origin:http://127.0.0.1:4200
Allow:GET, HEAD, OPTIONS
Connection:close
Content-Type:application/json
Date:Wed, 26 Apr 2017 15:09:26 GMT
Server:gunicorn/19.6.0
Set-Cookie:XSRF-TOKEN=...; expires=Wed, 25-Apr-2018 15:09:26 GMT; Max-Age=31449600; Path=/
Transfer-Encoding:chunked
Vary:Accept,Cookie,Origin
X-Frame-Options:SAMEORIGIN
The corresponding request had this:
Cookie:sessionid=...; XSRF-TOKEN=...
Host:example.com
Origin:http://127.0.0.1:4200
Referer:http://127.0.0.1:4200/
Now, to the actual problem:
Preflight request:
Request URL:https://example.com/REST/change_user_data/
Request Method:OPTIONS
Status Code:200 OK
Access-Control-Request-Headers:content-type
Access-Control-Request-Method:POST
Connection:keep-alive
Host:example.com
Origin:http://127.0.0.1:4200
Referer:http://127.0.0.1:4200/dashboard/account
Preflight response:
Access-Control-Allow-Credentials:true
Access-Control-Allow-Headers:accept, accept-encoding, authorization, content-type, dnt, origin, user-agent, x-csrftoken, x-requested-with, x-xsrf-token
Access-Control-Allow-Methods:DELETE, GET, OPTIONS, PATCH, POST, PUT
Access-Control-Allow-Origin:http://127.0.0.1:4200
Access-Control-Max-Age:86400
Connection:close
Content-Length:0
Content-Type:text/html; charset=utf-8
Date:Wed, 26 Apr 2017 15:36:56 GMT
Server:gunicorn/19.6.0
Vary:Origin
X-Frame-Options:SAMEORIGIN
Now my failing (403) POST request:
Accept:application/json
Accept-Encoding:gzip, deflate, br
Accept-Language:de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Content-Length:60
Content-Type:application/json
Cookie:sessionid=...; XSRF-TOKEN=...
Host:example.com
Origin:http://127.0.0.1:4200
Referer:http://127.0.0.1:4200/dashboard/account
The response headers:
HTTP/1.1 403 Forbidden
Date: Wed, 26 Apr 2017 15:36:56 GMT
Server: gunicorn/19.6.0
Vary: Accept,Cookie,Origin
X-Frame-Options: SAMEORIGIN
Content-Type: application/json
Access-Control-Allow-Credentials: true
Allow: POST, OPTIONS
Access-Control-Allow-Origin: http://127.0.0.1:4200
Set-Cookie: XSRF-TOKEN=...; expires=Wed, 25-Apr-2018 15:36:56 GMT; Max-Age=31449600; Path=/
Connection: close
Transfer-Encoding: chunked
Why wouldn't this request work? It makes little sense to me!
Best regards!
I had the same problem, trying to send a POST request to Django (port 8000) from my Angular CLI (port 4200). I thought it was a problem of Django so I installed cors package however the "problem" is with the browser (actually is not a problem, it is a security issue, see here). Anyway, I solved the problem adding a proxy rule for my Angular CLI, as follows:
First, instead of sending my requests to http://localhost:8000/api/... is send them to /api/ (i.e. to my ng server running at port 4200).
Then I added a file in my Angular project called "proxy.conf.json" with the following content:
{
"/api": {
"target": "http://localhost:8000",
"secure": false
}
}
Finally, run your ng server with the flag "--proxy-config":
ng serve --watch --proxy-config proxy.conf.json
All API requests will be sent to the port 4200 and Angular will internally redirect them to Django, avoiding the CORS problem.
Note that this is only valid for development and won't be used when you build your app code and add it as the static code of your Django server.
Finally, with this solution I didn't need anymore the python module for cors so you could remove it.
I have a Django view that process POST request with random size(between 20 char to 30k char). This API is only available for registered users and is validated with a session header. The API works well with my test cases but I notice some 502 in the Nginx log. The error log show this line::
2016/12/26 19:53:15 [error] 1048#0: *72 sendfile() failed (32: Broken pipe) while sending request to upstream, client: XXX.XXX.XXX.XXX, server: , request: "POST /api/v1/purchase HTTP/1.1", upstream: "uwsgi://unix:///opt/project/sockets/uwsgi.sock:", host: "staging.example.com"
After some tests, I managed to recreate this call with a big body request.
curl -XPOST https://staging.example.com/api/v1/purchase \
-H "Accept: application/json" \
-H "token: development-token" \
-H "session: bad-session" \
-i -d '{"receipt-data": "<25677 character string>"}'
HTTP/1.1 100 Continue
HTTP/1.1 502 Bad Gateway
Server: nginx/1.4.6 (Ubuntu)
Date: Mon, 26 Dec 2016 19:54:32 GMT
Content-Type: text/html
Content-Length: 181
Connection: keep-alive
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>
What it seems to happen is that the Django checks that the session is not valid and return the response(403) before the client finish delivers the body.
If I'm correct, is there a way to make Django send that 100 status after checking the headers instead of the Nginx?
If not, is there a more elegant solution than wait for the body before checking the headers?
I've found a statement that adding HTTP header connection:keep-alive to the client should fix this issue. I'll verify it later, but already posting it here, hope it will help someone.
When I make a post request using the following
curl -i -d "username=rock&password=rock" http://my_VM_IP/api/1.1/json/my_login/
it generates the required response generating a token like this(abridged):
HTTP/1.1 200 OK
Date: Mon, 22 Oct 2012 08:37:39 GMT
Vary: Authorization,Accept-Language,Cookie,Accept-Encoding
Content-Type: text/plain
Transfer-Encoding: chunked
OK{"success": {"my_token": "required_token"}}
But when I try the same including a header as:
curl -i -H "Content-Type:application/json" -d "username=rock&password=rock" http://my_VM_IP/api/1.1/json/my_login/
it gives me the following error:
HTTP/1.1 400 BAD REQUEST
Date: Mon, 22 Oct 2012 11:12:04 GMT
Vary: Authorization,Accept-Language,Cookie,Accept-Encoding
***Content-Type: text/plain***
Content-Language: en-us
Connection: close
Transfer-Encoding: chunked
Bad Request
I dont understand why this happens. And also why does the content-Type show text/plain, I also tried looking at some other questions like Why Setting POST Content-type:"Application/Json" causes a "Bad Request" on REST WebService? . It also addresses the same problem I have. Following the answer in that I tried giving the data in various formats as
{"username":"rock", "password":"rock"}
but without success. Thanks in advance.
By using -H "Content-Type:application/json" you're setting the Content-Type header for your request. The response will still return whatever your view tells it to return.
To return a response with Content-Type application/json, use something along these lines:
import json
from django.http import HttpResponse
def json_response(return_vars):
'JSON-encodes return_vars returns it in an HttpResponse with a JSON mimetype'
return HttpResponse(json.dumps(return_vars), content_type = "application/json")
#Usage: return json_response({'admin_token': admin_api_token.token})
You were close, but you need to send it as a JSON format via curl:
curl -i -H "Content-Type:application/json" -d '{"username":"rock", "password":"rock"}'
("password","admin" should be "password":"admin")
If that's not working, try:
curl --dump-header - -H "Accept:application/json" -H "Content-Type:application/json" -X POST --data '{"username": "admin", "password": "admin"}' http://my_VM_IP/api/1.1/json/my_login/
When you set -H parameter of curl command, you specify content type of request. Content type of response, that you see in response, is set on the server. In WSGI application you need to specify 'content-type' and 'content-length' manually. Some of framework provide utility method to return JSON responses (for example, jsonify method in Flask).