I am using nginx as a proxy server to send the requests to backend pool members .
I have a header -- Content-Type: multipart/form-data; boundary="21ad608c-8e00-4c0c-8a06-5ed1280e9532"
The boundary value in the header changes all the time.
How can I do a header check for only "Content-Type: multipart/form-data;" value and route the requests to the backend. when I do a print of $http_content_type value , everything including boundary value is getting printed . I am not able to do an if check on just " Content-Type: multipart/form-data; " value .
Can I put a wild card in the nginx config like -- "multipart/form-data; * " to pass the if loop ?
Please let me know.
An Nginx wildcard? Yes, sort of, via a regex:
if ($http_content_type !~ '^multipart/form-data;') deny-the-request
This means that the Content-Type must start with multipart/form-data;. After that, any boundary=...whatever... is allowed.
Complete example:
location /-/upload-public-file {
limit_except POST OPTIONS {
deny all;
}
if ($http_content_type !~ '^multipart/form-data;') {
return 406; # status code Not acceptable
}
Related
Consider my requested url is www.example.com/foo/emplooyee?names = test1;test2.
and varnish stores this entire URL along with query parameters to uniquely identify the cache.
now, in my backend, I'm running one service and which I'm supposed to configure as whenever there are changes in names (i.e. test1 or test2) is should fire an HTTP ban with an older name (single name at a time in ban expression) to invalidate all the cached URL which entered with similar names.
Questions:
My client request url could be like this,
www.example.com/foo/emplooyee?names = test1;test2
www.example.com/foo/emplooyee?names = test1;
www.example.com/foo/emplooyee?names = test2;test1;test3;test4
www.example.com/foo/emplooyee?names = test1;test4.
How to write a VCL code and in Ban expression to invalidate all object which has query parameter as test1?
This is the VCL code you need for banning:
vcl 4.1;
acl purge {
"localhost";
"192.168.55.0"/24;
}
sub vcl_recv {
if (req.method == "PURGE") {
if (!client.ip ~ purge) {
return(synth(405));
}
if(!req.http.x-invalidate-pattern) {
return(purge);
}
ban("obj.http.x-url ~ " + req.http.x-invalidate-pattern
+ " && obj.http.x-host == " + req.http.host);
return (synth(200,"Ban added"));
}
}
sub vcl_backend_response {
set beresp.http.x-url = bereq.url;
set beresp.http.x-host = bereq.http.host;
}
sub vcl_deliver {
unset resp.http.x-url;
unset resp.http.x-host;
}
Here's an example HTTP request to invalidate all requests that contain a test1 value in the query string:
PURGE / HTTP/1.1
Host: www.example.com
X-Invalidate-Pattern: ^/foo/emplooyee\?names=([^;]+;)*test1(;[^;]+)*$
Here's the same request via curl:
curl -XPURGE -H'X-Invalidate-Pattern: ^/foo/emplooyee\?names=([^;]+;)*test1(;[^;]+)*$' http://www.example.com
This VCL snippet has the flexibility to remove multiple items from cache through banning, but if you don't set the pattern through the X-Invalidate-Pattern header, it will just remove the URL itself from cache.
Here's an example where we just remove http://www.example.com/foo/emplooyee?names=test1 from the cache:
curl -XPURGE 'http://www.example.com/foo/emplooyee?names=test1'
Don't forget to modify the acl block in the VCL code and add the right IP addresses or IP ranges.
I have a website using aspnetcore 2.1 and Kestrel (IIS as proxy) and someone else on our domain sets a cookie that contains non-ASCII characters. I know this is wrong, but I can't make sure they don't and I still want the cookie (otherwise it is easy just to remove it in my web.config).
The result being: Cookie is set incorrectly on one site and when the user "unknowingly" goes to our website, our website gives an error 400 on all requests.
Is there a way of removing the bad characters in the cookie header before it hits Kestrel? I thought I could use a HttpModule/Middleware/etc to remove it, but it looks like Kestrel is the first one to get a HttpContext.
No workaround at the side of dotnetcore. If you are using nginx behind of your dotnetcore application, you can delete all of the non-ascii character like that:
set_by_lua_block $cookie_ascii {
local cookie = ngx.var.http_cookie
if cookie == nil or cookie == '' then return cookie end
local cookie_ascii, n, err = ngx.re.gsub(cookie, "[^\\x00-\\x7F]", "")
return cookie_ascii
}
listen 80;
server_name example.com;
location / {
proxy_pass http://localhost:5000;
...
proxy_set_header Cookie $cookie_ascii;
...
}
}
So on the Nginx documentation it is a bit unclear but it appears we can use Regex to match anything within the if statement
http://nginx.org/en/docs/http/ngx_http_rewrite_module.html#if
So i wanted to tweak this to make it so Nginx checks the value of any cookie.
But my following string what should be from my understanding * matching any cookie name and = matching any cookie contents.
if ($http_cookie ~* "*=*") {
return 444;
}
But i get the error "pcre_compile() failed: nothing to repeat in "" at "" in"
What i am trying to achieve is to Have nginx check the cookie like a WAF (web application firewall) to make sure it only contains A-Z uppercase a-z lowercase 0-9 + - _ . my PHP app is Joomla what does these checks too but be useful if i could perform these checks with Nginx too since it could deny the request faster.
EDIT to Show half solved issue / dilemma
set $block_cookie_exploits 0;
#If cookie name or contents does not contain the following
if ($http_cookie !~ "[a-zA-Z0-9\+\-\_]=[a-zA-Z0-9\+\-\_]") {
set $block_cookie_exploits 1;
}
#Block the client request
if ($block_cookie_exploits = 1) {
return 403;
}
New problem with above configuration is it will return 403 while no cookie is present. And if you put characters in the name or contents of the cookie like {} it does not return 403
I'm just getting started using the REST API to create pages.
I'm trying to configure a basic example and I thought of using libcurl.net to do it.
Does anyone see any reason why that wouldn't work?
UPDATE:
Here is what I have so far adapted from the curllib.net "bookpost" example
Option Explicit On
Imports System.IO
Imports System.Net
Imports SeasideResearch.LibCurlNet
Public Class CurlOperations
Public Shared Sub CurlPost()
Try
Dim credUserName As String = "username"
Dim credPassword As String = "password"
Dim response As String = Nothing
Dim outputStdErr As Stream = Nothing
Curl.GlobalInit(CURLinitFlag.CURL_GLOBAL_ALL)
Dim easy As Easy
easy = New Easy
' Set up write delegate
Dim wf As Easy.WriteFunction
wf = New Easy.WriteFunction(AddressOf OnWriteData)
easy.SetOpt(CURLoption.CURLOPT_WRITEFUNCTION, wf)
'Authentication
easy.SetOpt(CURLoption.CURLOPT_HTTPAUTH, CURLhttpAuth.CURLAUTH_BASIC)
easy.SetOpt(CURLoption.CURLOPT_USERPWD, credUserName & ":" & credPassword)
'disable ssl peer verification
easy.SetOpt(CURLoption.CURLOPT_SSL_VERIFYPEER, False)
'Header
easy.SetOpt(CURLoption.CURLOPT_HTTPHEADER, "Content-Type: application/json; charset=utf-8")
' Simple post - with a string
easy.SetOpt(CURLoption.CURLOPT_POSTFIELDS, WikiTools.CommREST.WebToCF.PostCurl())
' and the rest of the cURL options
easy.SetOpt(CURLoption.CURLOPT_USERAGENT, ".NET Framework Client")
easy.SetOpt(CURLoption.CURLOPT_URL, "https://domain.com/confluence/rest/api/content/")
easy.SetOpt(CURLoption.CURLOPT_POST, True)
response = easy.Perform().ToString
LoggingAndActivites.WriteLog("Response: " & response, GetFunctionName.GetCallerName, True, True)
Catch ex As Exception
LoggingAndActivites.WriteLog("Exception: " & ex.ToString, GetFunctionName.GetCallerName, True, True)
End Try
Curl.GlobalCleanup()
End Sub
' Called by libcURL.NET when it has data for your program
Public Shared Function OnWriteData(ByVal buf() As Byte, ByVal size As Int32, ByVal nmemb As Int32, ByVal extraData As Object) As Int32
LoggingAndActivites.WriteLog(System.Text.Encoding.UTF8.GetString(buf), GetFunctionName.GetCallerName, True, True)
Return size * nmemb
End Function
End Class
I am getting connected because if I remove the username and password I get a response through the "onWriteData" function as follows:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>
<head>
<title>401 Unauthorized</title>
</head>
<body>
<h1>Unauthorized</h1>
<p>This server could not verify that you
are authorized to access the document
requested. Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.</p>
<hr>
<address>Apache Server at domain.com Port 7080</address>
</body>
</html>
The problem now is that if I correctly log on I'm not getting any response other than the "CURLE_OK" from the "easy.Perform()" function.
It's good because I know it's working to some degree.
According to the libcurl.net docs : http://www.libcurl.org/
libcurl also supports HTTPS certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, and user+password authentication.
So I guess you should be able to make a REST API call with it. I have used curl (the linux version) to create and update pages, using something like this:
curl --globoff --insecure --silent -u username:password -X PUT -H 'Content-Type: application/json' --data #body.json confluenceRestAPIURL/pageId
where body.json is a file containing the data to update the page.
I wrote a blog about this here: https://javamemento.blogspot.no/2016/05/jira-confluence-3.html
You can get the code here: https://github.com/somaiah/jira-confluence-graphs
So it does work
Here is what I added/changed to make the code above work
'I added an Slist to store the header items (I only had one)
Dim slistHeaders As New Slist
slistHeaders.Append("Content-Type: application/json")
'Then I added the slist to the HTTPHEADER
easy.SetOpt(CURLoption.CURLOPT_HTTPHEADER, slistHeaders)
THINGS TO WATCH FOR:
(1) The URL of course is the number one thing
In my case I was using was https://domain.com/confluence/rest/api/content/ because the Confluence Documentation assumes you would be using the root domain name (as did I)
However, what I didn't know was that the URL I was given to test was already directing me into the "confluence" folder.
So my URI needed to be https://domain.com/rest/api/content/ instead
(2) An indicator that your HTTPHEADER needs to be put into an Slist is this return from the server: 415 Unsupported Media Type
(3) Be sure NOT to use the CURLOPT_HEADER property. If you are seeing this header in your responses you need to make sure it's not used:
HTTP/1.1 500 Internal Server Error
Date: Sun, 22 May 2016 17:50:32 GMT
Server: Apache
Content-Location: 500.en-GB.html
Vary: negotiate,accept-language
TCN: choice
Accept-Ranges: bytes
Content-Length: 7575
Connection: close
Content-Type: text/html; charset=UTF-8
Content-Language: en-gb
Refer to my post here for an explanation of why: CURLOPT_HEADER
(4) Lastly, when you build your app if you receive this error:
An unhandled exception of type 'System.AccessViolationException' occurred in libcurl.NET.dll
Additional information: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
This is caused by libcurl.net process not being cleaned up.
The "cleanup()" method isn't available in the DLL, despite being lsited in the examples.
So instead use EASY.Dispose() at the end of your procedure and this error will stop. (I kept the "GlobalCleanup()" method in just for good measure)
(5) Ironically I went the way of using CURL because I thought that the Confluence interface might require it.
But it appears now that it doesn't and that you can simply use the "HttpWebRequest" Class in .NET to get the same results.
However, the Jury is still out because the "lightweight" test server I was assigned to crashed and I'm waiting for them to fix it so I can verify this.
Either Way I hope all this helps someone
M
Ok, so my new problem in Elixir is that I can't set the explicit domain while creating cookies.
In this case:
HTTPoison.get("httpbin.org/cookies", [{"User-agent", #userAgent}], hackney: [
cookie: "cookie1=1 cookie2=2"] ) do
When I create a cookie it will store a domain like .httpbin.org but for dummy reason I need to set domain value like httpbin.org (without previous dot) .
I tried also with:
HTTPoison.get("httpbin.org/cookies", [{"User-agent", #userAgent}], hackney: [
cookie: "cookie1=1 domain=httpbin.org cookie2=2"] ) do
But of course the syntax expects domain as a cookie name and httpbin.org as a cookie value.
Thank you!
What's the reason you want to remove the dot in the beginning? It's optional and it should match the entire domain with/without the dot.
How do browser cookie domains work?
Also, I think the domain attribute would be for the Set-Cookie header returned from HTTP server rather than requesting from the client. The httpbin (https://httpbin.org/cookies/set) returns the Set-Cookie header, but it doesn't specify domain attribute (just Path=/). It would be taken as .httpbin.org by clients like browsers.
iex(25)> response = HTTPoison.get!("https://httpbin.org/cookies/set?k2=v2&k1=v1")
%HTTPoison.Response{body: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>Redirecting...</title>\n<h1>Redirecting...</h1>\n<p>You should be redirected automatically to target URL: /cookies. If not click the link.",
headers: [{"Server", "nginx"}, {"Date", "Fri, 18 Dec 2015 23:49:46 GMT"},
{"Content-Type", "text/html; charset=utf-8"}, {"Content-Length", "223"},
{"Connection", "keep-alive"}, {"Location", "/cookies"},
{"Set-Cookie", "k2=v2; Path=/"}, {"Set-Cookie", "k1=v1; Path=/"},
{"Access-Control-Allow-Origin", "*"},
{"Access-Control-Allow-Credentials", "true"}], status_code: 302}
iex(26)> :hackney.cookies(response.headers)
[{"k1", [{"k1", "v1"}, {"Path", "/"}]}, {"k2", [{"k2", "v2"}, {"Path", "/"}]}]
Sorry if I'm missing the point.