I'm having trouble redirecting several domains & associated sub-domains to one other domain.
Keep falling into a 301 infinite loop.
I have 3 domains, proxied to the same lighttpd process, say :
dom.co
dom.info
dom.net
dom.net is my domain of choice, what I want is to get every one, including www.*, going to my domain of choice. (dom.net is working)
My lighttpd.conf insteresting parts :
$HTTP["host"] =~ "(^|\.)dom\.net$" {
/* working */
}
$HTTP["host"] =~ "(^|\.)dom\.co$" {
url.redirect = ( "^/(.*)" => "dom.net/$1" )
}
the log :
IP dom.co - [16/Nov/2012:20:51:33 +0100] "GET /dom.net/ HTTP/1.0" 301 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11"
IP dom.co - [16/Nov/2012:20:51:33 +0100] "GET /dom.net/dom.net/dom.net/ HTTP/1.0" 301 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11"
et caetera.
I understand what's happening, not how to fix it. Please help !
I went with this :
$HTTP["host"] =~ "^dom\.co" {
/* ... */
}
# some redirections to dom.co
$HTTP["host"] =~ "dom.net|dom.info|www.dom.net|www.dom.info|www.dom.co" {
url.redirect = ( "^/(.*)" => "http://dom.co/$1" )
}
I think the absence of http:// in front of the target url was what caused trouble, didn't tweak and try things around though, it's working now.
Idea comes from https://serverfault.com/questions/105920/how-do-i-redirect-multiple-domains-to-a-single-domain-in-lighttpd
Related
I try to count the access on a specific URL which begins every time with "shop/product?traffic=ads" with AWK, but I failed.
The following code gives me a counter how often an IP address has accessed these URL:
awk -F'[ "]+' '$7 == "/shop/product?traffic=ads" { ipcount[$1]++ }
END { for (i in ipcount) {
printf "%15s - %d\n", i, ipcount[i] } }' /var/www/vhosts/domain.com/logs/access_ssl_log
An example for the access_log (input-file) is here:
66.249.68.xx- - [19/Dec/2022:09:14:15 +0100] "GET /shop/other-product/1.0" 404 16996 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.5304.xxx Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
109.42.242.xxx - - [19/Dec/2022:09:14:55 +0100] "GET /shop/product?traffic=ads&gclid=Cj0KCQiAtICdBhCLARIsALUBFcFMmvFbA_1EyTTMRDp9IWhDXFA_HCeuEsIBXl886PoaAapen2KdussaAniSEALw_wcB HTTP/1.0" 200 30589 "https://www.google.com/" "Mozilla/5.0 (Linux; Android 11; SM-A515F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Mobile Safari/537.36"
80.187.75.xx - - [20/Dec/2022:06:40:12 +0100] "GET /shop/product HTTP/1.0" 200 10821 "https://www.example.com/shop/product?traffic=ads&gclid=EAIaIQobChMIg_Ks5vWF_AIVAgGLCh3k_gBKEAAYASAAEgKBOfD_BwE&dt=1671461107791" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_0_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Mobile/15E148 Safari/604.1"
The "gclid" and and the "dt"(session cookie) are dynamic.
I try to play with ^ after ads, before /shop, but there will be no results.
I want for example the following output:
6 Clicks from 109.42.242.xxx to /shop/product?traffic=ads&gclid=Cj0KCQiAtICdBhCLARIsALUBFcFMmvFbA_1EyTTMRDp9IWhDXFA_HCeuEsIBXl886PoaAapen2KdussaAniSEALw_wcB
1 Clicks from 80.187.75.xx to https://www.example.com/shop/product?traffic=ads&gclid=EAIaIQobChMIg_Ks5vWF_AIVAgGLCh3k_gBKEAAYASAAEgKBOfD_BwE&dt=1671461107791"
You can check if the string occurs in field 7 using index(), and then store the values of field 1 and field 7 with a space in between as the key, to retrieve both values in the END block by splitting on a space again.
awk -F'[ "]+' 'index($7, "/shop/product?traffic=ads") { ipcount[$1 " " $7]++ }
END { for (i in ipcount) {
parts = split(i, a, " ")
printf ipcount[i] " Clicks from " a[1] " to " a[2] "\n"
}
}' file
Test data
66.249.68.xx- - [19/Dec/2022:09:14:15 +0100] "GET /shop/other-product/1.0" 404 16996 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.5304.xxx Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
109.42.242.xxx - - [19/Dec/2022:09:14:55 +0100] "GET /shop/product?traffic=ads&gclid=Cj0KCQiAtICdBhCLARIsALUBFcFMmvFbA_1EyTTMRDp9IWhDXFA_HCeuEsIBXl886PoaAapen2KdussaAniSEALw_wcB HTTP/1.0" 200 30589 "https://www.google.com/" "Mozilla/5.0 (Linux; Android 11; SM-A515F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Mobile Safari/537.36"
109.42.242.xxx - - [19/Dec/2022:09:15:55 +0100] "GET /shop/product?traffic=ads&gclid=Cj0KCQiAtICdBhCLARIsALUBFcFMmvFbA_1EyTTMRDp9IWhDXFA_HCeuEsIBXl886PoaAapen2KdussaAniSEALw_wcB HTTP/1.0" 200 30589 "https://www.google.com/" "Mozilla/5.0 (Linux; Android 11; SM-A515F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Mobile Safari/537.36"
80.187.75.xx - - [20/Dec/2022:06:40:12 +0100] "GET /shop/product HTTP/1.0" 200 10821 "https://www.example.com/shop/product?traffic=ads&gclid=EAIaIQobChMIg_Ks5vWF_AIVAgGLCh3k_gBKEAAYASAAEgKBOfD_BwE&dt=1671461107791" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_0_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Mobile/15E148 Safari/604.1"
109.42.242.xxx - - [19/Dec/2022:09:15:55 +0100] "GET /shop/product?traffic=ads&gclid=Aj0KCQiAtICdBhCLARIsALUBFcFMmvFbA_1EyTTMRDp9IWhDXFA_HCeuEsIBXl886PoaAapen2KdussaAniSEALw_wcB HTTP/1.0" 200 30589 "https://www.google.com/" "Mozilla/5.0 (Linux; Android 11; SM-A515F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Mobile Safari/537.36"
Output
1 Clicks from 109.42.242.xxx to /shop/product?traffic=ads&gclid=Aj0KCQiAtICdBhCLARIsALUBFcFMmvFbA_1EyTTMRDp9IWhDXFA_HCeuEsIBXl886PoaAapen2KdussaAniSEALw_wcB
2 Clicks from 109.42.242.xxx to /shop/product?traffic=ads&gclid=Cj0KCQiAtICdBhCLARIsALUBFcFMmvFbA_1EyTTMRDp9IWhDXFA_HCeuEsIBXl886PoaAapen2KdussaAniSEALw_wcB
With your shown samples please try following awk code. Using match function to match regex \/shop\/product\?traffic=ads\S+(where escaped / to match literal /) and if match is found then creating an array value with index of $1 FS and matched value. In the END block of this program printing the values as per requirement.
awk '
match($7,/\/shop\/product\?traffic=ads\S+/){
value[$1 FS substr($7,RSTART,RLENGTH)]++
}
END{
for(i in value){
split(i,arr)
print value[i] " Clicks from " arr[1] " to " arr[2]
}
}
' Input_file
I'm trying to protect a Django application with oauth2-proxy
In the oauth2-proxy configuration: (version 7.2.1 or 7.3.0)
When the upstream url is set to something like this: --upstream="http://127.0.0.1:8000"
the redirection works fine. (and it returns a home page I have defined in the application )
But, if I use an upstream like this: --upstream="http://127.0.0.1:8000/hello"
it returns 404 error instead of the hello page that is also defined in the application
The page http://127.0.0.1:8000/hello is working fine when invoked directly and it returns "GET /hello HTTP/1.1" 200 136
So I would say it is not a problem with the page.
This is the command line I'm using:
oauth2-proxy.exe ^
--http-address=127.0.0.1:4180 ^
--email-domain=* ^
--cookie-secure=false ^
--cookie-secret=adqeqpioqr809718 ^
--upstream="http://127.0.0.1:8000/hello" ^
--redirect-url=http://127.0.0.1:4180/oauth2/callback ^
--oidc-issuer-url=http://127.0.0.1:28081/auth/realms/testrealm ^
--insecure-oidc-allow-unverified-email=true ^
--provider=keycloak-oidc ^
--client-id=oauth2_proxy ^
--ssl-insecure-skip-verify=true ^
--client-secret=L2znXLhGX4N0j3nsZYxDKfdYpXHMGDkX ^
--skip-provider-button=true
When the oauth2-proxy succeeds to redirect (--upstream="http://127.0.0.1:8000"), I get the page and the following output:
This is the output for the oauth2-proxy:
[2022/09/08 10:52:06] [proxy.go:89] mapping path "/" => upstream "http://127.0.0.1:8000"
[2022/09/08 10:52:06] [oauthproxy.go:148] OAuthProxy configured for Keycloak OIDC Client ID: oauth2_proxy
[2022/09/08 10:52:06] [oauthproxy.go:154] Cookie settings: name:_oauth2_proxy secure(https):false httponly:true expiry:168h0m0s domains: path:/ samesite: refresh:disabled
[2022/09/08 10:57:01] [oauthproxy.go:866] No valid authentication in request. Initiating login.
127.0.0.1:54337 - 9bbfcf75-da91-487a-a55e-40472e4adb23 - - [2022/09/08 10:57:01] 127.0.0.1:4180 GET - "/" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.27" 302 380 0.001
127.0.0.1:54337 - e0d8ed12-e4dd-4da6-9fbb-cf689fc53f8f - mail#gmail.com [2022/09/08 10:57:09] [AuthSuccess] Authenticated via OAuth2: Session{email:mail#gmail.com user:93547bcc-2441-414a-9149-c7533c4f5d23 PreferredUsername:testuser token:true id_token:true created:2022-09-08 10:57:09.789934 -0300 -03 m=+303.019857301 expires:2022-09-08 11:02:09.7839238 -0300 -03 m=+603.013847101 refresh_token:true groups:[role:offline_access role:uma_authorization role:default-roles-testrealm role:account:manage-account role:account:manage-account-links role:account:view-profile]}
[2022/09/08 10:57:09] [session_store.go:163] WARNING: Multiple cookies are required for this session as it exceeds the 4kb cookie limit. Please use server side session storage (eg. Redis) instead.
127.0.0.1:54337 - e0d8ed12-e4dd-4da6-9fbb-cf689fc53f8f - - [2022/09/08 10:57:09] 127.0.0.1:4180 GET - "/oauth2/callback?state=ahuKzCYr7jR4P4mmjniIt67TttZKyxGv4mLfEwKlQio%3A%2F&session_state=86ac9bd1-9756-4916-83e9-ec0496b5b767&code=df3940e5-58f5-49ac-a821-5607f0f2faae.86ac9bd1-9756-4916-83e9-ec0496b5b767.cd30a162-8e4d-4a2d-bff6-168e444aed92" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.27" 302 24 0.029
127.0.0.1:54337 - d58ace6e-afe9-4737-9b12-dbc17fdd0ca2 - mail#gmail.com [2022/09/08 10:57:09] 127.0.0.1:4180 GET / "/" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.27" 200 138 0.005
On the Django side I get:
**"GET / HTTP/1.1" 200 138**
When the oauth2-proxy fails to redirect --upstream="http://127.0.0.1:8000/hello"), I get the following output:
This is the output for the oauth2-proxy:
[2022/09/08 10:33:58] [proxy.go:89] mapping path "/hello" => upstream "http://127.0.0.1:8000/hello"
[2022/09/08 10:33:58] [oauthproxy.go:148] OAuthProxy configured for Keycloak OIDC Client ID: oauth2_proxy
[2022/09/08 10:33:58] [oauthproxy.go:154] Cookie settings: name:_oauth2_proxy secure(https):false httponly:true expiry:168h0m0s domains: path:/ samesite: refresh:disabled
[2022/09/08 10:37:20] [oauthproxy.go:866] No valid authentication in request. Initiating login.
127.0.0.1:53615 - 54c0f3d8-b3c0-4d48-8353-fe69be0e4500 - - [2022/09/08 10:37:20] 127.0.0.1:4180 GET - "/" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.27" 302 380 0.001
127.0.0.1:53615 - 0bec934e-05a3-4cc8-9306-fffc28597c8f - mail#gmail.com [2022/09/08 10:37:28] [AuthSuccess] Authenticated via OAuth2: Session{email:mail#gmail.com user:93547bcc-2441-414a-9149-c7533c4f5d23 PreferredUsername:testuser token:true id_token:true created:2022-09-08 10:37:28.6527488 -0300 -03 m=+210.486252601 expires:2022-09-08 10:42:28.6468518 -0300 -03 m=+510.480355601 refresh_token:true groups:[role:offline_access role:uma_authorization role:default-roles-testrealm role:account:manage-account role:account:manage-account-links role:account:view-profile]}
[2022/09/08 10:37:28] [session_store.go:163] WARNING: Multiple cookies are required for this session as it exceeds the 4kb cookie limit. Please use server side session storage (eg. Redis) instead.
127.0.0.1:53615 - 0bec934e-05a3-4cc8-9306-fffc28597c8f - - [2022/09/08 10:37:28] 127.0.0.1:4180 GET - "/oauth2/callback?state=nox0LM3fIlVU1kamoLBaktByeLCcIWiBvRLdHFIuhd4%3A%2F&session_state=808c0654-c9e7-4593-b5dc-95d3231438ea&code=e220414d-e949-4e2d-8d33-55de96f8f5d4.808c0654-c9e7-4593-b5dc-95d3231438ea.cd30a162-8e4d-4a2d-bff6-168e444aed92" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.27" 302 24 0.024
127.0.0.1:53615 - 9454773f-cade-46fe-870f-70d09fc49ffb - mail#gmail.com [2022/09/08 10:37:28] 127.0.0.1:4180 GET - "/" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.27" 404 19 0.000
On the Django side I get:
Nothing. As the Django app is never reached and so there are no logs.
Could you please help me find out what could be happening? I will really appreciate it!!
It doesn't seem to be a problem with the application, as the pages work fine when invoked directly.
If it is a mistake in my oauth2-proxy command line/configuration, I would appreciate someone points me to the error, so I can correct it.
Otherwise, any hint would also be much appreciated.
The only thing I've noticed in the logs of oauth2-proxy is that no matter what I put in the --upstream, the final GET (I think it is the redirection to the upstream) is as follows: GET - "/" ... it is the same in both attempts, and it only succeeds in the first one, because it matches the [proxy.go:89] mapping path "/"
The reason it was giving the 404 error, was that the configuration --upstreams points to a url to which the proxy is going to pass the request once authenticated, but it is not going to redirect to that address unless you specifically ask for it in the original request.
So the correct way of making the request is http://127.0.0.1:4180/hello, which is including the whole path to the endpoint you want to reach. (instead of for example http://127.0.0.1:4180 )
I have the following line from which I want to replace space with whitespace (tab) but want to keep the spaces within the double quotes as it is. I am on Notepad++.
[11/May/2020:10:10:20 -0400] "GET / HTTP/1.1" 302 523 52197 url.com - - TLSv1.2 19922 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36" https://somelinkhere - -
Desired output:
[11/May/2020:10:10:20 -0400] "GET / HTTP/1.1" 302 523 52197 url.com - - TLSv1.2 19922 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36" https://somelinkhere - -
Through the following regex I was able to select the string under the double quotes, but it's of no use for me.
"([^"]*)"
Can you please help me how this can be achieved?
You can use
("[^"]*")|[ ]
Replace with (?1$1:\t).
Details:
("[^"]*") - Capturing group 1: a ", then zero or more chars other than " and then a "
| - or
[ ] - matches a space (you may remove [ and ] here , they are used to make the space pattern visible in the answer).
See the demo screenshot:
Working on an input extractor issue with IIS logs using an "advanced" IIS login tool to collect more than the basic logs provide. It's adding double quotes and spaces to many of the fields and we are trying to us the extractor to correct this. This is the beginning of an example message:
2016-02-08 16:46:35.957 "SITE" "SOURCE" XX.XX.XX.XX GET /blah/etc/etc/file.ext - 80 - "XX.XX.XX.XX" "HTTP/1.1" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; yie11; rv:11.0) like Gecko"
We've already written an extractor to remove all the added quotes before running it through all the other extractors to populate the fields, etc., but we want to replace all spaces between the quotes with + before we do that to match the old logging style.
Can anyone point us in the right direction for this? The closest I've come so far is catching " " between SITE and SOURCE and replacing that using something like "([\s]*)". Result:
2016-02-08 16:46:35.957 "SITE+SOURCE" XX.XX.XX.XX GET /blah/etc/etc/file.ext - 80 - "XX.XX.XX.XX+HTTP/1.1+Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; yie11; rv:11.0) like Gecko"
I can't seem to only look for spaces between the quotes.
Any help would be greatly appreciated. Thanks.
Further Clarification. This portion of the string:
"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; yie11; rv:11.0) like Gecko"
Should be:
"Mozilla/5.0+(Windows+NT+6.1;+WOW64;+Trident/7.0;+yie11;+rv:11.0)+like+Gecko"
Everything else should remain the same as those are the only spaces inside of a quoted section of the string.
Is this even possible with regex?
I'm afraid that regular expressions are not the best tool for this. You basically have to "count" quotes to determine whether a space is within quotes or not.
You can try something like this (Python):
text = '2016-02-08 16:46:35.957 "SITE" "SOURCE" XX.XX.XX.XX GET /blah/etc/etc/file.ext - 80 - "XX.XX.XX.XX" "HTTP/1.1" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; yie11; rv:11.0) like Gecko"'
escaped = ""
count = 0
for c in text:
if c == '"':
count += 1
if c == " " and count % 2 == 1:
escaped += "+"
else:
escaped += c
Afterwards, escaped is this:
2016-02-08 16:46:35.957 "SITE" "SOURCE" XX.XX.XX.XX GET /blah/etc/etc/file.ext - 80 - "XX.XX.XX.XX" "HTTP/1.1" "Mozilla/5.0+(Windows+NT+6.1;+WOW64;+Trident/7.0;+yie11;+rv:11.0)+like+Gecko"
Why does
AliasMatch .*\.(png|ico|gif|jpg|jpeg|js|css|woff|ttf|svg)$ /my-location/
+
GET /pages/index/index.js HTTP/1.1
=
[30/Jul/2014:12:55:28 -0700] "GET /pages/index/index.js HTTP/1.1" 404 433 "http://localhost/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
?
The solution was
AliasMatch (.*\.(png|ico|gif|jpg|jpeg|js|css|woff|ttf|svg))$ /my-location/$1
The reason for that is:"[...] Alias will automatically copy any additional part of the URI, past the part that matched, onto the end of the file path on the right side, while AliasMatch will not. This means that in almost all cases, you will want the regular expression to match the entire request URI from beginning to end, and to use substitution on the right side." (http://httpd.apache.org/docs/2.2/mod/mod_alias.html#aliasmatch)