i need some help getting the server info from a curl call done in a shell script.
In my script, I iterate over several URLs in a list and do a cURL on each URL.
But I struggle getting the server information as it is not in a static position of the result of cURL.
>curl -I -s $temp
where, $temp is some arbitrary URL, e.g. example.org
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=604800
Content-Type: text/html
Date: Mon, 03 Feb 2014 14:35:39 GMT
Etag: "359670651"
Expires: Mon, 10 Feb 2014 14:35:39 GMT
Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT
Server: ECS (iad/19AB)
X-Cache: HIT
x-ec-custom-error: 1
Content-Length: 1270
Above the result for example.org.
Question: how do I extract the part where it says server?
The result should be such that
>echo $server
will yield (so basically the entire rest of the line after the "Server: ")
ECS (iad/19AB)
Thanks a bunch!
Using awk:
server=$(curl -I -s http://example.org | awk -F': ' '$1=="Server"{print $2}')
echo "$server"
ECS (cpm/F858)
OR else you can use grep -oP:
server=$(curl -I -s http://example.org | grep -oP 'Server: \K.+')
echo "$server"
ECS (cpm/F858)
Related
I created a test to fill my SNS dead letter queue to help me develop code to read from this queue. Long story short, I thought an HTTP error would be easiest to simulate failures, but surprisingly, they seem to be counted as success.
In case I am doing it wrong and for the benefit of anyone else who wants to try this out, here is my methodology. I created an HTTP/s endpoint specifically for this test using a bash one liner:
while true; do echo -e "HTTP/1.1 200 OK\n" | nc -Nl 9078; echo "" && date; done
So far so good. I decided that returning a 401 code might be easiest. Capturing a 401 page output with netcat:
HTTP/1.1 401 Unauthorized
Server: nginx/1.21.0
Date: Wed, 01 Sep 2021 12:22:03 GMT
Content-Type: text/html
Content-Length: 179
Connection: keep-alive
WWW-Authenticate: Basic realm="Restricted example.com"
Strict-Transport-Security: max-age=31536000
<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx/1.21.0</center>
</body>
</html>
I altered my one liner accordingly:
while true; do echo -e "$(cat 401error)\n" | nc -Nl 9078; echo "" && date; done
I verified that visiting this page in Firefox would pop up a password dialog.
Come test time, SNS blunders along and delivers the message without fear. The message never appears in the DLQ:
POST /poot/testingevent HTTP/1.1
x-amz-sns-message-type: Notification
x-amz-sns-message-id: REDACTED
x-amz-sns-topic-arn: REDACTED
x-amz-sns-subscription-arn: REDACTED
x-amz-sns-rawdelivery: true
Content-Length: 24
Content-Type: text/plain; charset=UTF-8
Host: example.com:9078
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
Accept-Encoding: gzip,deflate
{"401 for sure man": 11}
Wed Sep 1 12:25:31 UTC 2021
Does anyone know? Nothing so far uncovered in duckduckgoing "http code" sns. If I can capture some other codes (403,500,etc) using netcat, I thought it might be useful to know which, if any, are honored.
Any status code outside of the range 200 - 499 will be considered as a failure and retried according to your retry policy as per https://docs.aws.amazon.com/sns/latest/dg/sns-message-delivery-retries.html. Once the max number of retries has been exhausted, the message will be delivered to a DLQ if one is configured.
I have a string which is like:
Return-Path: bT.41aywtru20=krja5b54hplm=k29fsc7grl#fake.link.com
Received-SPF: pass (fake.link.com: Sender is authorized to use 'bt.41aywtru20=krja5b54hplm=k29fsc7grl#fake.link.com' in 'mfrom' identity (mechanism 'include:spf.smtp2go.com' matched)) receiver=pmxlab01.permission.email; identity=mailfrom; envelope-from="bt.41aywtru20=krja5b54hplm=k29fsc7grl#fake.link.com"; helo=e2i353.smtp2go.com; client-ip=103.2.141.97
Received: from e2i353.smtp2go.com (e2i353.smtp2go.com [103.2.141.97])
by mailserver.fake.com(Proxmox) with ESMTP id A4F983E1048
for <fake#fake.com>; Tue, 24 Aug 2021 14:47:20 +0100 (BST)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=smtpcorp.com; s=a1-4; h=Feedback-ID:X-Smtpcorp-Track:Message-Id:Subject:
Date:To:From:Reply-To:Sender:List-Unsubscribe;
bh=cTg4MkkE2uaIjpApjJYQFK3RgYiMF3bwCj8UZjFO4NE=; b=STU7lctit7L5LJ2tA3Re1fe4II
lXJbY/SBXTGqCHh9p4K86aLK5Bvz98Q7eR9xwjFib6x4NoZZ5L1fke0XQERd1eQvxkl9R+kRIGU8A
QOtrLPpt8coN8P+syoaTRR4pDJQG9OfJO1fON9OaOP8HwnEg/91ie6Cm+wQRxjwyat859uAcu89Xv
6/mrcequkSp6kfiQN4goZ7vMYJYfBYuooslbTciaK4SYIfxdINyrrWGA6QhJPobdW0uuedRNY5jBG
OdMbVmm7FTpxDJs51rB1PTIcFQ8W1oypcttqSgCjI+5eMVrabU/IoIxhX5F0Cn3zm7E9CHlaJuLt1
CRXVbwdw==;
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=fake.com; i=#fakelink.com; q=dns/txt; s=s575655;
t=1629812840; h=from : subject : to : message-id : date;
bh=cTg4MkkE2uaIjpApjJYQFK3RgYiMF3bwCj8UZjFO4NE=;
b=TEeEsPNLf7Wi6b8aaxE6JvfymfBKYjLq7izcUVrOXTW7sGIznxOA5udhfmDh15Fgp6Qgh
Kv5HX9uPNa8TEeoaJ+gV/4KERuscnc4GXEHwo0eclktx6f6JI5h1/q+qCe34+cN/EweaP5n
iOs+nrzsRuWn/iQ0Yck+b4IXVWHoTW8298xmBNuC1JF4jIVXREJFAC0nACfGU03OlpjDXf/
qvI6Ffnn5YGTNxgIkOdrtymaqOvjG9NM0PWtgSkvsTCJdUvxkrI+rRUG6ixiNi+vifqwvox
aQ6BRnMmeNK7A954Dy9r9r09QzbTthsBsi+lORKH7DntBKhm7Rb5/Q9j0xVA==
Received: from [10.176.58.103] (helo=SmtpCorp) by smtpcorp.com with esmtpsa
(TLS1.2:ECDHE_SECP256R1__RSA_SHA256__AES_256_GCM:256)
(Exim 4.94.2-S2G) (envelope-from <tomtest#fakelink.com>)
id 1mIWls-TRjyEC-AK for fake#fake.com; Tue, 24 Aug 2021 13:47:20 +0000
Received: from [10.86.20.232] (helo=DESKTOP-69OG2R3)
by smtpcorp.com with esmtpsa (TLS1.2:ECDHE_RSA_SECP256R1__AES_256_GCM:256)
(Exim 4.94.2-S2G) (envelope-from <fakename#fakelink.com>)
id 1mIWlr-9EFPsz-U0 for fake#fake.com; Tue, 24 Aug 2021 13:47:19 +0000
MIME-Version: 1.0
From: fake#fake2.com
To: fake#fake.com
Date: 24 Aug 2021 14:46:30 +0100
Subject: Test Email 2xM9e5Dj
Content-Type: multipart/alternative;
boundary=--boundary_11_ddba370a-13e2-4ffc-8b36-0eb7a5cde80e
Message-Id: <E1mIWlr-9EFPsz-U0#message-id.smtpcorp.com>
X-Smtpcorp-Track: 1XmW_r9EFeszl0.JChXLDDjoy7xH
Feedback-ID: 575655m:575655aVI_MaS:575655sNpPp5WOdD
X-Report-Abuse: Please forward a copy of this message, including all headers,
to <abuse-report#smtp2go.com>
----boundary_11_ddba370a-13e2-4ffc-8b36-0eb7a5cde80e
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
This is a text message
----boundary_11_ddba370a-13e2-4ffc-8b36-0eb7a5cde80e
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
This is a html message
----boundary_11_ddba370a-13e2-4ffc-8b36-0eb7a5cde80e--
This is stored in a variable called $emailText
I'm trying to use a regex to take the From part out of the text
From: fake#fake2.com
My regex isnt super strong, however my testing looks like this works: (?<=From: ).*.
But when I try and take the text out, it appears I can't get the regex to go through properly.
echo [[ $emailText =~ (?<=From: ).*. ]]
bash regex doesn't support lookbehind or lookahead assertions.
It is much easier to use a non-regex approach using awk here:
awk -F ': ' '$1 == "From" {print $2}' <<< "$emailText"
fake#fake2.com
With bash:
[[ "$emailText" =~ From:\ ([^$'\n']*) ]] && echo "${BASH_REMATCH[1]}"
Output:
fake#fake2.com
With your shown samples, attempts; please try following awk code. Simple explanation would be, checking condition if 1st field is From: then print 2nd field of that line.
awk '$1=="From:"{print $2}' Input_file
2nd solution: In case you have only 1 entry of From: in whole file then try following, where we can use exit function to exit from Input_file after printing the matched line, to stop un-necessary reading of whole Input_file.
awk '$1=="From:"{print $2;exit}' Input_file
Assuming you only want the email terminus, here's a quick and dirty Awk script.
awk '/^$/ { exit 1 }
/^From: .* <[^<>#]+#[^<>]+>/ {
split($0, g, /[<>]/); print g[1]; exit }
/^From: / { print $2; exit }' file.eml
This should work correctly for all these cases:
From: Real Name <real.name#example.com>
From: "Name, Real" <outlook#torture.example.com>
From: terminus#example.com
From: terminus#example.com (Real Name)
From: =?q?utf-8?Real_N=A3=E4me?= <real.name#example.com>
As especially the last example should convince you, you will need significantly more work if you also need the full name of the correspondent in normalized form.
If there should be a mail address present, you can match it first using awk (without the unsupported need for lookarounds)
awk 'match($0, /^From: [^[:space:]#]+#[^[:space:]#]+$/) {
print $2
}' <<< "$emailText"
Output
fake#fake2.com
I want to place my cache.manifest in my web site but it should have "text/cache-manifest" as mime type. I'd like not to pack the static content in a war if possible. It's my first time deploying a web app so I don't know if I can put a web.xml along with the files and serve them like this:
sudo java -cp jetty-runner-9.2.13.v20150730.jar
org.eclipse.jetty.runner.Runner --port 80 --path /
site/MyStaticWebSite
without packing it into a war.
In Apache I would write this into .htaccess :
AddType text/cache-manifest .manifest
But how i can set it into jetty-runner?
I should place a WEB-INF directory inside the root of the served content with a web.xml file that contains:
<web-app>
<mime-mapping>
<extension>manifest</extension>
<mime-type>text/cache-manifest</mime-type>
</mime-mapping>
</web-app>
Then restart jetty-runner, and check :
wget --server-response --spider http://XX.XX.XXX.XXX/file.manifest
I get :
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Date: Thu, 19 Nov 2015 16:16:39 GMT
Content-Type: text/cache-manifest
Last-Modified: Thu, 19 Nov 2015 15:58:57 GMT
Accept-Ranges: bytes
Content-Length: 143
Server: Jetty(9.2.13.v20150730)
Length: 143 [text/cache-manifest]
Remote file exists.
I would like to use procmail to forward a message to another email address. Both the headers and body of the incoming message should be in the body of the outgoing message (inline forwarding).
Example incoming message:
From: outside#example.com
To: me#example.com
Subject: Test
Date: Mon, 03 Nov 2014 05:00:04 GMT
This is a test
The forwarded message should be like this:
From: me#example.com
To: thirdparty#example.com
Subject: Fwd: Test
Date: Mon, 03 Nov 2014 05:01:00 GMT
From: outside#example.com
To: me#example.com
Subject: Test
Date: Mon, 03 Nov 2014 05:00:04 GMT
This is a test
Can this be done using procmail, maybe in conjunction with something like formail?
Easy enough.
:0
* Some conditions, perhaps? Omit this line to forward unconditionally
* ^Subject:[ ]*\/.*
| (echo From: me#example.com; echo To: thirdparty#example.com; \
echo "Subject: Fwd: $MATCH"; echo; cat -) | $SENDMAIL -t
If you don't care about forwarding the original Subject header verbatim, this can be simplified additionally.
The -t flag to sendmail says to use whatever To: and Cc: headers are in the message to determine the recipient. I omitted generating a Date: because (most imitations of) Sendmail will do that for you.
The stuff in the square brackets should be one space and one tab, as usual.
If you want to keep a copy, either add Bcc: yourself (and take care to not have the incoming copy trigger a mail loop!) or change :0 to :0c which makes Procmail continue down the rest if the recipe file.
I try to fiddle a RegEx, which returns me only the Sender IP Address:
http://regexr.com?38atl
This is the RegEx I build, but cant complete:
(?<=\bReceived: from .*\[)(?:\d{1,3}\.){3}\d{1,3}
or
(?<=\bReceived: from )(.*\[)(?:\d{1,3}\.){3}\d{1,3}
So it should only match this (on lines beginning with: Received: from)
127.0.0.1
127.0.0.1
21.22.23.24
And this are a example Mail-Headers i'm search in:
To: a#domain.de
Return-Path: <t#domain.de>
X-Original-To: a#domain.de
Delivered-To: c#domain.tld
Received: from localhost (localhost [127.0.0.1])
by mail1.domain.tld (Postfix) with ESMTP id 3fT3TR72zNz8m8
for <a#domain.de>; Tue, 18 Feb 2014 14:54:35 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at mail1.domain.tld
X-Spam-Flag: YES
X-Spam-Score: 5.773
X-Spam-Level: *****
X-Spam-Status: Yes, score=5.773 tagged_above=1 required=4.5
tests=[BAYES_05=-0.5, MISSING_MID=0.497, RCVD_IN_PBL=3.335,
RCVD_IN_RP_RNBL=1.31, RDNS_DYNAMIC=0.982, TO_NO_BRKTS_DYNIP=0.139,
T_RCVD_IN_SEMBLACK=0.01] autolearn=no
Received: from mail1.domain.tld ([127.0.0.1])
by localhost (mail1.domain.tld [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id lDJqiZjBn2t4 for <a#domain.de>;
Tue, 18 Feb 2014 14:54:34 +0100 (CET)
Received: from mail.domain.tld (pAAAAAAAA.dip0.t-ipconnect.de [21.22.23.24])
by mail1.domain.tld (Postfix) with SMTP id 3fT3TQ4Nwgz8m5
for <a#domain.de>; Tue, 18 Feb 2014 14:54:34 +0100 (CET)
Date: Tue, 18 Feb 2014 15:02:11 +0100
Sender: "From" <t#domain.de>
From: "From" <t#domain.de>
Subject: Subbbb (192.168.123.123)
Reply-To: t#domain.de
MIME-Version: 1.0
Content-type: text/plain; charset=UTF-8
Message-Id: <3fT3TR72zNz8m8#mail1.domain.tld>
Try this expression:
Received: +from[^\n]*?\[([0-9\.]+)\]
Edit:
For a PHP script try something like this (where $emailHeader contains the data you are searching):
$regex = '/Received: +from[^\\n]*?\\[([0-9\\.]+)\\]/s';
if (preg_match_all($regex, $emailHeader, $matches_out)) {
print_r($matches_out);
} else {
print('Sender IP not found');
}
The <= in the star looks funny, but other than that it seems to be working fine:
(?:\bReceived: from .*\[)((\d{1,3}\.){3}\d{1,3})(?:]\))
I believe what you're looking for is:
(?:\bReceived: from .*?\[)(?<ip>(?:\d{1,3}\.){3}\d{1,3})
the matched IP address will be in capture group named "ip".