I'm trying to forward e-mails matching a certain pattern to a set of addresses, and BCC the same e-mail to some other e-mail addresses. From looking at my procmail log, it appears that all my rules are matching, yet the BCC recipient does not receive the message.
The relevant lines of my .procmailrc look like this:
:0fhw
* ^From.*#example.com
* ! ^X-Loop: test
| formail -A "Bcc: $BCCS"
:0fhw
* ^From.*#example.com
* ! ^X-Loop: test
| formail -A "X-Loop: test"
:0
* ^From.*#example.com
* ! $DEST ?? ^$
! $DEST
At the point this part of the procmailrc is reached, the BCCS variable contains the address(es) to BCC, and the DEST variable contains the address(es) to forward to.
In the log I see something like this:
procmail: Match on "^From.*#example.com"
procmail: Match on ! "^X-Loop: test"
procmail: Executing "formail,-A,Bcc: bcctest#somewhere.com"
procmail: Match on "^From.*#example.com"
procmail: Match on ! "^X-Loop: test"
procmail: Executing "formail,-A,X-Loop: test"
procmail: Match on "^From.*#example.com"
procmail: Match on ! "^$"
procmail: Executing "/usr/sbin/sendmail,-oi,user1#somewhere.com,user2#somewhere.com"
procmail: Assigning "LASTFOLDER=/usr/sbin/sendmail -oi user1#somewhere.com,user2#somewhere.com"
procmail: Notified comsat: "michael#:/usr/sbin/sendmail -oi user1#somewhere.com,user2#somewhere.com"
It appears that a Bcc: header is being added and the e-mail forwarded as I expect. My assumption from what I have gathered from my research is that to BCC in the forward I need to add a "Bcc:" header, and sendmail will copy the message to any addresses it specifies and strip the Bcc: header off in the actually sent e-mail. However I am not 100% sure, as all the questions I have found regarding BCC deal with people wanting to trigger on BCC on the incoming message, which can't really be done if the sending server is configured properly.
Am I doing something wrong?
This all resolves to a very common FAQ: the headers don't ultimately decide where a message actually gets delivered. Your action
! $DEST
tells Sendmail to send the message to $DEST and nowhere else.
(You can tell Sendmail to actually examine the recipient headers with sendmail -t.)
With that understanding, you can actually remove the recipe to add an explicit Bcc: header, and simply change the last line to
! $DEST $BCCS
(Calling formail twice was superfluous anyway. It's sometimes useful and necessary, but you can have two -A options in the same formail invocation. But adding the Bcc is not useful; Sendmail will strip it off again.)
With Sendmail -t (which inside Procmail can be used in an action ! -t) the headers are examined for To:, Cc:, Bcc: etc, and the parsed out addresses are copied to the outgoing message's envelope recipients. Without -t, the command-line arguments are copied as envelope recipients, and the headers are not consulted. Once the message is inside the envelpe, only the envelope information decides where it goes.
formail -A "Bcc: $BCCS" adds a "Bcc:" header. ! $DEST forwards the message to "$DEST", however, ! ... will ignore the "Bcc:" header. ! ... effectively works like Bcc (except there is no header added and removed). Instead of your 3rd forwarding rule you can use 2 rules, where the 1st works on a copy of the message (note the c flag in the 1st rule):
:0 c
* ^From.*#example.com
* ! $BCC ?? ^$
! $BCC
:0
* ^From.*#example.com
* ! $DEST ?? ^$
! $DEST
An alternative is sendmail -t. It reads recipients from the mail headers, so it would also see your "Bcc:" header and process the message accordingly. I'd advise against using sendmail -t though in general unless you have a very controlled environment where you are sure there are no "To:", "CC:" and "BCC:" headers in the messages that you don't want.
Procmail by default uses sendmail, not sendmail -t. So, you'd have to pipe the message to sendmail -t like in
:0
* ^From.*#example.com
* ! $DEST ?? ^$
| sendmail -i -t $DEST
Related
I have the following in a bash script:-
#!/bin/bash
re="/(\W|^)php[5-9]{1}.[0-9]{1}-fpm.sock(\W|$)/gm"
while ! [[ "${socket}" =~ ${re} ]]
do
echo "enter socket string:"
read socket
done
A valid $socket string from the user would equal php8.1-fpm.sock using the regex we're testing for. What actually happens is, the loop continues with a user unable to break out of it despite a valid string?
I should be able to use my $socket variable in the script following a successful validation. What am I missing?
Edit:
Stuff I've tried:-
re="/php[5-9]{1}.[0-9]{1}-fpm.sock/" omitting (\W)
This could be done more correctly for the match, using standard globbing pattern matching of case in, with POSIX-shell grammar only.
#!/bin/sh
while case $socket in *php[5-9].[0-9]-fpm.sock) false ;; esac; do
printf 'Enter socket string: '
read -r socket
done
It could even test the socket is really an actual socket by testing -S:
#!/bin/sh
while case $socket in *php[5-9].[0-9]-fpm.sock) ! [ -S "$socket" ] ;; esac; do
printf 'Enter socket string: '
read -r socket
done
See man test:
-S FILE
FILE exists and is a socket
I have some text with a password which may contain special characters (like /, *, ., [], () and other that may be used in regular expressions). How to remove that password from the text using Korn shell or maybe sed or awk? I need an answer which would be compatible with Linux and IBM AIX.
For example, the password is "123*(/abc" (it is contained in a variable).
The text (it is also contained in a variable) may look like below:
"connect user/123*(/abc#connection_string"
As a result I want to obtain following text:
"connect user/#connection_string"
I tried to use tr -d but received wrong result:
l_pwd='1234/\#1234'
l_txt='connect target user/1234/\#1234#connection'
print $l_txt | tr -d $l_pwd
connect target user\connection
tr -d removes all characters in l_pwd from l_txt that's why the result is so strange.
Try this:
l_pwd="1234/\#1234";
escaped_pwd=$(printf '%s\n' "$l_pwd" | sed -e 's/[]\/$*.^[]/\\&/g')
l_txt="connect target user/1234/\#1234#connection";
echo $l_txt | sed "s/$escaped_pwd//g";
It prints connect target user/#connection in bash at least.
Big caveat is this does not work on newlines, and maybe more, check out where I got this solution from.
With ksh 93u+
Here's some parameter expansion madness:
$ echo "${l_txt//${l_pwd//[\/\\]/?}/}"
connect target user/#connection
That takes the password variable, substitutes forward and back slashes into ?, then uses that as a pattern to remove from the connection string.
A more robust version:
x=${l_pwd//[^[:alnum:]]/\\\0}
typeset -p x # => x='1234\/\\\#1234'
conn=${l_txt%%${x}*}${l_txt#*${x}}
typeset -p conn # => connect target user/#connection
I am trying to reply to messages larger than a certain size then forward to another user. Got this, but nothing happens. Its seem I am only able to add text to the end of the message.
:0
* > 1000
{
:0 fhw
| cat - ; echo "Insert this text at the top of the body"
:0
| formail -rk
| $SENDMAIL -t
}
Using sed helped a lot.
SEDSCRIPT='0,/^$/ s//\nLarge message rejected [Max=4MB]\n/'
MAILADDR=me#nowhere
:0
* > 4000000
* !^FROM_DAEMON
* !^X-Loop: $MAILADDR
| formail -rk -A "X-Loop: $MAILADDR" \
| sed "$SEDSCRIPT" \
| $SENDMAIL -t
It's not clear what exactly is wrong, but if you want to append text at the beginning, you obviously need to echo before cat, and work on the body (b), not the headers (h).
:0 fbw
| echo "Insert this"; cat -
I suppose you could technically break the headers by appending something at the end, but if you want it to appear in the body, it needs to have a neck (a newline) before it.
:0 fhw
| cat -; echo; echo "Insert this"
There is also a sed syntax which allows for somewhat more flexible manipulation (sed addressing lets you say things like "before the first line which starts with > for example) but getting newlines into sed command lines inside Procmail is hairy. As a workaround, I often use a string, and then just interpolate that. (How hairy exactly depends on details of sed syntax which are not standard. Some implementations seem to require newlines in the a and i commands.)
sedscript='1i\
insert this\
'
:0 fbw
| sed "$sedscript"
(If you are lucky, your sed will accept something simpler like sed '1i insert this'. The variant above seems to be the only one I can get to work on macOS, and thus probably generally *BSD.)
As an aside, a message which is 1000 bytes long isn't by any means large. I recall calculating an average message length of about 4k in my own inbox, but this was before people started to use HTML email clients. Depending on your inbound topology, just the headers could easily be more than 1000 bytes.
I am trying to grep by regex but I am having trouble figuring out what is wrong with my regex. I couldnt find any bash regex testers out there so this has been really hard to figure out.
Here is my regex
[0-9]*\.[0-9]*[G][:space:]*\.\/[bbg-sevent-test-][0-9]*
I am trying to match my regex to this piece of text
2.0G ./bbg-sevent-test-132^M
The command I am running is:
./kafka_prefill.sh | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" | grep '[0-9]*\.[0-9]*[G][:space:]*\.\/[bbg-sevent-test-][0-9]*' > data3.txt
What this does is run my script, translate/remove parts of my output, then grep based on regex and put it in the file data3.txt
I am currently getting this error:
grep: Invalid range end
** update ** thanks to Ed Plunkett
updated regex:
^[0-9]*\.[0-9]*[G][[:space:]]*\.\/bbg-sevent-test-[0-9]*$
My command no longer has a regex error. However nothing is matching. Here is a sample output:
********************************************************************************^M
This is a private computer system containing information that is proprietary^M
and confidential to the owner of the system. Only individuals or entities^M
authorized by the owner of the system are allowed to access or use the system.^M
Any unauthorized access or use of the system or information is strictly^M
prohibited.^M
^M
All violators will be prosecuted to the fullest extent permitted by law.^M
********************************************************************************^M
Last login: Tue Dec 29 16:43:23 2015 from 10.81.64.204^M^M
sudo bash^M
cd /data/kafka/tmp/kafka-logs/^M
du -kh . | egrep "bbg-sevent-test-*"^M
-bash: ulimit: open files: cannot modify limit: Operation not permitted^M
### Trinity env = prod ###^M
### Kafka Broker Id = 1 ###^M
### Kafka Broker must be started as root!! ###^M
exit^M
exit^M
### Trinity env = prod ###^M
### Kafka Broker Id = 1 ###^M
### Kafka Broker must be started as root!! ###^M
^[]0;root#ip-10-81-66-20:/home/ec2-user^G^[[?1034h[root#ip-10-81-66-20 ec2-user]# cd /data/kafka/tmp/kafka-logs/^M
^[]0;root#ip-10-81-66-20:/data/kafka/tmp/kafka-logs^G[root#ip-10-81-66-20 kafka-logs]# du -kh . | egrep "bbg-sevent-test-*"^M
2.2G ./bbg-sevent-test-439^M
2.2G ./bbg-sevent-test-638^M
2.2G ./bbg-sevent-test-679^M
2.2G ./bbg-sevent-test-159^M
I am only trying to match this bit
2.2G ./bbg-sevent-test-159
Why is this in square brackets?
[bbg-sevent-test-]
If you are matching that entire literal string, including brackets, escape them:
\[bbg-sevent-test-\]
If you're not matching the brackets as literal characters, leave them out:
bbg-sevent-test-
Looks to me like you don't really want them there. In a regex, text you match literally is just slapped in there as-is, no special syntax required except for escaping special characters like []*+?(). etc.
What you've got there is, syntactically, a range -- but a broken one, since there's nothing after the last hyphen. However, a range is clearly not your intent.
Summary / 'gist of' version,
if I have a set of messages with subject [SUB] and body [BODY] like below, How can I add a newline after the subject only if [BODY] exists (And replace the place holders with *)
[SUB] some subject. [BODY] some body lines
with newline chars and !### bunch of other *## chars
without [(BODY)] or [(SUB)]... and more stuff
[SUB] Another subject. with no body [BODY]
[SUB] another [BODY] some body.
I want this to be formatted like
* some subject.
some body lines
with newline chars and !### bunch of other *## chars
without [(BODY)] or [(SUB)]... and more stuff
* Another subject. with no body
* another
some body.
What I really wanna do,
So I am trying to auto-generate my CHANGELOG.md file from the git log output. The problem is, I need to put newline char only if the body of the commit message is non empty.
The current code looks like this, (broken into two lines)
git log v0.1.0..v0.1.2 --no-merges --pretty=format:'* %s -- %cn | \
[%h](http://github.com/../../commit/%H) %n%b' | grep -v Minor | grep . >> CHANGELOG.md
and a sample output,
* Added run information display (0.1.2) -- ... | [f9b1f6c](http://github.com/../../commit/...)
+ Added runs page to show a list of all the runs and run inforation, include sorting and global filtering.
+ Updated run information display panel on the run-info page
+ Changed the links and their names around.
* Update README.md -- abc | [2a90998](http://github.com/../../commit/...)
* Update README.md -- xt | [00369bd](http://github.com/../../commit/...)
You see here, the lines starting with the * are the commits, and the lines starting on + are just a part of the body for the first commit. Right now it adds a %n (newline) in front of all the body sections regardless of whether it's empty or not. I want to add this ONLY if its non empty (probably even after removing the whitespaces)
How would I achieve this? my knowledge of sed and awk is almost non-existing, and trying to learn didn't help much.
(I will can make sure all the code in the body is indented, so it wont confuse list of commits with lists in the body)
My Answer
i'm sure jthills answer is correct (and maye even a better way to do it), but while I was looking to figure out what his meant, i came up wit this. Hope it will help myself or someone in he future,
I am pasting the full shell script that I used,
mv CHANGELOG.md CHANGELOG.md.temp
printf '### Version '$1' \n\n' > CHANGELOG.md
git log $2..$1 --no-merges --pretty=format:'[SUB]%s -- %cn | \
[%h](http://github.com/<user>/<gitrepo>/commit/%H) [BODY]%b' | grep -v Minor | \
sed '{:q;N;s/\s*\[BODY\][\n\s]*\[SUB\]/\n\[SUB\]/;b q}' | \
sed 's/\[SUB\]/* /g' |
sed 's/\[BODY\]/\n\n/'>> CHANGELOG.md
cat CHANGELOG.md.temp >> CHANGELOG.md
rm CHANGELOG.md.temp
I am basically prepending the new commit log to the CHANGELOG.md using the temp file. Please feel free to suggest shorter versions for this 3 sed commands
Tag your syntax in the git log output. This will handle inserting the newlines properly, the rest you know:
git log --pretty=tformat:'%s%xFF%x01%b%xFF%x02' \
| sed '1h;1!H;$!d;g # buffer it all (see comments for details)
s/\xFF\x01\xff\x02//g # strip null bodies
s/\xFF\x01/\n/g # insert extra newline before the rest
s/\xFF.//g # cleanup
'
(edit: quote/escape typos)
For your first file in your question, you could try the following:
awk -f r.awk input.txt
where input.txt is the input file, and r.awk is :
{
line=line $0 ORS
}
END {
while (getSub()) {
getBody()
print "* " subj
if (body) {
print ""
print body
}
}
}
function getBody(ind) {
ind=index(line,"[SUB]")
if (ind) {
body=substr(line,1,ind-1)
line=substr(line,ind)
}
else
body=line
sub(/^[[:space:]]*/,"",body)
sub(/[[:space:]]*$/,"",body)
}
function getSub(ind,ind2) {
ind=index(line,"[SUB]")
if (ind) {
ind=ind+5
ind2=index(line,"[BODY]")
subj=substr(line, ind, ind2-ind)
line=substr(line,ind2+6)
return 1
}
else
return 0
}
gives output:
* some subject.
some body lines
with newline chars and !### bunch of other *## chars
without [(BODY)] or [(SUB)]... and more stuff
* Another subject. with no body
* another
some body.
I wrestled with this way longer than expected, simply trying to get a git log output with some sed tweaking of the git message to format/extract our JIRA messages.
Here is my solution:
logsheet = "!f() { git log --format='%h ^ %<(80,trunc)%s ^ A:%<(20,trunc)%an ^ D:%ad ' --no-merges --date=short $1 | sed -e 's/\\([A-Z]*-[0-9]*\\)/\\1 ^/'; }; f"
The escapes, the shell function with a ! were all needed because I had an arg as well as a pipe.
:-)