bash return string between 2 patterns - regex

I have this file: 1.txt
===================================================================
File: .cvsignore Status: Up-to-date
Working revision: 1.1 Thu Jan 16 11:00:00 2020
Repository revision: 1.1 /usr/local/rancid/var/CVS/Routers/.cvsignore,v
Sticky Tag: (none)
Sticky Date: (none)
Sticky Options: (none)
===================================================================
File: router.db Status: Up-to-date
Working revision: 1.6 Fri Jan 17 11:57:39 2020
Repository revision: 1.6 /usr/local/rancid/var/CVS/Routers/router.db,v
Sticky Tag: (none)
Sticky Date: (none)
Sticky Options: (none)
===================================================================
File: .cvsignore Status: Up-to-date
Working revision: 1.1 Thu Jan 16 11:00:00 2020
Repository revision: 1.1 /usr/local/rancid/var/CVS/Routers/configs/.cvsignore,v
Sticky Tag: (none)
Sticky Date: (none)
Sticky Options: (none)
===================================================================
===================================================================
File: 1.1.1.1 Status: Up-to-date
Working revision: 1.2 Fri Jan 17 07:56:21 2020
Repository revision: 1.2 /usr/local/rancid/var/CVS/Routers/configs/1.1.1.1,v
Sticky Tag: (none)
Sticky Date: (none)
Sticky Options: -ko
I want to extract string between File: and Status: (.cvsignore, router.db, 1.1.1.1)
and between Repository revision: and /usr/local/rancid/ (1.1, 1.6, 1.1 )
so final output should be:
.cvsignore 1.1
router.db 1.6
.cvsignore 1.1
1.1.1.1 1.2
for filename i created following filter:
sed 's/^.*File: //; s/Status:.*$//' 1.txt
Output:
===================================================================
.cvsignore
Working revision: 1.1 Thu Jan 16 11:00:00 2020
Repository revision: 1.1 /usr/local/rancid/var/CVS/Routers/.cvsignore,v
Sticky Tag: (none)
Sticky Date: (none)
Sticky Options: (none)
===================================================================
router.db
Working revision: 1.6 Fri Jan 17 11:57:39 2020
Repository revision: 1.6 /usr/local/rancid/var/CVS/Routers/router.db,v
Sticky Tag: (none)
Sticky Date: (none)
Sticky Options: (none)
===================================================================
.cvsignore
Working revision: 1.1 Thu Jan 16 11:00:00 2020
Repository revision: 1.1 /usr/local/rancid/var/CVS/Routers/configs/.cvsignore,v
Sticky Tag: (none)
Sticky Date: (none)
Sticky Options: (none)
===================================================================
1.1.1.1
Working revision: 1.2 Fri Jan 17 07:56:21 2020
Repository revision: 1.2 /usr/local/rancid/var/CVS/Routers/configs/172.18.1.24,v
Sticky Tag: (none)
Sticky Date: (none)
Sticky Options: -ko
===================================================================
For Repository revision i'm trying
sed 's/^.'*Repository revision:' //; s#usr#local#rancid:.*$//' 1.txt
But getting
sed: -e expression #1, char 15: unterminated `s' command
I need one regex for filename and second for Repository revision

Your example isn't clear but is this what you're trying to do?
$ awk 'sub(/^[[:space:]]*(File|Repository revision):[[:space:]]*/,""){print $1}' file
.cvsignore
1.1
router.db
1.6
.cvsignore
1.1
1.1.1.1
1.2
or maybe this:
$ awk 'sub(/^[[:space:]]*(File|Repository revision):[[:space:]]*/,""){printf "%s%s", $1, ((++c)%2 ? OFS : ORS)}' file
.cvsignore 1.1
router.db 1.6
.cvsignore 1.1
1.1.1.1 1.2

$ awk '/^File:/ { printf "%s # filename ", $2}
/Repository revision:/{printf "%s # Repository version\n", $3}' input
.cvsignore # filename 1.1 # Repository version
router.db # filename 1.6 # Repository version
.cvsignore # filename 1.1 # Repository version
1.1.1.1 # filename 1.2 # Repository version

This might work for you (GNU sed):
sed -En '/^File:\s*(\S+).*/{s//\1/;h};/^\s*Repository revision:\s*(\S+).*/{s//\1/;H;g;s/\n/ /p}' file
Store the file name in the hold space and append the revision, then replace the introduced newline by a space and print the result.

Related

Salt states. If variables have some word in stdout

There is a web page with a large piece of text on it.
I want to configure the state to perform a certain action if curl returns an error.
If the variable doesn't contain 'StatusDescription : OK'
How can I set up a check for a piece of text that is inside a variable
{% set seqstat = salt['cmd.run']('powershell.exe curl http://127.0.0.1:5001 -UseBasicParsing') %}
{% if seqstat is sameas '*StatusDescription : OK*' %}
module_run:
cmd.run:
- name: 'powershell.exe WRITE-HOST have no Error'
{% else %}
module_run1:
cmd.run:
- name: 'powershell.exe WRITE-HOST have Error'
{%- endif -%}
Salt Version:
Salt: 3002.1
Dependency Versions:
cffi: 1.12.2
cherrypy: unknown
dateutil: 2.7.3
docker-py: 3.4.1
gitdb: 2.0.5
gitpython: 2.1.11
Jinja2: 2.10
libgit2: 0.27.7
M2Crypto: Not Installed
Mako: 1.0.7
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: 1.3.10
pycparser: 2.19
pycrypto: 2.6.1
pycryptodome: 3.6.1
pygit2: 0.27.4
Python: 3.7.3 (default, Jul 25 2020, 13:03:44)
python-gnupg: 0.4.4
PyYAML: 3.13
PyZMQ: 17.1.2
smmap: 2.0.5
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.1
System Versions:
dist: debian 10 buster
locale: UTF-8
machine: x86_64
release: 4.19.0-6-amd64
system: Linux
version: Debian GNU/Linux 10 buster
I want to configure the state to perform a certain action if curl returns an error.
There is a Salt state called http which can query a URL and return the status. Using this (instead of curl) we can check for the status code(s) (200, 201, etc.), as well as matching text. Then we can use requisites to run subsequent states depending on the success/failure of the http.query.
Example:
I have added a check for status code of 200, you can omit - status: 200 if you don't care about the status code.
check-application:
http.query:
- name: http://127.0.0.1:5001
- status: 200
- match: 'StatusDescription : OK'
app-running:
cmd.run:
- name: 'powershell.exe WRITE-HOST have no Error'
- require:
- http: check-application
app-not-running:
cmd.run:
- name: 'powershell.exe WRITE-HOST have Error'
- onfail:
- http: check-application

Cannot run puma with systemd on debian while the same start command in terminal works (fe_sendauth: no password supplied error)

I try to manage Puma server on my ruby on rails web-site with systemd. Puma cannot start with following error: PG::ConnectionBad: fe_sendauth: no password supplied. When I start Puma myself in terminal with the same start command as in systemd it run correctly. Please help.
I use RoR 4.2.11.1, postgresql 11.2, on Debian 9.12, which run on VirtualBox 6.0.
web-site file structure:
/mytarifs/current - symlink to last release
/mytarifs/releases - relseas
/mytarifs/shared - shared files like database connections
I start Puma in terminal with success by following command:
root#mt-staging-1:/mytarifs/current# bundle exec puma -C config/puma.production.rb
Database_URL environment variable:
DATABASE_URL=postgresql://login_name:password#localhost:5432/db_tarif
With this database url I can connect to my db with psql
error log:
Mar 07 02:20:39 mt-staging-1 systemd[1]: Started puma for mytarifs (production).
Mar 07 02:20:40 mt-staging-1 puma[12237]: [12237] Puma starting in cluster mode...
Mar 07 02:20:40 mt-staging-1 puma[12237]: [12237] * Version 4.3.3 (ruby 2.3.8-p459), codename: Mysterious Traveller
Mar 07 02:20:40 mt-staging-1 puma[12237]: [12237] * Min threads: 0, max threads: 5
Mar 07 02:20:40 mt-staging-1 puma[12237]: [12237] * Environment: production
Mar 07 02:20:40 mt-staging-1 puma[12237]: [12237] * Process workers: 1
Mar 07 02:20:40 mt-staging-1 puma[12237]: [12237] * Preloading application
Mar 07 02:20:47 mt-staging-1 puma[12237]: The PGconn, PGresult, and PGError constants are deprecated, and will be
Mar 07 02:20:47 mt-staging-1 puma[12237]: removed as of version 1.0.
Mar 07 02:20:47 mt-staging-1 puma[12237]: You should use PG::Connection, PG::Result, and PG::Error instead, respectively.
Mar 07 02:20:47 mt-staging-1 puma[12237]: Called from /mytarifs/releases/20200306184828/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.11.1/lib/active_support/dependencies.rb:240:in `load_dependency'
/mytarifs/current/config/puma.production.rb
threads Integer(ENV['MIN_THREADS'] || 0), Integer(ENV['MAX_THREADS'] || 5)
workers Integer(ENV['PUMA_WORKERS'] || 1)
preload_app!
bind 'unix:///mytarifs/shared/tmp/sockets/puma.sock'
pidfile '/mytarifs/shared/tmp/pids/puma.production.pid'
state_path '/mytarifs/shared/tmp/pids/puma.state'
rackup DefaultRackup
environment ENV['RACK_ENV'] || 'production'
on_worker_boot do
ActiveSupport.on_load(:active_record) do
ActiveRecord::Base.establish_connection
end
end
/mytarifs/current/config/database.yml
default: &default
adapter: postgresql
encoding: unicode
pool: 125
username: <%= ENV["PG_USERNAME"] %>
password: <%= ENV["PG_PASSWORD"] %>
host: localhost
template: template0
reconnect: true
production:
<<: *default
url: <%= ENV["DATABASE_URL"] %>
/etc/systemd/system/puma.service
[Unit]
Description=puma for mytarifs (production)
After=network.target
[Service]
Type=simple
Environment=RAILS_ENV=production
Environment=PUMA_DEBUG=1
WorkingDirectory=/mytarifs/current
ExecStart=/root/.rbenv/shims/bundle exec puma -e production -C config/puma.production.rb
ExecReload=/bin/kill -TSTP $MAINPID
ExecStop=/bin/kill -TERM $MAINPID
User=root
Group=root
RestartSec=1
Restart=on-failure
SyslogIdentifier=puma
[Install]
WantedBy=multi-user.target
ok, I found the reason for mistake. It is because environment variables are not available (equal to "") when systemd executes.
I do not know how get environment variables from memory, but systemd can take them from file with directive EnvironmentFile=/absolute/path/to/environment/file

Grep(exclude) lines that have regex matching next line

I have a log file that I am trying to grep -v all the unnecessary information so I can see only useful information. I cannot seem to figure out how to exclude the date if the next line after is also a date.
What I have so far:
Fri Apr 7 01:11:01 PDT 2017
Upgrading certbot-auto 0.12.0 to 0.13.0...
Replacing certbot-auto...
Installation succeeded.
Sat Apr 8 01:11:01 PDT 2017
Sun Apr 9 01:11:01 PDT 2017
Mon Apr 10 01:11:01 PDT 2017
Tue Apr 11 01:11:01 PDT 2017
Wed Apr 12 01:11:01 PDT 2017
Thu Apr 13 01:11:01 PDT 2017
Fri Apr 14 01:11:01 PDT 2017
Sat Apr 15 01:11:01 PDT 2017
Sun Apr 16 01:11:01 PDT 2017
Mon Apr 17 01:11:01 PDT 2017
Tue Apr 18 01:11:01 PDT 2017
Wed Apr 19 01:11:01 PDT 2017
Thu Apr 20 01:11:01 PDT 2017
Fri Apr 21 01:11:01 PDT 2017
WARNING: unable to check for updates.
Sat Apr 22 01:11:01 PDT 2017
Sun Apr 23 01:11:01 PDT 2017
Mon Apr 24 01:11:01 PDT 2017
Tue Apr 25 01:11:01 PDT 2017
What I want:
Fri Apr 7 01:11:01 PDT 2017
Upgrading certbot-auto 0.12.0 to 0.13.0...
Replacing certbot-auto...
Installation succeeded.
Fri Apr 21 01:11:01 PDT 2017
WARNING: unable to check for updates.
Well I came up with this one. See if this one helps.
Regex: ([A-Za-z]{3}\s[A-Za-z]{3}\s*\d{1,2}\s(?:\d{2}:){2}\d{2}\s[A-Z]{3}\s\d{4}\n?){2,}
Regex101 Demo
SOLVED!
Got it using:
grep -v '^.*\(PDT\|PST\)\s*[0-9]\{4\}' -B 1 | grep -v '^--$'
Here is the final command:
cat certbot.log |
grep -v '^-*$' |
grep -v '^Processing ' |
grep -v '(skipped)' |
grep -v 'No renewals were' |
grep -v 'not due for renewal yet' |
grep -v 'No hooks' |
grep -v 'DeprecationWarning' |
grep -v 'not yet due for' |
grep -v '^Saving debug' |
grep -v 'Installing Python packages' |
grep -v 'Creating virtual' |
grep -v '^.*\(PDT\|PST\)\s[0-9]\{4\}' -B 1 |
grep -v '^--$' |
sed '/\(PDT\|PST\)/i\\n' |
sed 's/.*\(PDT\|PST\).*/--- & --- /'
Here is the final result:
--- Fri Mar 3 01:11:01 PST 2017 ---
Upgrading certbot-auto 0.11.1 to 0.12.0...
Replacing certbot-auto...
Installation succeeded.
--- Thu Mar 23 01:11:01 PDT 2017 ---
WARNING: unable to check for updates.
--- Wed Mar 29 01:11:01 PDT 2017 ---
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for {mydomain}.com
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0001_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0001_csr-certbot.pem
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/{mydomain}.com/fullchain.pem
Congratulations, all renewals succeeded. The following certs have been renewed:
/etc/letsencrypt/live/{mydomain}.com/fullchain.pem (success)
--- Thu Apr 6 01:11:01 PDT 2017 ---
WARNING: unable to check for updates.
--- Fri Apr 7 01:11:01 PDT 2017 ---
Upgrading certbot-auto 0.12.0 to 0.13.0...
Replacing certbot-auto...
Installation succeeded.
--- Fri Apr 21 01:11:01 PDT 2017 ---
WARNING: unable to check for updates.
If you perl installed, try running this on the shell:
perl -0777 -ne 'while(m/^(.*?succeeded\..*?\d{4}.)|WARNING:.*?\./simg){print "$&";}' your_file
Output:
Fri Apr 7 01:11:01 PDT 2017
Upgrading certbot-auto 0.12.0 to 0.13.0...
Replacing certbot-auto...
Installation succeeded.
WARNING: unable to check for updates.
Regex Demo

Rails: Timezone difference on local and production

I have a Sidekiq worker, fetching records from a remote Oracle database and saving it in a postgres database. Most of the data consists of date and time fields. The issue I'm getting on production is that it saves the data time in different timezone with 5 hours difference. Although it works fine on local. Here is an example:
From Oracle DB => 2016-01-27 07:59:29 +0500
Saved on local => 2016-01-27 07:59:29 -0500
On Production => 2016-01-27 02:59:29 -0500
In config/application.rb I have set:
config.time_zone = 'Eastern Time (US & Canada)'
What should I do to make it consistent on both local and production?
check this http://www.elabs.se/blog/36-working-with-time-zones-in-ruby-on-rails, seems like your are doing some from the dont's section:
Time.now # Returns system time and ignores your configured time zone. (2015-08-27 14:09:36 +0200)
Time.parse("2015-08-27T12:09:36Z") # Will assume time string given is in the system's time zone. (2015-08-27 12:09:36 UTC)
Time.strptime("2015-08-27T12:09:36Z", "%Y-%m-%dT%H:%M:%S%z") # Same problem as with Time.parse. (2015-08-27 12:09:36 UTC)
Date.today # This could be yesterday or tomorrow depending on the machine's time zone, see https://github.com/ramhoj/time-zone-article/issues/1 for more info. (Thu, 27 Aug 2015)
to aviod problem with TimeZone, just use:
Date.current # => Mon, 01 Feb 2016
DateTime.current # => Mon, 01 Feb 2016 17:32:18 +0000
Time.current # => Mon, 01 Feb 2016 17:33:17 UTC +00:00

RegEx match IP on Mail-Header Received:

I try to fiddle a RegEx, which returns me only the Sender IP Address:
http://regexr.com?38atl
This is the RegEx I build, but cant complete:
(?<=\bReceived: from .*\[)(?:\d{1,3}\.){3}\d{1,3}
or
(?<=\bReceived: from )(.*\[)(?:\d{1,3}\.){3}\d{1,3}
So it should only match this (on lines beginning with: Received: from)
127.0.0.1
127.0.0.1
21.22.23.24
And this are a example Mail-Headers i'm search in:
To: a#domain.de
Return-Path: <t#domain.de>
X-Original-To: a#domain.de
Delivered-To: c#domain.tld
Received: from localhost (localhost [127.0.0.1])
by mail1.domain.tld (Postfix) with ESMTP id 3fT3TR72zNz8m8
for <a#domain.de>; Tue, 18 Feb 2014 14:54:35 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at mail1.domain.tld
X-Spam-Flag: YES
X-Spam-Score: 5.773
X-Spam-Level: *****
X-Spam-Status: Yes, score=5.773 tagged_above=1 required=4.5
tests=[BAYES_05=-0.5, MISSING_MID=0.497, RCVD_IN_PBL=3.335,
RCVD_IN_RP_RNBL=1.31, RDNS_DYNAMIC=0.982, TO_NO_BRKTS_DYNIP=0.139,
T_RCVD_IN_SEMBLACK=0.01] autolearn=no
Received: from mail1.domain.tld ([127.0.0.1])
by localhost (mail1.domain.tld [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id lDJqiZjBn2t4 for <a#domain.de>;
Tue, 18 Feb 2014 14:54:34 +0100 (CET)
Received: from mail.domain.tld (pAAAAAAAA.dip0.t-ipconnect.de [21.22.23.24])
by mail1.domain.tld (Postfix) with SMTP id 3fT3TQ4Nwgz8m5
for <a#domain.de>; Tue, 18 Feb 2014 14:54:34 +0100 (CET)
Date: Tue, 18 Feb 2014 15:02:11 +0100
Sender: "From" <t#domain.de>
From: "From" <t#domain.de>
Subject: Subbbb (192.168.123.123)
Reply-To: t#domain.de
MIME-Version: 1.0
Content-type: text/plain; charset=UTF-8
Message-Id: <3fT3TR72zNz8m8#mail1.domain.tld>
Try this expression:
Received: +from[^\n]*?\[([0-9\.]+)\]
Edit:
For a PHP script try something like this (where $emailHeader contains the data you are searching):
$regex = '/Received: +from[^\\n]*?\\[([0-9\\.]+)\\]/s';
if (preg_match_all($regex, $emailHeader, $matches_out)) {
print_r($matches_out);
} else {
print('Sender IP not found');
}
The <= in the star looks funny, but other than that it seems to be working fine:
(?:\bReceived: from .*\[)((\d{1,3}\.){3}\d{1,3})(?:]\))
I believe what you're looking for is:
(?:\bReceived: from .*?\[)(?<ip>(?:\d{1,3}\.){3}\d{1,3})
the matched IP address will be in capture group named "ip".