How can I make multiple replacements on the same file using one saltstack state? - replace

Here's my target file:
Sonatype Nexus
# ==============
# This is the most basic configuration of Nexus.
# Jetty section
application-port=8081
application-host=0.0.0.0
nexus-webapp=${bundleBasedir}/nexus
nexus-webapp-context-path=/nexus
# Nexus section
nexus-work=/opt/nexuswork
runtime=${bundleBasedir}/nexus/WEB-INF
I know there's an easy way to do this with regex or a simple sed script:
sed -i 's/${bundleBasedir}\/..\/my\/second\/path\/002\/\/nexus/\/myfirstdir001\/g'
However, I would, ideally, prefer the saltstack way.
I would like it to look something like this:
Sonatype Nexus
# ==============
# This is the most basic configuration of Nexus.
# Jetty section
application-port=8081
application-host=0.0.0.0
nexus-webapp=/my/second/path/002/nexus # changed
nexus-webapp-context-path=/nexus
# Nexus section
nexus-work=/opt/nexuswork
runtime=/myfirstdir001/nexus/WEB-INF # changed
I haven't yet made sense of the saltstack documentation on this.
Saltstack's documentation for salt.states.file.replace seems fairly straightforward:
http://docs.saltstack.com/en/latest/ref/states/all/salt.states.file.html#salt.states.file.replace
Here's what I tried:
/opt/nexus-2.8.0/conf/nexus.properties
file: # state
- replace
- pattern: '\$\{bundleBasedir\}' # without escapes: '${bundleBasedir}/nexus'
- repl: '/my/second/path/002/nexus'
# - name: /opt/nexus-2.8.0/conf/nexus.properties
# - count=0
# - append_if_not_found=False
# - prepend_if_not_found=False
# - not_found_content=None
# - backup='.bak'
# - show_changes=True
- pattern: '\$\{bundleBasedir\}\/WEB-INF' # without escapes: ${bundleBasedir}/WEB-INF
- repl: '/myfirstdir001/'
I could maybe try multiple state IDs, but that seems inelegant.
If there's anything else I'm fuffing up, please advise!
I'd shore love to find a solution to this.
Also, if there's any demand for people improving the salt documentation, I think my team could be convinced to pitch in some.
Here's the closest thing I've found to someone else asking this question:
http://comments.gmane.org/gmane.comp.sysutils.salt.user/15138

For such a small file I would probably go with a template as ahus1 suggested.
If the file was bigger and/or we didn't want to control other lines just ensure that those two are correct, I think multiple state IDs (as mentioned by OP) is a good way to go. Something like:
/opt/nexus-2.8.0/conf/nexus.properties-jetty:
file:
- replace
- name: /opt/nexus-2.8.0/conf/nexus.properties
- pattern: '\$\{bundleBasedir\}' # without escapes: '${bundleBasedir}/nexus'
- repl: '/my/second/path/002/nexus'
/opt/nexus-2.8.0/conf/nexus.properties-nexus:
file:
- replace:
- name: /opt/nexus-2.8.0/conf/nexus.properties
- pattern: '\$\{bundleBasedir\}\/WEB-INF' # without escapes: ${bundleBasedir}/WEB-INF
- repl: '/myfirstdir001/'
I have a similar setup in my configuration but I use salt.states.file.line to replace some lines with my values. In addition I used salt.states.file.managed with a template and replace: False to initialize the file if it's missing but once it exists, only the line states are doing changes.

The salt way to do this as I understand it: Place a template file for nexus.properties inside salt and use file.managed like shown in the docs http://docs.saltstack.com/en/latest/ref/states/all/salt.states.file.html
You will end up with something like:
/opt/nexus-2.8.0/conf/nexus.properties:
file.managed:
- source: salt://nexus/nexus.properties.jinja
- template: jinja
- defaults:
bundleBasedir: "..."
You'll then use Jinja templating in your file:
# Jetty section
application-port=8081
application-host=0.0.0.0
nexus-webapp={{ bundleBasedir }}/nexus
nexus-webapp-context-path=/nexus
See here for Jinja templating: http://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.jinja.html
I hope it helps.

Related

How to disable JSON format and send only the log message to Sumologic with Fluentbit?

We are using Fluentbit as as Sidecar container in our ECS fargate Cluster which is running a dotnet application, initially we faced the issue of fluentbit sending the logs in multiline and we solved it using Fluentbit Multilne feature. Now the logs are being sent to Sumologic in Multiple however it is being sent as Json format whereas we just want fluentbit send only the raw log
Logs are currently
{
date:1675120653.269619,
container_id:"xvgbertytyuuyuyu",
container_name:"XXXXXXXXXX",
source:"stdout",
log:"2023-01-30 23:17:33.269Z DEBUG [.NET ThreadPool Worker] Connection.ManagedDbConnection - ComponentInstanceEntityAsync - Executing stored proc: dbo.prcGetComponentInstance"
}
We want only the line
2023-01-30 23:17:33.269Z DEBUG [.NET ThreadPool Worker] Connection.ManagedDbConnection - ComponentInstanceEntityAsync - Executing stored proc: dbo.prcGetComponentInstance
You need to modify Fluent Bit configuration to have the following filters and output configuration:
fluent.conf:
## prepare headers for Sumo Logic
[FILTER]
Name record_modifier
Match *
Record headers.content-type text/plain
## Set headers as headers attribute
[FILTER]
Name nest
Match *
Operation nest
Wildcard headers.*
Nest_under headers
Remove_prefix headers.
[OUTPUT]
Name http
...
# use log key as body
body_key $log
# use headers key as headers
headers_key $headers
That way, you are going to craft HTTP request manually. This is going to send request per log, which is not necessary a good idea. In order to mitigate that you can add the following parser and use it (flush_timeout may need an adjustment):
parsers.conf
# merge everything as one big log
[MULTILINE_PARSER]
name multiline-all
type regex
flush_timeout 500
#
# Regex rules for multiline parsing
# ---------------------------------
#
# configuration hints:
#
# - first state always has the name: start_state
# - every field in the rule must be inside double quotes
#
# rules | state name | regex pattern | next state
# ------|---------------|--------------------------------------------
rule "start_state" ".*" "cont"
rule "cont" ".*" "cont"
fluent.conf:
[INPUT]
name tail
...
multiline.parser multiline-all

How to set variable inline in gitlab-ci.yaml based on regex matching?

I am trying to create a variable in gitlab-ci.yaml based on the name of the branch.
Suppose I am pushing to a branch named 3.2.7
Here is the situation:
include:
- template: "Workflows/Branch-Pipelines.gitlab-ci.yml"
variables:
PRODUCTION_BRANCH: "master"
STAGING_BRANCH: (\d)\.(\d)\.(\d)
.deploy_rules:
rules:
- if: '$CI_COMMIT_BRANCH =~ /$STAGING_BRANCH/'
variables:
SERVER_PORT: 3007 # TODO: should be 300d ; d is the second digit
I want to generate 3002 inline using regex matching.
How can I do this?
I have done some research and seems I have to use sed but I am not sure if it is the best way to do it and how to do it.
TO MAKE THE PROBLEM SIMPLER
include:
- template: "Workflows/Branch-Pipelines.gitlab-ci.yml"
variables:
TEST_VAR: sed -E 's/(\d)\.(\d)\.(\d)/300\2/gm;t;d' <<< $CI_COMMIT_BRANCH
stages:
- temp
temp:
stage: temp
script:
- echo $TEST_VAR
Should be echoing 3002 but it is echoing sed -E 's/(\d)\.(\d)\.(\d)/300\2/gm;t;d' <<< 3.2.7
You can't use variables in the regex pattern. You just have to write the regex verbatim, it cannot be directly parameterized. You also cannot use sed or other Linux utilities in variables: or other parts of your yaml. You're bound to the limitations of YAML specification and features provided by GitLab.
However, there is an option available to you that will fit your stated use case.
Dynamic variables
TEST_VAR: sed -E 's/(\d).(\d).(\d)/300\2/gm;t;d' <<< $CI_COMMIT_BRANCH
While you can't use sed or other utilities directly in variables: declarations, you can use dotenv artifacts via artifacts:reports:dotenv to set variables dynamically.
For example, a job can use sed or whatever other utilities you like to create variables which will be used by the rest of the pipeline.
stages:
- temp
create_variables:
stage: .pre
script:
- TEST_VAR="$(sed -E 's/(\d)\.(\d)\.(\d)/300\2/gm;t;d' <<< ${CI_COMMIT_BRANCH})"
- echo "TEST_VAR=${TEST_VAR}" >> dotenv.txt
artifacts:
reports:
dotenv: dotenv.txt
temp:
stage: temp
script:
- echo $TEST_VAR
Here, the .pre stage is used, which is a special stage that is always ordered before every other stage. The dotenv artifact from the create_variables job will dynamically create variables for the jobs in subsequent stages that receive the artifact.

Create a cakephp filter for fail2ban

i would like to create a filter in fail2ban for searching and blocking bad request like "Controller class * could not be found."
For this problem i was create a cakephp.conf file in the filter.d directory in fail2ban. The Content:
[Definition]
failregex = ^[0-9]{4}\-[0-9]{2}\-[0-9]{2}.*Error:.*\nStack Trace:\n(\-.*|\n)*\n.*\n.*\nClient IP: <HOST>\n$
ignoreregex =
My example error log looks like this:
...
2020-10-08 19:59:46 Error: [Cake\Http\Exception\MissingControllerException] Controller class Webfig could not be found. in /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Controller/ControllerFactory.php on line 158
Stack Trace:
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Controller/ControllerFactory.php:46
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/BaseApplication.php:249
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:77
- /home/myapplication/htdocs/vendor/cakephp/authentication/src/Middleware/AuthenticationMiddleware.php:122
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:73
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:77
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Middleware/CsrfProtectionMiddleware.php:146
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:73
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:58
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Routing/Middleware/RoutingMiddleware.php:172
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:73
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Routing/Middleware/AssetMiddleware.php:68
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:73
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Error/Middleware/ErrorHandlerMiddleware.php:121
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:73
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Runner.php:58
- /home/myapplication/htdocs/vendor/cakephp/cakephp/src/Http/Server.php:90
- /home/myapplication/htdocs/webroot/index.php:40
Request URL: /webfig/
Referer URL: http://X.X.X.X/webfig/
Client IP: X.X.X.X
...
X.X.X.X are replaced
But i can't match any ip adresses. The fail2ban tester says:
root#test:~# fail2ban-regex /home/myapplication/htdocs/logs/error.log /etc/fail2ban/filter.d/cakephp.conf
Running tests
=============
Use failregex filter file : cakephp, basedir: /etc/fail2ban
Use log file : /home/myapplication/htdocs/logs/error.log
Use encoding : UTF-8
Results
=======
Failregex: 0 total
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [719] {^LN-BEG}ExYear(?P<_sep>[-/.])Month(?P=_sep)Day(?:T| ?)24hour:Minute:Second(?:[.,]Microseconds)?(?:\s*Zone offset)?
`-
Lines: 15447 lines, 0 ignored, 0 matched, 15447 missed
[processed in 10.02 sec]
Missed line(s): too many to print. Use --print-all-missed to print all 15447 lines
i can't see any problems. Can you help me? :)
Thanks
The issue is your log is poor suitable to parse - it is a multiline log-file (IP takes place in other line as the failure message).
Let alone the line with IP does not has any ID (common information with line of failure), it can be still worse if several messages are crossing (so Client IP from other message that is not a failure, coming after failure message).
If you can change the log-format better do that (so date, IP and failure sign are in the same line), e.g. if you use nginx, organize a conditional logging for access log from php-location in error case like this.
See Fail2ban :: wiki :: Best practice for more info.
If you cannot do that (well better would be to change it), you can use multi-line buffering and parsing using maxlines parameter and <SKIPLINES> regex.
Your filter would be something like that:
[Definition]
# we ignore stack trace, so don't need to hold buffer window too large,
# 5 would be enough, but to be sure (if some log-messages crossing):
maxlines = 10
ignoreregex = ^(?:Stack |- /)
failregex = ^\s+Error: \[[^\]]+\] Controller class \S+ could not be found\..*<SKIPLINES>^((?:Request|Referer) URL:.*<SKIPLINES>)*^Client IP: <HOST>
To test it directly use:
fail2ban-regex --maxlines=5 /path/to/log '^\s+Error: \[[^\]]+\] Controller class \S+ could not be found\..*<SKIPLINES>^((?:Request|Referer) URL:.*<SKIPLINES>)*^Client IP: <HOST>' '^(?:Stack |- /)'
But as already said, it is really ugly - better you find the way to log everything in a single line.

Replace a string only from a specific line , While string occurs in multiple lines via ansible

Currently my /etc/default/grub looks like this :
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet selinux=0 idle=poll isolcpus=2-4 intel_idle.max_cstate=1 nosoftlookup nohalt nmi_watchdog=0"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
GRUB_CMDLINE_LINUX=$GRUB_CMDLINE_LINUX" net.ifnames=0 biosdevname=0"
GRUB_CMDLINE_LINUX=$GRUB_CMDLINE_LINUX" isolcpus=2-4"
I need to remove the isolcpus=2-4 only from 6th line , but the last line should stay as it is . So far I have been unable to do so . What ever I have tried removes it from both the lines .
Need to do it via ansible only .
Edit : The expected output should look like :
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet selinux=0 idle=poll intel_idle.max_cstate=1 nosoftlookup nohalt nmi_watchdog=0"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
GRUB_CMDLINE_LINUX=$GRUB_CMDLINE_LINUX" net.ifnames=0 biosdevname=0"
GRUB_CMDLINE_LINUX=$GRUB_CMDLINE_LINUX" isolcpus=2-4"
Not sure if everything is fixed like you said, but you can use
sed '6 s/isolcpus=2-4//' /etc/default/grub
add -i if you like what you get :)
Here is the simplest way you can achieve it using lineinmodule.
- name: line replace
lineinfile:
path: /etc/default/grub
regexp: 'GRUB_CMDLINE_LINUX=\$GRUB_CMDLINE_LINUX" isolcpus=2-4"'
line: 'GRUB_CMDLINE_LINUX=$GRUB_CMDLINE_LINUX" "'

Is it possible to set a top-level only/except in a .gitlab-ci.yml file?

I have three stages in my CI file, they all have only/except like this:
test:
only:
- tags
except:
- branches
script:
- npm run test
Seems redundant to have the only/except in three places. Is there a way to set this at the top level of the script config? Don't see anything like that in the docs.
You can use the map merging feature: https://docs.gitlab.com/ee/ci/yaml/#special-yaml-features
.job_template: &job_definition
only:
- tags
except:
- branches
test1:
<<: *job_definition
script:
- npm run test
test2:
<<: *job_definition
script:
- # ...