Request param is logged in access log with embedded jetty server of spring boot application - regex

I have got an issue with my application, it logs request along with its query param which may contain sensitive data in access log. application is configured with logback.xml & embedded jetty.
jetty server is customized with below accessLogCustomer
public JettyServerCustomizer accessLogCustomizer() {
return server -> {
Slf4jRequestLog requestLog = new Slf4jRequestLog();
requestLog.setExtended(true);
requestLog.setLogLatency(true);
requestLog.setPreferProxiedForAddress(true);
requestLog.setLogTimeZone(userTimezone == null ? ZoneId.systemDefault().getId() : userTimezone);
requestLog.setLogDateFormat("Y-MM-dd HH:mm:ss, SSS Z");
RequestLogHandler requestLogHandler = new RequestLogHandler();
requestLogHandler.setRequestLog(requestLog);
requestLogHandler.setHandler(server.getHandler());
server.setHandler(requestLogHandler);
};
}
logback.xml
<appender name="access" class="ch.qos.logback.core.rolling.RollingFileAppender">
<File>${logs.dir}/abc-access.log</File>
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>%m %n</Pattern>
</layout>
<charset>UTF-8</charset>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<FileNamePattern>${logs.dir}/abc-access.%d.log.gz</FileNamePattern>
</rollingPolicy>
</appender>
<logger name="org.eclipse.jetty.server.RequestLog" additivity="false">
<appender-ref ref="access"/>
</logger>
request logged in access log
192.168.0.100 - - [2021-05-20 15:48:15,093 +0530] "POST /myAPI/v2/customer/message?myID=123&messageText=hello HTTP/1.0" 200 0 "-" "PostmanRuntime/7.26.8" 475
I am trying to avoid messageText from access log, but not getting any solution.

Use the CustomRequestLog and Slf4jRequestLogWriter instead.
You'll want the special format option %U which emits the URL path, without the query string (which is available as %q btw)
Your resulting configuration would look like this ...
Slf4jRequestLogWriter slfjRequestLogWriter = new Slf4jRequestLogWriter();
String format = "%{client}a - %u %t %m \"%U\" %s %O \"%{Referer}i\" \"%{User-Agent}i\"";
CustomRequestLog customRequestLog = new CustomRequestLog(slfjRequestLogWriter, format);
server.setRequestLog(customRequestLog);
Play with the format line, read the Javadoc on CustomRequestLog to know what you can do.
Some notes:
The example format is not strictly following the Extended NCSA format (as it's missing the HTTP version portion, and the HTTP method is outside of the quoted section, but that is usually not a problem for many users)
Slf4jRequestLogWriter is only concerned with taking the formatted log line and sending it to the slf4j-api, it does nothing else.
RequestLogHandler is deprecated and not a recommended usage anymore (as it does not log bad requests and context-less requests), use the Server.setRequestLog(RequestLog) instead.
Jetty will use the CustomRequestLog's Pattern to produce a String, this String is forwarded to the Slf4jRequestLogWriter as a slf4j logging event message, which is then logged per your existing slf4j + logback configuration.

Related

How to properly use/configure YT gem?

I'm trying to reproduce this tutorial : YouTube API, Version 3 on Rails
in order to apply it on my own project. But I'm having a hard with it since few days.
At first, I had this error :
A request to YouTube API caused an unexpected server error: To display
more verbose errors, change the configuration of Yt with: Yt.configure
do |config| config.log_level = :debug end
I updated RVM and Ruby and I'm getting this error now :
Yt::Errors::Forbidden in VideosController#create A request to YouTube
API was considered forbidden by the server: To display more verbose
errors, change the configuration of Yt with: Yt.configure do |config|
config.log_level = :debug end
I already :
get ruby and rvm updated
tried different version of the yt gem
tried that : OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE
tried that : config.force_ssl = false
this
curl -X GET -H "content-length: 0" -H "user-agent: Yt::Request (gzip)" -H "host: www.googleapis.com" "https://www.googleapis.com/youtube/v3/videos?id=wuZfOIWwM_Y&part=snippet"
return that :
Using Rails 4.2.4, Ruby 2.3.0;
Source code at : https://github.com/NeimadTL/YT_Sample_App
Any help, suggestions would be strongly and sincerely appreciated.
forbidden (403) forbidden Access forbidden. The request may not be properly authorized.
Answer: The request you are making is not authorized. update: Change key= to access_token=
Possible cause:
https://www.youtube.com/annotations_invideo?key=
You are trying to run a request annotations_invideo (which I cant actually find any were in the documentation) and you are applying an API key to it. API keys only work with public data. Either annotations_invideo is not a valid request to the API or its something that you need to be authenticated for. If you need to be authenticated then you will need an access token and then apply access_token= instead of key=
where exactly did you find annotations_invideo ?
Update:
Lucky for me it has been under an hour since you posted your question I was able to take
https://www.youtube.com/annotations_invideo?access_token=AIzaSyBSvIOM0EGX1tcrf5IAlYJuH_ttqVgTO4Q&video_id=BPNYv0vd78A
and dump it in a web browser it returned data.
<document>
<annotations>
<annotation author="" id="annotation_1585555999" log_data="ei=B2k9WIOCB8X0dNKokKAG&a-id=annotation_1585555999&xble=1&a-type=4&a-v=BPNYv0vd78A" style="title" type="text">
<TEXT>Hello, world!</TEXT>
<segment>
<movingRegion type="rect">
<rectRegion d="0" h="25.2779998779" t="0:00.000" w="75.0" x="13.1540002823" y="67.3239974976"/>
<rectRegion d="0" h="25.2779998779" t="0:02.089" w="75.0" x="13.1540002823" y="67.3239974976"/>
</movingRegion>
</segment>
<appearance bgAlpha="0.25" bgColor="0" borderAlpha="0.10000000149" effects="" fgColor="16777215" fontWeight="bold" highlightFontColor="16777215" textSize="21.6642"/>
</annotation>
<annotation id="channel:563d3ce4-0000-20cc-8fd5-001a11463304" style="playlist" type="promotion" log_data="ei=B2k9WIOCB8X0dNKokKAG&a-type=12&a-ch=UCwCnUcLcb9-eSrHa_RQGkQQ&xble=1&a-id=563d3ce4-0000-20cc-8fd5-001a11463304&l-class=2&link-id=PLuW4g7xujBWfU26JUTW1DGs3hk4LD5KaL&a-v=BPNYv0vd78A">
<data>
{"playlist_length":"200","session_data":{"itct":"CAIQwTcY____________ASITCMOh497wzdACFUU6HQodUhQEZCj4HTICaXZIwN_33vSX1vkE","annotation_id":"563d3ce4-0000-20cc-8fd5-001a11463304","feature":"iv","ei":"B2k9WIOCB8X0dNKokKAG","src_vid":"BPNYv0vd78A"},"is_mobile":false,"text_line_2":"Adorable Kids","text_line_1":"Check this playlist","image_url":"https:\/\/i.ytimg.com\/vi\/yDrLVqRHAsw\/mqdefault.jpg","start_ms":1000,"collapse_delay_ms":86400000,"end_ms":3000}
</data>
<segment/>
<action trigger="click" type="openUrl">
<url type="hyperlink" target="new" value="https://www.youtube.com/watch?v=yDrLVqRHAsw&list=PLuW4g7xujBWfU26JUTW1DGs3hk4LD5KaL"/>
</action>
</annotation>
</annotations>
</document>
Note: I wonder why this is returning XML and not Json it has me thinking this is an older api. Found it you are using the YouTube API v2 which is deprecated It should have been shut down .
https://youtube-eng.googleblog.com/2014/09/have-you-migrated-to-data-api-v3-yet.html
you should drop this and move to the YouTube API v3

WSO2IS logging client IP into file wso2carbon.log

I have an environment where WSO2IS is behind a balancer and would like the client IP also be logged into file wso2carbon.log beyond http_access.log file.
on application log we need to manually parse the value to log. you shall get the IP address from request. You may add this to Thread local and configure it as pattern layout[5].
String ipAddress = request.getHeader("X-FORWARDED-FOR");
if (ipAddress == null) {
ipAddress = request.getRemoteAddr();
}
You may use request header "X-Forwarded-For"[1].
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="${carbon.home}/repository/logs"
               prefix="http_access_management_console_" suffix=".log"
               pattern="%{X-Forwarded-For}i %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-Agent}i"" />
[2]
[1].https://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#Proxies_Support
[2].https://docs.wso2.com/display/ESB490/Access+Logs
[3].http://tomcat.apache.org/tomcat-6.0-doc/config/valve.html#Access_Log_Valve
[4].http://www.techstacks.com/howto/configure-access-logging-in-tomcat.html
[5].https://logging.apache.org/log4j/2.x/manual/thread-context.html
If you can programmatically get the User's IP, then you can write a User Operation Event Listener [1] and print logs. A sample is in [2].
[1] http://tharindue.blogspot.com/2016/08/user-operation-event-listener-in-wso2.html
[2] http://tharindue.blogspot.com/2016/08/writing-user-operation-event-listener.html

AWS logs agent setup

We have recently setup AWS logs agent on one of our test servers. Our log files usually contain multi-line events. e.g one of our log event is:
[10-Jun-2016 07:30:16 UTC] SQS Post Response: Array
(
[Status] => 200
[ResponseBody] => <?xml version="1.0"?><SendMessageResponse xmlns="http://queue.amazonaws.com/doc/2009-02-01/"><SendMessageResult><MessageId>053c7sdf5-1e23-wa9d-99d8-2a0cf9eewe7a</MessageId><MD5OfMessageBody>8e542d2c2a1325a85eeb9sdfwersd58f</MD5OfMessageBody></SendMessageResult><ResponseMetadata><RequestId>4esdfr30-c39b-526b-bds2-14e4gju18af</RequestId></ResponseMetadata></SendMessageResponse>
)
The log agent reference documentation says to use 'multi_line_start_pattern' option for such logs. Our AWS Log agent config is as follows:
[httpd_info.log]
file = /var/log/httpd/info.log*
log_stream_name = info.log
initial_position = start_of_file
log_group_name = test.server.name
multi_line_start_pattern = '(\[)+\d{2}-[a-zA-Z]{3}+-\d{4}'
However, the logs agent reporting breaks on aforementioned and similar events. The way it is being reported to CloudWatch Logs is as follows:
Event 1:
[10-Jun-2016 11:21:26 UTC] SQS Post Response: Array
Event 2:
( [Status] => 200 [ResponseBody] => <?xml version="1.0"?><SendMessageResponse xmlns="http://queue.amazonaws.com/doc/2009-02-01/"><SendMessageResult><MessageId>053c7sdf5-1e23-wa9d-99d8-2a0cf9eewe7a</MessageId><MD5OfMessageBody>8e542d2c2a1325a85eeb9sdfwersd58f</MD5OfMessageBody></SendMessageResult><ResponseMetadata><RequestId>4esdfr30-c39b-526b-bds2-14e4gju18af</RequestId></ResponseMetadata></SendMessageResponse>
Event 3:
)
Despite of the fact that its only a single event. Any clue whats going on here?
I think all you need to add is the following to your awslogs.conf
datetime_format = %d-%b-%Y %H:%M:%S UTC
time_zone = UTC
multi_line_start_pattern = {datetime_format}
http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html
multi_line_start_pattern
Specifies the pattern for identifying the start of a log message. A log message is made of a line that matches the pattern and any following lines that don't match the pattern. The valid values are regular expression or {datetime_format}. When using {datetime_format}, the datetime_format option should be specified. The default value is ‘^[^\s]' so any line that begins with non-whitespace character closes the previous log message and starts a new log message.
If that datetime format didn't work, you would need to update your regex to actually match your specific datetime. I don't think the one you have listed above actually works for your given format.
You could try this for instance:
[\d{2}-[\w]{3}-\d{4}\s{1}\d{2}:\d{2}:\d{2}\s{1}\w+]
does match
[10-Jun-2016 11:21:26 UTC]
See here: http://www.regexpal.com/?fam=96811
Once completed, issue a restart of the service and check to see if its parsing correctly.
$ sudo service awslogs restart

RegexFilter with RollingFileAppender not working properly

I am trying to use Regexfilter in RollingFileAppender. For 1st matching instance it retreived the logger, but after that I different patttern but nothing is logged in the file. Here is what I am using:
Main Class:
public class MainApp {
public static void main(String[] args) {
final Logger logger = LogManager.getLogger(MainApp.class.getName());
ApplicationContext context = new ClassPathXmlApplicationContext("Beans.xml");
HelloWorld obj = (HelloWorld) context.getBean("helloWorld");
logger.trace("NPF:Trace:Entering Log4j2 Example.");
logger.debug("NTL:debug Entering Log4j2 Example.");
obj.getMessage();
Company comp = new Company();
comp.setCompName("ANC");
comp.setEstablish(1889);
CompanyBusiness compBus = (CompanyBusiness)context.getBean("compBus");
compBus.finaceBusiness(comp.getCompName(), comp.getEstablish());
logger.trace("NTL: Trace: Exiting Log4j2 Example.");
}
}
log4j2.xml:
<?xml version="1.0" encoding="UTF-8" ?>
<Configuration>
<Appenders>
<Console name="STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="%d{yyyy-MM-dd [%t] HH:mm:ss} %-5p %c{1}:%L - %m%X%n" />
</Console>
<RollingFile name="RollingFile" fileName="C:\logTest\runtime\tla\els3.log" append="true" filePattern="C:\logTest\runtime\tla\els3-%d{yyyy-MM-dd}-%i.log" >
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %m%X%n" />
<RegexFilter regex=".*business*." onMatch="ACCEPT" onMismatch="DENY"/>
<Policies>
<SizeBasedTriggeringPolicy size="20 MB" />
</Policies>
</RollingFile>
</Appenders>
<Loggers>
<Logger name="com.anc" level="trace"/>
<Root level="trace">
<AppenderRef ref="STDOUT" />
<AppenderRef ref="RollingFile"/>
</Root>
</Loggers>
</Configuration>
When I ran for the first time, in my logfile I got logs having only "business" related line. Latter I changed the patter from .business (pattern has astreik before and after business word). to "business", logging did not happen in file nor on the console. Also my application terminated without any kind of logging.
Then I tried to revert back the pattern to '.business.' (pattern has astreik before and after business word), thereafter no logging happened on the log file, but on the console all the log trace is printed. When I comment out the Regexfilter after trying for long time, my logs was printed in the log file.
I am not sure if this is a bug of Regexfilter works only for one time. Also if we do not pass any patter matching characters, the application stops without any log printing either on console or file.
If you want to log all events containing the word "business", then you shall use the regex .*business.* instead of .*business*.. Here is an example:
<RegexFilter regex=".*business.*" onMatch="ACCEPT" onMismatch="DENY"/>
For information, .*business*. means: anything, followed by business, followed by s character 0 or more time, followed by any single character.
More explaining:
. means any single character
* means 0 or more times
so .* means any character, 0 or more times.

Regexp: ProFTPD auth-logs

I've configurated ProFTPD to log all authentifications on a Plesk powered server. This setting is not set by default since Plesk 10 (whyever...). I want to configure fail2ban, to detect unsuccessful login-attempts for Brute Force prevention.
/etc/proftpd.include:
ExtendedLog /var/log/proftpd/auth.log AUTH auth
LogFormat auth "%v [%h] %s"
Example of unsuccessfull login (530):
/var/log/proftpd/auth.log:
ProFTPD [12.89.47.3] 331
ProFTPD [12.89.47.3] 530
What's the correct regexp for fail2ban ? My configuration seems not to match the pattern:
/etc/fail2ban/filter.d/proftpd.conf:
failregex = ProFTPD(.)+\[<HOST>\] 530$
<HOST> seems to be a fail2ban variable, and $ ends up a rule (if you want to set several rules inside the failregex = variable).
For me the following is working on Plesk 10.4.
proftpd.include:
ExtendedLog /var/log/proftpd/auth.log AUTH auth
LogFormat auth "%v %t \"%r\" [%h] %s"
/etc/fail2ban/filter.d/proftpd.conf:
failregex = \[<HOST>\]\s+530$