rsyslog stale file handler with catalina.out - amazon-web-services

Question

Problem:
After deploying a microservices as a war via AWS EBS Tomcat 7 container...noticed that the log rotation which occurs at UTC day boundary leaves a stale inode file.
The log rotation is more of a copy n truncate, which causes a stale file handler for rsyslog, which is listening for changes to catalina.out. What's the best way to prevent stale inode descriptors? Should I specify a rollover policy in logback.xml or logrotate or ...?
output of sudo lsof /var/log/tomcat7/catalina.out (and sudo stat report latest inode)
rsyslogd 18970 root 2r REG 202,1 1250 134754 /var/log/tomcat7/catalina.out
but doesn't match log output of rsyslog in debug mode.
4638.114765354:7fc839b8c700: stream checking for file change on '/var/log/tomcat7/catalina.out', inode 135952/135952file 7 read 0 bytes
Workaround
Stop Tomcat, remove catalina.out, then restart tomcat. This allowed rsyslog to continue streaming of new records.
However, after a few hours, rsyslog fails to stream newer log records to rsyslog destination server. The debug log of rsyslog contains the same inode as output of stat and lsof. If you run
sudo stat /var/log/tomcat7/catalina.out
rsyslog starts streaming again.
Have you noticed rsyslog stop streaming intermittently outside of the log rollover use case?
Why would a sudo stat /var/log/tomcat7/catalina.out cause rsyslog to stream again?

Answer 1

I also have a problem with rsyslog not sending new records as soon as logrotated does its rotation on catalina.out. issuing a stat doesn't quite cut it for me as the problem is tomcat stops writing to catalina.out (!) ...after perusing various forums and blogs, I was able to address this through the following steps:
make sure that we have $WorkDirectory defined in rsyslog configuration; this allows rsyslog write the "state file" for catalina.out (or for any other log file(s) it watches, for that matter)
As noted here on Loggly's blog, you need to stop rsyslog, delete this state file, then restart rsyslog on a postrotate entry.
my logrotate setting for catalina.out (state files are in /var/lib/rsyslog):
/opt/tomcat/logs/catalina.out {
rotate 7
size 50M
notifempty
missingok
postrotate
service rsyslog stop
rm /var/lib/rsyslog/*
service rsyslog start
endscript
}