Monitoring the Beanstalk production environment for errors(code level) - amazon-web-services

Question

We have our production site in Elastic Beanstalk. SNS notifications is good feature to keep us updated about the environment status whenever it changes. But, we want to watch the production environment logs closely.
Our project is a java webapplication, we want to check the status of the production environment from other beanstalk environments i.e., beta and staging environment which are also in the same region and within the same application.
Our goals are to
use aws sdk or other aws tools to get the production beanstalk tomcat logs and display in our beta site on some page.
Run some tool periodically from the Beta environment on Live environment. Which basically does the testing of the sites, i.e., whether all code level mappings are good, if any exceptions then email them.
if we break down the point2 into further more -
We have quartz scheduler to schedule a job at a particular time. We are planning to add some script which test the complete environment periodically. Are there any Beanstalk built in tools that tests the complete site, accessing all URLS and testing the DB to java serialize object classes mappings (hibernate mappings) etc.,
We do use S3 elastic beanstalk bucket to check tomcat logs, but would like to implement the step1 & step2 if possible.
--
Thanks

Answer 1

For Item #1:
I don't recommend using beta and dev to watch production. Instead, here's what I'd do:
Setup Pingdom on all the three environments, so we could have a close eye on uptime
Review the Logging Code. Do you have a explicitly pattern/idiom for exception handling in place? Are your logging functioning?
Setup Papertrail with Logback. Why? You'll have realtime aggregate log tailing for each and every machine you setup a syslog receiver for. For beanstalk-maven-plugin, we are about to release an archetype (see an example 'blank' project created out of it). Even if you're not using, its worth it so see how to use it.
Setup Log Rollout to S3. As it is, the usage is quite useless. I suggest you work into something to import for analysis (or better yet: Export for usage from Hive - Which is something Papertrail Does)
Define your Health Check Code accordingly. Think about what could go wrong, in terms of dependencies
Look / Set up some CloudWatch metrics. If you application is heavy and you're on a t1.micro, which conditions it would spike? Use that at your advantage
That are just a few ideas.
w/r/t Item #2:
I suggest you rethink your structure. I actually dislike the idea of using crontab in elastic beanstalk servers, since its error prone (leader_only? Managing output?). Instead, I use my newer favourite crontab webapp - Jenkins, and set up an integration testing / smoke testing artifact, with only the relevant bits to remotely test the instance. Selenium might help, but I guess if your Services are critical, you might be more happy relying in rest-assured, for instance.
Hope it helps.