I need to check who has created the instance or who has stopped/terminated/rebooted instance along with time.
Use AWS Cloud Trail.
Please see the documentation: AWS CloudTrail.
You can get complete history of api calls to your account.
It is not expensive. Check pricing at: AWS CloudTrail Pricing
For Linux, log files are located under the /var/log directory and its subdirectories. Within this directory there are several log files with different names and which record different types of info. Some examples include, but are not limited to:
/var/log/message
Contains global system messages, including the messages that are logged during system startup. Includes mail, cron, daemon, kern, auth, etc.
/var/log/auth.log
Authenication logs
/var/log/kern.log
Kernel logs
/var/log/cron.log
Crond logs
https://blog.logentries.com/2013/11/where-are-my-aws-logs/
You will be able to access the details of the EC2 instance status from the console dashboard for a short period of time.
Until and unless you enable Cloud trail , you wont be able to access the logs and activities of what has happened in the AWS console some days back.
Cloud Trail requires you to use and S3 bucket to store the logs, and the cost you incur for Cloudtrail service is the cost of the space used to store logs in s3.
Related
The other day, I received the following alert in GuardDuty.
Behavior:EC2/NetworkPortUnusual
port:80
Target:3.5.154.156
The EC2 that was the target of the alert was not being used for anything in particular. (However, it had been started up.)
There was no communication using port 80 until now.
Also, the IPAddress of the Target seems to be AWS S3.
The only recent change is that I recently deleted the EC2 InstanceProfile.
Therefore, there is currently no InstanceProfile attached to anything.
Do you know why this EC2 suddenly tried to use port 80 to communicate with the S3 page?
I looked at CloudTrail, etc., and found nothing suspicious.
(If there are any other items I should check, please let me know.)
Thankyou.
We have experienced similar alerts and after tedious debugging we found that SSM agent is responsible for this kind of GuardDuty findings.
SSM Agent communications with AWS managed S3 buckets
"In the course of performing various Systems Manager operations, AWS Systems Manager Agent (SSM Agent) accesses a number of Amazon Simple Storage Service (Amazon S3) buckets. These S3 buckets are publicly accessible, and by default, SSM Agent connects to them using HTTP calls."
I suggest to review CloudTrail logs and look for "UpdateInstanceInformation" event (this is how we found it eventually)
I have an application that is distributed over two AWS accounts.
One part of the application ingest data from one account into the other account.
The producer part is realised as python lambda microservices.
The consumer part is a spring-boot app in elastic beanstalk and additional python lambdas that further distribute data to external systems after they have processed by the spring-boot app in EBeanstalk.
I don't have an explicit X-Ray daemon running anywhere.
I am wondering if it is possible to send the x-ray traces of the one account to the account so i can monitor my application in one place.
I could not find any hints in the documentation regarding cross account usage. Is this even doable ?
If you running X-Ray daemon, you can provide RoleARN to the daemon, so it assumes the role and sends data it receives from X-Ray SDK from Account 1 to Account 2.
However if you have enabled X-Ray on API Gateway or AWS Lambda, segments generated by these services are sent to the account they run in and its not possible to send data cross account for these services.
Please let me know if you have questions. If yes, include the architecture flow and solution stack you are using to better guide you.
Thanks,
Yogi
It is possible but you'd have to run your own xray-daemon as a service.
By default, lambda uses its own xray daemon process to send traces to the account it is running in. However, the X-Ray SDK supports environment variables which can be used to use a custom xray daemon process instead. These environment variables are applicable even if the microservice is running inside a lambda function.
Since your lambda is written in python, you can refer to this AWS Doc which talks about an environment variable. You can set the value to the address of the custom xray daemon service.
AWS_XRAY_DAEMON_ADDRESS = x.x.x.x:2000
Let's say you want to send traces from Account 1 to Account 2. You can do that by configuring your daemon to assume a role. This role must be present in the Account 2 (where you want to send your traces). Then use this role's ARN by passing in the options while running your XRay daemon service in Account 1 (from where you want the traces to be sent). The options to use are mentioned in this AWS Doc here.
--role-arn, arn:aws:iam::123456789012:role/xray-cross-account
Make sure you attach permissions in Account 1 as well to send traces to Account 2.
This is now possible with this recent launch: https://aws.amazon.com/about-aws/whats-new/2022/11/amazon-cloudwatch-cross-account-observability-multiple-aws-accounts/.
You can link accounts to another account to share traces, metrics, and logs.
As far as I understand there are special logs (like for EC2 Instances, RDS) which can be enabled to collect metrics and are not enabled automatically. Is there an OVERVIEW somewhere stating for which resources do special AWS Logs exist and how they can be enabled (via CloudFormation)?
If not I would suggest to create such an Overview since AWS DOCUMENTATION doesn’t have it.
Merci A
Amazon CloudWatch collects metrics from AWS services that you use. (Or, more accurately, the services you use send metrics to Amazon CloudWatch.)
The documentation for each service shows the metrics that are collected.
The only logs that go beyond this are special ones sent from within an Amazon EC2 instance. By default, the Amazon EC2 service can only examine your virtual machine ('instance') from outside of the instance. This provides basic metrics for CPU, disk and network. It is not possible to see inside your instance.
However, if you run additional software inside the instance that communicates with the operating system, then the software can send additional metrics to Amazon CloudWatch. This is typically done with the CloudWatch Agent.
See: Collect Metrics and Logs from Amazon EC2 Instances and On-Premises Servers with the CloudWatch Agent - Amazon CloudWatch
There are additional Metrics Collected by the CloudWatch Agent that can provide details such as free memory and free disk space because the agent communicates with the Operating System running inside the instance. These metrics are then sent to Amazon CloudWatch as custom metrics.
The CloudWatch Agent can also send application logs from an Amazon EC2 instance to Amazon CloudWatch Logs, which retains the individual log entries and can trigger alarms based on their content.
The CloudWatch Agent must be installed into the Amazon EC2 instance, either via a startup script ('User Data' script) or via AWS Systems Manager.
Similarly, Amazon RDS can send database logs to Amazon CloudWatch Logs. However, I couldn't see an option in Amazon CloudFormation to activate these log exports.
I have an AWS account, and have several EC2 servers and an ElasticSearch domain set up to take the syslogs from these servers. However, in Cloudwatch and when investigating a specific server instance in the EC2 control panel, I see specific metrics and graphs for things like CPU, memory load, storage use, etc. Is there some way I can pipe this information into my ElasticSearch as well?
Set up Logstash and use this plugin https://github.com/EagerELK/logstash-input-cloudwatch
Or go the other way and use AWS Logs agent to put your syslogs into Cloudwatch and stop using ElasticSearch
I am programming a Jersey service on Tomcat via EBS with LoadBalancer. I am finding getting the EC2's S3 catalina files very cumbersome. Currently I need to determine the EC2 instance(s) then work my way to each of the S3 locations, download the files, then I can diagnose.
The snapshot doesn't help due to the amount of requests that come in, it doesn't hold enough info and by the time I get the snapshot, it has "rolled" off the snapshot.
Two questions:
1) Is there an easier approach to logs files via AWS? (Increase time before rotation which I don't believe is supported as of now, scripts, etc)
2) Is there any software or scripts to access all the logs under load balancer? I am basically wanting to say "give me all logs for this EBS" and have it get all logs for that day under all servers for that load balancer (up or down)". The clincher is down. Problem becomes more complex when the load balancer takes down an instance right when the issue occurs.
Thanks!
As an immediate solution to your problem you can follow the approach suggested in this answer. Essentially you can modify the logrotate configuration to rotate for a bigger log size using ebextensions.
Then snapshot logs should work for you.
Let me know if you need more clarifications on this approach.
AWS has released CloudWatch Logs just last week, which enables you to to monitor and troubleshoot your systems and applications using your existing system, application, and custom log files:
You can send your existing system, application, and custom log files to CloudWatch Logs and monitor these logs in near real-time. [...] you can store your logs using highly durable, low-cost storage for later access.
See the introductory blog post Store and Monitor OS & Application Log Files with Amazon CloudWatch for an illustrated walk through, which touches on using Elastic Beanstalk and CloudWatch Logs already - this is further detailed in Using AWS Elastic Beanstalk with Amazon CloudWatch Logs.