After enabling network policy logging on a vpc-native cluster, it turned out that some suspicious ICMP traffic is blocked.
According to the log json payload, the Internet ICMP traffic is somehow reaching pods (including those which are not exposed by any service or ingress). Example log below:
"src": {
"instance": "redacted_public_ip"
},
"node_name": "redacted_node_name",
"count": 1,
"disposition": "deny",
"dest": {
"workload_name": "redacted_workload_name",
"workload_kind": "ReplicaSet",
"pod_namespace": "redacted_pod_namespace",
"namespace": "redacted_namespace",
"pod_name": "redacted_pod_name"
},
"connection": {
"protocol": "icmp",
"dest_ip": "redacted_private_pod_ip",
"direction": "ingress",
"src_ip": "redacted_public_ip"
}
There are multiple entries like the one above, and public IPs are owned by multiple different organisations and located in different countries. What might be the next step with investigating this issue?
Simply block ICMP unless you really need it. There are two basic types of ICMP, one used for routing and the other for ping-pong messages. You do not need either one enabled.
The next tip is that there is nothing to investigate. The public Internet will poke and prod every public IP address non-stop. Otherwise, you will need to deploy a firewall and blocklists to block known bad actors.
Related
We have recently analyzed our AWS data transfers/NAT gateway charges and what we have observed is that we are sending 80% of the traffic to an AMAZON service in this IP address range.
{
"ip_prefix": "3.237.107.0/25",
"region": "us-east-1",
"service": "AMAZON",
"network_border_group": "us-east-1"
}
When looking at AWS public IP address range in this link
https://ip-ranges.amazonaws.com/ip-ranges.json
This just mentions the above range as "AMAZON" but doesn't give many details about it.
Anyway can we know which AWS service falls under this IP address category?
Tried looking at S3/Dynamo DB/RDS/Elastic Cache and other AWS services we were using and this doesn't fall under those services.
I found this question (very helpful)
Cannot ping AWS EC2 instance
So I'm setting up my EC2 instance so I can ping it.
But I don't want to allow ping requests from anyone on the Internet. I want to limit the source to my current location's subnet.
When I setup ssh, I also limited who can ssh to my EC2 instance but don't recall how I set it up.
Searching for a solution
I found this: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/security-group-rules-reference.html#sg-rules-ping which was helpful to only allow ICMP Echo Requests (protocol 1).
Later I found this command which helped me find the one I created earlier.
aws ec2 describe-security-groups
{
"SecurityGroups": [
{
"Description": "launch-wizard-2 created 2021-07-16T14:45:19.317-04:00",
"GroupName": "launch-wizard-2",
"IpPermissions": [
{
"FromPort": 80,
"IpProtocol": "tcp",
"IpRanges": [
{
"CidrIp": "192.168.1.0/24" // NOTE THIS IS NOT REAL
}
],
"Ipv6Ranges": [],
"PrefixListIds": [],
"ToPort": 80,
"UserIdGroupPairs": []
},
So now I realize the one that was previously created is only setup for TCP (not ICMP) so it won't work for me.
Still searching for how specify a source CIDR block to limit who can ping my EC2 instance. For the sake of discussion, lets say my CIDR block was 123.321/16 I'm trying to figure out how to enter that when creating the inbound rule.
Well, 123.321 cannot exist as a CIDR as the number 321 is too big. However, if it could, your inbound security group rule would need to look like the following:
You could also specify 123.321.123.12/32 for the exact IP of whatever is trying to ping it.
Looking for advice on what I'm missing. Fargate container fails to start with the error CannotPullContainerError: Error response from daem
EDIT: Longer error response
Status reason CannotPullContainerError: Error response from daemon: Get https://{account}.dkr.ecr.{region}.amazonaws.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I created a non-default VPC with CIDR block 50.10.0.0/16. In the VPC, I have 3 public subnets with an internet gateway attached and subnet route table with 0.0.0.0/0 as a record. (I am able to launch an EC2 instance in these subnets, it received a public IP, and I can connect to it via SSH.) My VPC also has DNS hostname and DNS resolution enabled.
On the Task definition side, I created a Fargate task using network mode = awsvpc, requires FARGATE compatibility, and is trying to pull a container from a private ECR that looks like {account}.dkr.ecr.{region}.amazonaws.com/{imagename}:latest. Launching the container I use platform v1.3.0 and enable AUTO-ASSIGN PUBLIC IP. In my security group, I allow All Traffic both inbound and outbound on 0.0.0.0/0. For my Role, I am using one that has the following policies:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kms:*",
"secretsmanager:*",
"ssm:*",
"logs:*",
"sqs:*",
"s3:*",
"ecr:*",
"ecs:*",
"ec2:*"
],
"Resource": "*"
}
]
}
and trusted entities for ecs.amazonaws.com and ecs-tasks.amazonaws.com.
Watching the Task as it goes from PENDING to STOPPED, I can see that the ENI receives a public IP. Always, I get the same error. Thanks in advance!
I figured it out...I had my DHCP options set configured incorrectly.
Question
Is there a way to log the connections blocked by the AWS Network Firewall, or filter the logs of blocked connections?
Background
Currently having setup the rules, and would like to know which IP or domains have been blocked.
Looking at Logging network traffic from AWS Network Firewall but not clear if it is possible.
You can record flow logs and alert logs from your Network Firewall stateful engine.
Flow logs are standard network traffic flow logs. Each flow log record captures the network flow for a specific 5-tuple.
Alert logs report traffic that matches your stateful rules that have an action that sends an alert. A stateful rule sends alerts for the rule actions DROP and ALERT.
From the flow logs, it is not clear if it is passed or blocked.
{
"firewall_name": "network-firewall-sagemaker-studio-anfw",
"availability_zone": "us-east-1a",
"event_timestamp": "1628236046",
"event": {
"timestamp": "2021-08-06T07:47:26.000068+0000",
"flow_id": 1108238612337889,
"event_type": "netflow",
"src_ip": "51.222.5.114",
"src_port": 57528,
"dest_ip": "10.2.2.60",
"dest_port": 8088,
"proto": "TCP",
"netflow": {
"pkts": 1,
"bytes": 40,
"start": "2021-08-06T07:46:24.365793+0000",
"end": "2021-08-06T07:46:24.365793+0000",
"age": 0,
"min_ttl": 239,
"max_ttl": 239
},
"tcp": {
"tcp_flags": "02",
"syn": true
}
}
}
Yes, you can get network logs.
AWS Network Firewall is a managed service that you can use to deploy essential network protections for your Amazon Virtual Private Cloud instances. AWS Network Firewall works together with AWS Firewall Manager so you can build policies based on AWS Network Firewall rules and then centrally apply those policies across your VPCs and accounts.
https://docs.aws.amazon.com/athena/latest/ug/querying-network-firewall-logs.html
How to create amazon cloudwatch logs
https://docs.aws.amazon.com/network-firewall/latest/developerguide/logging-cw-logs.html
AWS Network Firewall logging destinations
https://docs.aws.amazon.com/network-firewall/latest/developerguide/firewall-logging-destinations.html
In your firewall , Log configuration. Please enable logs for Alerts, you can then choose where the Alert ( Logs from blocked requests) should be shipped. You will have options like s3 or cloudwatch log group.
Good luck.
I'm using a pivotal cloud foundry to host my application. I need to whitelist slack-webhooks URL IP in the Application Security Group(ASG's). For now, I have hardcoded 3 IPs.
{
"description": "Slack-ip-1",
"destination": "13.226.19.152",
"protocol": "tcp",
"ports": "443"
},
{
"description": "Slack-ip-2",
"destination": "13.249.130.154",
"protocol": "tcp",
"ports": "443"
},
{
"description": "Slack-ip-3",
"destination": "13.227.32.151",
"protocol": "tcp",
"ports": "443"
}
But Slack uses AWS and there is a high possibility that these IPs will change frequently. Instead of hardcoded ips can I whitelist domain like hooks.slack.com?
Unfortunately no. Application security groups translate down to iptables rules and using domain names instead of IP addresses there is not recommended (for security and performance reasons).
I think your best bet would be to have a cron script that runs periodically (maybe daily) which runs dig or host and resolves hooks.slack.com to fetch the IP addresses. If those change, you could either send yourself an alert, probably through slack :), so you can be reminded to update your ASGs or you could perhaps automatically update your ASGs from the script. Depends on how fancy you want to get.
Hope that helps!