Docker swarm containers not able to access internet - amazon-web-services

Question

I am trying to setup a swarm cluster in AWS, however the containers in the host are not able to access the internet. The ping command for both address resolution or direct connectivity via IP is not working from inside the container.
Before creating this ticket I had a look at this issue, but I don't think there is CIDR overlap in my case.
I have the following configurations:
Public Subnet CIDR : 10.2.1.0/24
Namespace server inside this is :10.2.0.2
Ingress overlay network --> 10.255.0.0/16
docker_gwbridge --> 172.18.0.0/1
I have also tried creating the new overlay(192.168.1.0/24) and docker_gwbridge(10.11.0.0/16) network with no luck.
I am creating the service with these options(removing the mount and env parameters):
docker service create --publish 8098:8098 <Imagename>
Please note when I was creating the overlay network by myself I was adding the option --network my-overlay as well in the create command.
Any pointers as to what I might be missing/doing wrong?
Edit 1 Adding more info
Below is the inspect of container when I am not creating a new overlay network and going with the default one:
"NetworkSettings": {
"Bridge": "",
"SandboxID": "eb***",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"5005/tcp": null,
"8080/tcp": null
},
"SandboxKey": "/var/run/docker/netns/e***9",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"ingress": {
"IPAMConfig": {
"IPv4Address": "10.255.0.4"
},
"Links": null,
"Aliases": [
"30**"
],
"NetworkID": "g7w**",
"EndpointID": "291***",
"Gateway": "",
"IPAddress": "10.255.0.4",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:4***"
}
And below is from when I am creating the overlay network:
"Networks": {
"ingress": {
"IPAMConfig": {
"IPv4Address": "10.255.0.4"
},
"Links": null,
"Aliases": [
"42***"
],
"NetworkID": "jl***3",
"EndpointID": "792***86c",
"Gateway": "",
"IPAddress": "10.255.0.4",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:4***"
},
"my-overlay": {
"IPAMConfig": {
"IPv4Address": "192.168.1.3"
},
"Links": null,
"Aliases": [
"42**"
],
"NetworkID": "4q***",
"EndpointID": "4c***503",
"Gateway": "",
"IPAddress": "192.168.1.3",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:4***"
}

Answer 1

I am answering my question as I found out that the reason for this behavior was my custom chef recipe for docker installation. I was setting up iptables=false in the docker config and hence it was not working for any docker container other than those in host network mode.
I got the following advice from Bret(Docker champion in docker community) which helped me to get to the root of the problem. In short it was a issue with something I was doing wrongly, however posting the suggestion below in case you want to troubleshoot such issues in future.
Hey Manish,
Suggestion: get a single container working correctly without swarm or overlays before trying them.
so you should be able to just docker run --rm nginx:alpine ping 8.8.8.8 and get a response.
That verifies that containers on that host have a way to the internet.
Then trying docker run --rm nginx:alpine ping google.com and get a response.
That verifies DNS resolution is working.
*Then you can try creating a single overlay network on one node in a single node swarm:*
*docker swarm init *
*docker network create --driver overlay --attachable mynet *
*docker run --rm --network mynet nginx:alpine ping google.com *
That verifies they have internet and DNS on a overlay network.
If you then add multiple nodes and have issues, then you likely need to ensure all swarm nodes can talk over swarm ports, which you find a link to the firewall port list in The Swarm Section under the Creating a 3-Node Swarm Cluster resources.

Answer 2

As Manish said, first try to ping public network without overlay network:
docker run --rm nginx:alpine ping 8.8.8.8
If it doesn't work, then you have a problem with firewall or something else.
In my case, iptables firewall restricted DOCKER-USER chain to get access public network.
So I have flushed all docker rules:
sudo iptables -F DOCKER-USER
Then reinitialized:
sudo iptables -I DOCKER-USER -i eth0 -s 0.0.0.0/0 -j ACCEPT

Answer 3

I had a similar issue and was able to fix it by configuring the daemon.json file in the /etc/docker directory. Add following lines if they are not present already.
"iptables":true,
"dns": ["8.8.8.8", "8.8.4.4"]
Your daemon.json file should look something like below
{
"labels": ....,
"data-root": ....,
"max-concurrent-downloads": ....,
"iptables":true,
"dns": ["8.8.8.8", "8.8.4.4"]
}
Then restart the docker service
sudo service docker restart