How to make a TCP socket work with SO_BINDTODEVICE (against routing table) - c++

Question

Background of the question:
On our machine we have multiple network interfaces which lead to different networks. There are possible overlapping IP addresses (e.g. two different machines in two different networks with the same IP address). So when we want to connect with specific peer then we need to specify not only it's IP address but also our network interface which lead to the proper network.
We want to write application in C/C++ able to connect with specific peers via TCP.
Question:
I'm trying to make a TCP connection using socket with SO_BINDTODEVICE set. Here is a simplified snippet:
sockfd = socket(AF_INET, SOCK_STREAM, 0);
setsockopt(sockfd, SOL_SOCKET, SO_BINDTODEVICE, interface_name,
strlen(interface_name));
connect(sockfd, (sockaddr *) &serv_addr, sizeof(serv_addr));
(I know about struct ifreq, but it seems that only the first field in it (ifr_name field is in use). So I can pass only name of the interface.)
If forced interface is the same as interface according to the routing table, then everything works correctly.
If forced interface is different, then (tested with Wireshark):
SYN is sent from forced interface to desired peer.
SYN,ACK from desired peer is received on forced interface.
ACK is not sent from forced interface and connection is not established. (And goto step 2.)
How to check where SYN,ACK or ACK is rejected by our system? And how correctly force TCP socket to make connection using specific interface (against routing table)?
Maybe there are some other, more convenient ways to create TCP connection on desired interface?
Thanks!

Answer 1

I know it wouldn't be your quite answer, but you could disable other interfaces and just enable the network you want, in your case it seems that you need all the interfaces, but I think this approach could help others. you could enable/disable network interface with something like this :
enable
ifr.ifr_flags = true;
strcpy(ifr.ifr_name, "eth0"); //you could put any interface name beside "eth0"
res = ioctl(sockfd, SIOCSIFFLAGS, &ifr);
and for disable you just need to set flag to false and the rest of the code is the same :
ifr.ifr_flags = true;

Answer 2

Don't use SO_BINDTODEVICE. Its not supported on all platforms and there's an easier way.
Instead bind the socket to the local IP address on the correct network that you want to use to connect to the remote side.
Ie,
sockfd = socket(AF_INET, SOCK_STREAM, 0);
struct sockaddr_in sin;
memset(&sin, 0, sizeof(sin));
sin.sin_family = AF_INET;
sin.sin_port = 0; //Auto-determine port.
sin.sin_addr.s_addr = inet_addr("192.168.1.123"); //Your IP address on same network as peer you want to connect to
bind(sockfd, (sockaddr*)&sin, sizeof(sin));
Then call connect.
For the server side you'd do the same thing except specify a port instead of 0, then call listen instead of connect.

Answer 3

This is a problem with kernel configuration - on many distributions it is by default configured to reject incoming packets in this specific case.
I found the solution to this problem in this answer to another similar question:
To allow such traffic you have to set some variables on your machine (as root):
sysctl -w net.ipv4.conf.all.accept_local=1
sysctl -w net.ipv4.conf.all.rp_filter=0
sysctl -w net.ipv4.conf.your_nic.rp_filter=0
where your_nic is the network interface receiving the packet. Beware to change both net.ipv4.conf.all.rp_filter and net.ipv4.conf.your_nic.rp_filter, it will not work otherwise (the kernel defaults to the most restrictive setting).