paul Posted September 20, 2004 Report Share Posted September 20, 2004 I've stumble upon a problem that has me stumped I have a multipath router using 2.6.8.1 with patches from here http://www.ssi.bg/~ja/#routes basic setup ___ ISP1 LAN---ROUTER---[ |___ ISP2 I have also setup ipsec-tools, and loaded the correct modules for IPSEC to work here's my /etc/ipsec.conf #!/usr/sbin/setkey -f flush; spdflush; spdadd 10.99.99.0/24[500] 192.168.10.0/24[500] any -P out ipsec esp/transport/0.0.0.0-192.168.10.x/require; spdadd 192.168.10.0/24[500] 10.99.99.0/24[500] any -P in ipsec esp/tunnel/192.168.10.x-0.0.0.0/require; 192.168.10.0/24 being my internal lan, and 10.99.99.0/24 being my remote lan here's my racoon.conf remote anonymous { exchange_mode aggressive; doi ipsec_doi; generate_policy on; passive on; lifetime time 24 hour; peers_identifier user_fqdn "### <removed to protect the inocent> ###"; proposal_check obey; proposal { encryption_algorithm 3des; hash_algorithm sha1; authentication_method pre_shared_key; dh_group modp1024; } } sainfo anonymous { lifetime time 12 hour; encryption_algorithm 3des; authentication_algorithm hmac_md5; compression_algorithm deflate; } I'm using shorewall as a frontend to IPTABLES grep vpn /etc/shorewall/* hosts:vpn1 eth2:10.99.99.0/24 hosts:vpn2 eth3:10.99.99.0/24 policy:vpn1 local ACCEPT info rules:ACCEPT fw vpn1 udp 500 - rules:ACCEPT fw vpn1 esp - - rules:ACCEPT vpn1 local ICMP - - tunnels:ipsec vpn1 0.0.0.0/0 vpn1 tunnels:ipsec vpn2 0.0.0.0/0 vpn2 zones:vpn1 VPN1 Remote Subnet from ISP1 zones:vpn2 VPN2 Remote Subnet from ISP2 My routing table can be quite clearly described by this document http://www.ssi.bg/~ja/nano.txt NOW .. lets get to the problem :) If initiate a TCP port 80 connection from outside to my ISP1 public IP address, multipath returns the request out the correct gateway. example from 0.0.0.0 --> 202.x.x.x -->192.168.10.x(Web server) --> 192.168.10.x(router) --> 202.x.x.x --> 0.0.0.0 Perfect !!! Like wise if I initiate a TCP port 80 connection from outside to my ISP2 public IP address, multipath returns the request out the correct gateway. example from 0.0.0.0 --> 203.x.x.x -->192.168.10.x(Web server) --> 192.168.10.x(router) --> 203.x.x.x --> 0.0.0.0 PERFECT! if I'm inside my lan, multipath randomly selects from the two gateways. Example: lan-->192.168.10.x(router)-->some randomness--> Either 202.x.x.x or 203.x.x.x PERFECT!!! :) now here comes ipsec .. from outside 10.99.99.x ->202.x.x.x (ipsec does its thing, and passes the packet to routing) --> Either 202.x.x.x or 203.x.x.x ... NOT COOL! if I'm lucky and the packet is returned out the interface I initiated the IPSEC connection from then it works perfect. often I'm not lucky, and ipsec fails because the response is sent out the "other" gateway interface. For testing purposes I have set a static route to a public IP address example: ip route add 205.x.x.x. via 202.x.x.x. dev eth2 src 202.x.x.x now I can ipsec perfectly from 205.x.x.x from the source IP address 10.99.99.x But I want to use this as a "road warrior" so my IP address will be 0.0.0.0/0 which, on the router has two gateways 202.x.x.x and 203.x.x.x Understand? can anybody help? is this perhaps an IPTABLES problem? if so can anybody suggest anything? (I'm not very good at IPTABLES, but shorewall makes it easy for me) is it perhaps a routing issue that I may have missed? any suggestions? THanks Paul. Quote Link to comment Share on other sites More sharing options...
paul Posted September 20, 2004 Author Report Share Posted September 20, 2004 quick update before bed ;) I have completely removed all entries in /etc/ipsec.conf and I can still get a connection using my psk this suggests my problem is not with racoon/ipsec at all, but with my routing just incase anybody is interested, here's a run down of my routing rama-kandra root # ip route list 202.37.230.64/26 dev eth2 proto kernel scope link src 202.37.230.93 192.168.10.0/24 dev eth0 proto kernel scope link src 192.168.10.1 203.96.216.0/24 via 203.96.212.1 dev eth3 proto static src 203.96.213.101 203.96.212.0/23 dev eth3 proto kernel scope link src 203.96.213.101 172.30.0.0/20 dev eth1 proto kernel scope link src 172.30.7.4 127.0.0.0/8 via 127.0.0.1 dev lo scope link rama-kandra root # rama-kandra root # ip rule list 0: from all lookup local 50: from all lookup main 201: from 202.37.230.64/26 lookup 201 202: from 203.96.212.0/23 lookup 202 222: from all lookup 222 32766: from all lookup main 32767: from all lookup default rama-kandra root # rama-kandra root # ip route show table 201 default via 202.37.230.65 dev eth2 proto static src 202.37.230.93 prohibit default proto static metric 1 rama-kandra root # rama-kandra root # ip route show table 202 default via 203.96.212.1 dev eth3 proto static src 203.96.213.101 prohibit default proto static metric 1 rama-kandra root # rama-kandra root # ip route show table 222 default proto static nexthop via 202.37.230.65 dev eth2 weight 1 nexthop via 203.96.212.1 dev eth3 weight 1 rama-kandra root # rama-kandra root # ip link list 1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: <BROADCAST,MULTICAST,PROMISC,UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:40:33:d3:e2:78 brd ff:ff:ff:ff:ff:ff 3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:50:ba:6c:33:33 brd ff:ff:ff:ff:ff:ff 4: eth2: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:00:b5:94:02:b0 brd ff:ff:ff:ff:ff:ff 5: eth3: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:20:ed:14:4c:81 brd ff:ff:ff:ff:ff:ff 6: sit0: <NOARP> mtu 1480 qdisc noop link/sit 0.0.0.0 brd 0.0.0.0 rama-kandra root # and just to prove the routing works rama-kandra root # ip route get 203.96.212.1 203.96.212.1 dev eth3 src 203.96.213.101 cache mtu 1500 advmss 1460 rama-kandra root # rama-kandra root # ip route get 202.37.230.65 202.37.230.65 dev eth2 src 202.37.230.93 cache mtu 1500 advmss 1460 rama-kandra root # rama-kandra root # ip route get 172.30.7.1 172.30.7.1 dev eth1 src 172.30.7.4 cache mtu 1500 advmss 1460 rama-kandra root # rama-kandra root # ip route get 192.168.10.3 192.168.10.3 dev eth0 src 192.168.10.1 cache mtu 1500 advmss 1460 rama-kandra root # rama-kandra root # host google.co.nz google.co.nz has address 216.239.39.99 google.co.nz has address 216.239.57.99 google.co.nz has address 216.239.37.99 rama-kandra root # ip route get 216.239.39.99 216.239.39.99 via 202.37.230.65 dev eth2 src 202.37.230.93 cache mtu 1500 advmss 1460 rama-kandra root # ip route get 216.239.57.99 216.239.57.99 via 203.96.212.1 dev eth3 src 203.96.213.101 cache mtu 1500 advmss 1460 rama-kandra root # oh and of course I have setup MASQ for both eth2 and eth3 rama-kandra root # cat /etc/shorewall/masq ############################################################################### #INTERFACE SUBNET ADDRESS PROTO PORT(S) eth2 eth0 eth3 eth0 #LAST LINE -- ADD YOUR ENTRIES ABOVE THIS LINE -- DO NOT REMOVE rama-kandra root # if anybody is interested I can post my kernel config and the patch for "Linux Advanced Routing and Traffic control" P. Quote Link to comment Share on other sites More sharing options...
paul Posted September 20, 2004 Author Report Share Posted September 20, 2004 anybody? Quote Link to comment Share on other sites More sharing options...
paul Posted September 20, 2004 Author Report Share Posted September 20, 2004 ok .. now I'm just leaving notes so that I consult this thread when I get home :) And it might be of use to someone in the future fwmark routing allows you to use iptables to alter the routing of packets. For example: # packets marked with a "1" go to table 206 ip rule add fwmark 1 table 206 ip rule add fwmark 2 table 207 do some prerouting # packets going to port 500 of my server(202) are marked with "1" iptables -A PREROUTING -t mangle -p udp -d 202.37.230.93 --dport 500 -j MARK --set-mark 1 # packets going to port 500 of my server(203) are marked with "2" iptables -A PREROUTING -t mangle -p udp -d 203.96.213.101 --dport 500 -j MARK --set-mark 2 Set the routes inseide the two new tables # add a route which specifies the source and gateway to each of the tables ip route add 0.0.0.0/0 src 202.37.230.93 via 202.37.230.65 table 206 ip route add 0.0.0.0/0 src 203.96.216.101 via 203.96.212.1 table 207 The above has the net effect that packets going to port 500 of my server go via one gateway. I want traffic for one each VPN to go over it's own dedicated link. Quote Link to comment Share on other sites More sharing options...
paul Posted September 21, 2004 Author Report Share Posted September 21, 2004 more notes: ip rule add prio 206 fwmark 1 table 206 ip route add default via 202.37.230.65 dev eth2 \ src 202.37.230.93 proto static table 206 ip rule add prio 207 fwmark 2 table 207 ip route add default via 203.96.212.1 dev eth3 \ src 203.96.213.101 proto static table 207 Quote Link to comment Share on other sites More sharing options...
paul Posted September 21, 2004 Author Report Share Posted September 21, 2004 fwmark and prerouting the shorewall way #MARK SOURCE DESTINATION PROTOCOL USER/GROUP 1 eth2 202.37.230.93 udp 500 2 eth3 203.96.213.101 udp 501 1 eth0 0.0.0.0/0 udp 500 2 eth0 0.0.0.0/0 udp 501 1 fw 0.0.0.0/0 udp 500 2 fw 0.0.0.0/0 udp 501 I will have to change my racoon.conf listen setting listen { isakmp 202.37.230.93 [500]; isakmp 203.96.213.101 [501]; } sounds good ;) I'll have to try it ! Quote Link to comment Share on other sites More sharing options...
paul Posted September 21, 2004 Author Report Share Posted September 21, 2004 I better ad some round robin DNS entries as well :) in db.loudas.com loudas.com. 60 IN A 202.37.230.93 loudas.com. 60 IN A 203.96.213.101 mail IN A 202.37.230.93 mail IN A 203.96.213.101 www IN A 202.37.230.93 www IN A 203.96.213.101 IN MX 10 mail.loudas.com. now I can add all my CNAMES I guess pop3 IN CNAME loudas.com. smtp IN CNAME loudas.com. webmail IN CNAME loudas.com. Quote Link to comment Share on other sites More sharing options...
paul Posted September 21, 2004 Author Report Share Posted September 21, 2004 note to self: shorewall can be used to mark packets in /etc/shorewall/tcrules but for this to work you must enable TC_ENABLED= Yes in shorewall.conf Note to everyone else. I haven't test this setup yet .. I have just this minute added the rules. I will test it when I get to work tomorrow :) Quote Link to comment Share on other sites More sharing options...
paul Posted September 21, 2004 Author Report Share Posted September 21, 2004 didn't work :( more investigation Quote Link to comment Share on other sites More sharing options...
paul Posted December 14, 2004 Author Report Share Posted December 14, 2004 still working on it :) here's my latest problem routing versus ipsec here's what I sent to the netfilter list I'm using shorewall, 2 ISP connections, and I want to force udp 500 traffic out one particular interface here's what I have so far /etc/shorewall/tcrules: 4 0.0.0.0/0 202.37.230.93 udp 500 4 202.37.230.93 0.0.0.0/0 udp 500 shorewall show mangle | grep MARK 14 4203 MARK udp -- * * 0.0.0.0/0 202.37.230.93 udp dpt:500 MARK set 0x4 0 0 MARK udp -- * * 202.37.230.93 0.0.0.0/0 udp dpt:500 MARK set 0x4 routing: /sbin/ip rule add prio 223 fwmark 4 table 223 /sbin/ip route add default via 202.37.230.65 dev eth2 \ src 202.37.230.93 proto static table 223 routing rules: ip rule show 0: from all lookup local 50: from all lookup main 201: from 202.37.230.64/26 lookup 201 202: from 203.96.212.0/23 lookup 202 222: from all lookup 222 223: from all fwmark 0x4 lookup 223 32766: from all lookup main 32767: from all lookup default ip route show table 223 default via 202.37.230.65 dev eth2 proto static src 202.37.230.93 the load balancing work flawlessly :) however I have racoon (kernel 2.6.8.1) on the firewall, and only sometimes it responds using 202.37.230.93 .. sometimes it uses my other ISP connection .. which is no good :( I've got to the point where I guess I need to use fwmark before routing, but it doesn't seem to work :( Any help Thanks Paul. Quote Link to comment Share on other sites More sharing options...
paul Posted December 14, 2004 Author Report Share Posted December 14, 2004 iptables -A PREROUTING -t mangle -p udp --dport 500 -j MARK --set-mark 4 this needs testing but it should work along with my routing table 223 I may be able to do this in /etc/shorewall/tcrules 4 0.0.0.0/0 0.0.0.0/0 udp 500 I'll have to do some testing to see if it works Quote Link to comment Share on other sites More sharing options...
paul Posted December 14, 2004 Author Report Share Posted December 14, 2004 that didn't work :( next ... /etc/shorewall/start iptables -A PREROUTING -t mangle -p udp --dport 500 -j MARK --set-mark 4 iptables -A POSTROUTING -t mangle -p udp --dport 500 -j MARK --set-mark 4 and the routing /sbin/ip rule add prio 55 fwmark 4 table 55 /sbin/ip route add default via 202.37.230.65 dev eth2 table 55 notes: The routing table is inserted after the main table (50) and before the balanced table (222) my other routing tables have "src 203.96.213.73 proto static table 202" I have removed the src from table 55 because the packet originates from the firewall so terefore doesn't have a known source address. testing now .. .:) Quote Link to comment Share on other sites More sharing options...
paul Posted December 14, 2004 Author Report Share Posted December 14, 2004 didn't work Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.