Ultra Monkey and ldirectord [solved]

[ianw1974]

I'm hoping someone's used this (paul might have, so hope he notices this!!) :P

 

Setting up a load-balancing system, on the premise that I'll have two squid servers, that the load-balancer will redirect the requests accordingly. So far, this is my config file:

 

virtual = 172.20.12.231:3128
 real = 172.20.12.232:3128 ipip
 real = 172.20.12.233:3128 ipip

 

the problem is the last bit. There's three config options:

 

gate
ipip
masq

 

and I've no idea which one I'm supposed to use or even if I've got the config file correct in the first place! Service starts though. Hopefully someone has some ideas

 

As you can see, my idea is an ldirector (172.20.12.231) to pass squid proxy requests to either the real server 1 (172.20.12.232) or real server 2 (172.20.12.233). All are three systems and each IP address listed is bound to the network card in each of the machines. There's only one network card in each machine.

[paul]

I haven't had the time to start playing with it yet .. so I can't help.

 

But I'll put this thread on my watch list :)

[ianw1974]

Well, my basic config didn't work, but I'm getting somewhere :P

 

Here is my config now:

 

virtual = 172.20.12.231:3128
 real = 172.20.12.232:3128 gate
 real = 172.20.12.233:3128 gate
 checktype = connect
 protocol = tcp

 

A "netstat -tan" shows a load of high numbers 32xxx port numbers opened pointing to the two squid proxies.

 

The results of "ipvsadm -L -n" show:

 

IP Virtual Server version 1.0.8 (size=65536)
Prot LocalAddress:Port Scheduler Flags
 -> RemoteAddress:Port		   Forward Weight ActiveConn InActConn
TCP  172.20.12.231:3128 wrr
 -> 172.20.12.232:3128		   Masq	1	  0		  0		 
 -> 172.20.12.233:3128		   Masq	1	  0		  0

 

so I'm getting somewhere. I think the problem now seems that port 3128 isn't redirecting as it should be, so I need to try and figure out why. If anyone has managed to do anything with this, your input very much appreciated.

[ianw1974]

Further config file, this time getting somewhere:

 

virtual = 172.20.12.231:3128
	real = 172.20.12.232:3128 masq
	real = 172.20.12.233:3128 masq
	request = "/etc/squid/welcome.msg"
	receive = "Welcome"
	protocol = tcp
	checktype = negotiate

 

and here is the debug, so it looks like it's working kinda, but still unable to get the virtual server to connect to one of the real servers:

 

/usr/sbin/ldirectord -d /etc/ha.d/ldirectord.cf start
DEBUG2: Running exec(/usr/sbin/ldirectord -d /etc/ha.d/ldirectord.cf start)
Running exec(/usr/sbin/ldirectord -d /etc/ha.d/ldirectord.cf start)
DEBUG2: Starting Linux Director v1.77.2.32 with pid: 1533
Starting Linux Director v1.77.2.32 with pid: 1533
DEBUG2: Running system(/sbin/ipvsadm -A -t 172.20.12.231:3128 -s wrr )
Running system(/sbin/ipvsadm -A -t 172.20.12.231:3128 -s wrr )
DEBUG2: Added virtual server: 172.20.12.231:3128
Added virtual server: 172.20.12.231:3128
DEBUG2: Running system(/sbin/ipvsadm -a -t 172.20.12.231:3128 -r 172.20.12.233:3128 -m -w 0)
Running system(/sbin/ipvsadm -a -t 172.20.12.231:3128 -r 172.20.12.233:3128 -m -w 0)
DEBUG2: Quiescent real server: 172.20.12.233:3128 mapped from 172.20.12.233:3128 ( x 172.20.12.231:3128) (Weight set to 0)
Quiescent real server: 172.20.12.233:3128 mapped from 172.20.12.233:3128 ( x 172.20.12.231:3128) (Weight set to 0)
DEBUG2: Disabled server=172.20.12.233
DEBUG2: Running system(/sbin/ipvsadm -a -t 172.20.12.231:3128 -r 172.20.12.232:3128 -m -w 0)
Running system(/sbin/ipvsadm -a -t 172.20.12.231:3128 -r 172.20.12.232:3128 -m -w 0)
DEBUG2: Quiescent real server: 172.20.12.232:3128 mapped from 172.20.12.232:3128 ( x 172.20.12.231:3128) (Weight set to 0)
Quiescent real server: 172.20.12.232:3128 mapped from 172.20.12.232:3128 ( x 172.20.12.231:3128) (Weight set to 0)
DEBUG2: Disabled server=172.20.12.232
DEBUG2: Checking negotiate: real server=negotiate:none:tcp:172.20.12.232:3128:::\/etc\/squid\/welcome\.msg:Welcome (virtual=tcp:172.20.12.231:3128)
DEBUG2: Checking none
DEBUG2: Running system(/sbin/ipvsadm -e -t 172.20.12.231:3128 -r 172.20.12.232:3128 -m -w 1)
Running system(/sbin/ipvsadm -e -t 172.20.12.231:3128 -r 172.20.12.232:3128 -m -w 1)
DEBUG2: Restored real server: 172.20.12.232:3128 ( x 172.20.12.231:3128) (Weight set to 1)
Restored real server: 172.20.12.232:3128 ( x 172.20.12.231:3128) (Weight set to 1)
DEBUG2: Enabled server=172.20.12.232
DEBUG2: Checking negotiate: real server=negotiate:none:tcp:172.20.12.233:3128:::\/etc\/squid\/welcome\.msg:Welcome (virtual=tcp:172.20.12.231:3128)
DEBUG2: Checking none
DEBUG2: Running system(/sbin/ipvsadm -e -t 172.20.12.231:3128 -r 172.20.12.233:3128 -m -w 1)
Running system(/sbin/ipvsadm -e -t 172.20.12.231:3128 -r 172.20.12.233:3128 -m -w 1)
DEBUG2: Restored real server: 172.20.12.233:3128 ( x 172.20.12.231:3128) (Weight set to 1)
Restored real server: 172.20.12.233:3128 ( x 172.20.12.231:3128) (Weight set to 1)
DEBUG2: Enabled server=172.20.12.233
DEBUG2: Checking negotiate: real server=negotiate:none:tcp:172.20.12.232:3128:::\/etc\/squid\/welcome\.msg:Welcome (virtual=tcp:172.20.12.231:3128)
DEBUG2: Checking none
DEBUG2: Enabled server=172.20.12.232
DEBUG2: Checking negotiate: real server=negotiate:none:tcp:172.20.12.233:3128:::\/etc\/squid\/welcome\.msg:Welcome (virtual=tcp:172.20.12.231:3128)
DEBUG2: Checking none
DEBUG2: Enabled server=172.20.12.233

 

so just need to figure out why it's not passing the connection ;)

[ianw1974]

OK, so come to the conclusion that the config files are OK.

 

I'm wondering if it has something to do with my vm's, and not being able to set it up correctly with two separate networks. Although, even configured with ftp config:

 

virtual=192.168.6.240:21
	real=192.168.7.4:21 masq
	real=192.168.7.5:21 masq
	service=ftp
	request="welcome.msg"
	receive="Welcome"
	login="anonymous"
	passwd="anon@anon.anon"
	protocol=tcp
	checktype=negotiate

 

and couldn't get a connection to the ftp server - this is a sample config provided, which effectively should work then. The only other thing, I might be missing a package that I must need to install on the real servers, an arp package. Will come back to it in a bit, need a fresh head on this one methinks :P

[ianw1974]

Nope, arp package red-herring. I installed this and still doesn't work. Hmmm.

 

Incidently, tried telnet with telnet-server installed on the real server, and getting better. At least packets are being passed but nothing with an actual connection as of yet.

 

Will update again soon, and if anyone has an ideas, go for it and post ;)

[ianw1974]

Update:

 

Fixed

 

All working now with telnet, ftp, and unsure about squid at present, as I need to set it up and get it working the other way around. So, for completeness:

 

Director

 

This is what does the load-balancing between the two real-servers. So, I allocated the IP addresses of:

 

eth0 - 172.20.12.231
eth1 - 192.168.0.1

 

in my example, as this is my normal work network. This was on an RHEL3 platform on a virtual machine (vm).

 

Once this was done, the /etc/ha.d/ldirectord.cf file was created:

 

checktimeout = 10
checkinterval = 2
autoreload = no
logfile="/var/log/ldirectord"
quiescent=yes

virtual = 172.20.12.231:23
	real = 192.168.0.2:23 masq
	real = 192.168.0.3:23 masq
	service = none
	request = "/etc/welcome.msg"
	receive = "Welcome"
	scheduler = rr
	protocol = tcp
	checktype = negotiate

virtual = 172.20.12.231:21
	real = 192.168.0.2:21 masq
	real = 192.168.0.3:21 masq
	service = none
	request = "/etc/welcome.msg"
	receive = "Welcome"
	scheduler = rr
	protocol = tcp
	checktype = negotiate

 

It's important that the details in each virtual section are indented as above. The first section is global settings.

 

Now, we need to enable packet forwarding, so, edit /etc/sysctl.conf and find this line:

 

net.ipv4.ip_forward = 1

 

normally it is set to "0", so change it to "1", and then do:

 

sysctl -p

 

to activate it.

 

Next, since I'm using the masq option instead of gate or ipip, you have to add an iptables rule, so:

 

service iptables stop
iptables -t nat -A POSTROUTING -j MASQUERADE 192.168.0.0/24
iptables -t nat -A POSTROUTING -j LOG
/etc/init.d/iptables save

 

the last command saves it (on my system /etc/sysconfig/iptables). Of course, now make sure iptables is running as a service, so:

 

chkconfig iptables on

 

If you're using ftp, add this line to /etc/rc.local or /etc/modprobe.preload if you have it on your system:

 

modprobe ip_vs_ftp (for rc.local)
ip_vs_ftp (for modprobe.preload)

 

and that's the config finished on the director, other than having to do these commands later once the real servers are ready:

 

chkconfig ldirectord on
service ldirectord start

 

Real Servers

 

Two Real Servers were built as vm's and allocated two ips:

 

192.168.0.2
192.168.0.3

 

both HAVE to have the default gateway pointing to the director, so make sure it's pointing to:

 

GATEWAY = 192.168.0.1

 

this is to ensure any requests are sent back to the director, so that it can return them to the clients. And make sure this is correct with:

 

ip route show 0/0

 

Services Running?

 

So, now you've configured it. At the end of the director section I had:

 

chkconfig ldirectord on
service ldirectord start

 

so, let's go ahead and start it then. Then do this:

 

ipvsadm -ln

 

this will show the services running, their virtual ip will be the director, and the addresses that it will get processed to. You can check connections with:

 

ipvsadm -ln --connection

 

and stats with:

 

ipvsadm -ln --stats

 

and you should be done and using a nice cluster of ftp servers, or whatever you happened to configure. Of course, make sure you have a ftp server installed on your system! My config above shows config for an ftp server and telnet server (oops), telnet was for testing so I know it's not secure!!

 

And if you're having connection problems, this will be because your weight=0 in the ipvsadm -ln command. This needs to be 1 for it to be working, and is the weight it assumes if it's not configured.