lvs+keepalived+nginx實現高性能負載均衡集群

一、為什麼要使用負載均衡技術？

1、系統高可用性

2、系統可擴展性

3、負載均衡能力

LVS+keepalived能很好的實現以上的要求，LVS提供負載均衡，keepalived提供健康檢查，故障轉移，提高系統的可用性！採用這樣的架構以後很容易對現有系統進行擴展，只要在後端添加或者減少realserver，只要更改lvs的配置文件，並能實現無縫配置變更！

二、LVS+Keepalived介紹

1、 LVS

LVS是一個開源的軟體，可以實現LINUX平台下的簡單負載均衡。LVS是Linux Virtual Server的縮寫，意思是Linux虛擬伺服器。目前有三種IP負載均衡技術（VS/NAT、VS/TUN和VS/DR）；八種調度演算法（rr,wrr,lc,wlc,lblc,lblcr,dh,sh）。

2、 keepalived

Keepalived 是運行在lvs 之上，它的主要功能是實現真實機的故障隔離及負載均衡器間的失敗切換，提高系統的可用性

三、環境：

四台伺服器，系統全為CentOS6.8：

192.168.2.203 master lvs+keepalived

192.168.2.202 backup lvs+keepalived

192.168.2.204 web1（nginx）

192.168.2.205 web2 （nginx）

vip：192.168.2.13

其中nginx已預裝好，這裡不再寫搭建過程

四、搭建並配置

1、分別在backup lvs和master lvs上安裝lvs

1 root@bogon src]# yum -y install ipvsadm

2 已載入插件：fastestmirror

3 設置安裝進程

4 Determining fastest mirrors

5 epel/metalink | 5.4 kB 00:00

6 * base: mirror.lzu.edu.cn

7 … …

8 已安裝:

9 ipvsadm.x86_64 0:1.26-4.el6

11 作為依賴被安裝:

12 libnl.x86_64 0:1.1.4-2.el6

14 完畢！

2、把ipvsadm模塊載入進系統

1 [root@bogon src]# ipvsadm

2 IP Virtual Server version 1.2.1 (size=4096)

3 Prot LocalAddress:Port Scheduler Flags

4 -> RemoteAddress:Port Forward Weight ActiveConn InActConn

5 [root@bogon src]# lsmod | grep ip_vs

6 ip_vs 126897 0

7 libcrc32c 1246 1 ip_vs

8 ipv6 336282 270 ip_vs,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6

3、分別在backup lvs和master lvs上安裝keepalived（keepalived官網：http://www.keepalived.org/）

[root@bogon src]# tar zxf keepalived-1.2.24.tar.gz

[root@bogon src]# cd keepalived-1.2.24

[root@bogon keepalived-1.2.24]# ./configure –sysconf=/etc –with-kernel-dir=/lib/modules/2.6.32-642.3.1.el6.x86_64/

報錯：

configure: error:

!!! OpenSSL is not properly installed on your system. !!!

!!! Can not include OpenSSL headers files. !!!

系統缺少openssl-devel包所致

安裝openssl-devel

root@bogon keepalived-1.2.24]# yum -y install openssl-devel

再次編輯安裝

[root@bogon keepalived-1.2.24]# ./configure –sysconf=/etc –with-kernel-dir=/lib/modules/2.6.32-642.3.1.el6.x86_64/

Keepalived configuration

————————

Keepalived version : 1.2.24

Compiler : gcc

Preprocessor flags : -I/lib/modules/2.6.32-642.3.1.el6.x86_64//include

Compiler flags : -Wall -Wunused -Wstrict-prototypes

Linker flags :

Extra Lib : -ldl -lssl -lcrypto

Use IPVS Framework : Yes

IPVS use libnl : No

IPVS syncd attributes : No

IPVS 64 bit stats : No

fwmark socket support : Yes

Use VRRP Framework : Yes

Use VRRP VMAC : Yes

Use VRRP authentication : Yes

With ip rules/routes : Yes

SNMP keepalived support : No

SNMP checker support : No

SNMP RFCv2 support : No

SNMP RFCv3 support : No

SHA1 support : No

Use Debug flags : No

Stacktrace support : No

Memory alloc check : No

libnl version : None

Use IPv4 devconf : No

Use libiptc : No

Use libipset : No

Build genhash : Yes

Build documentation : No

[root@bogon keepalived-1.2.24]# make && make install

[root@bogon keepalived-1.2.24]# ln -s /usr/local/sbin/keepalived /sbin/

[root@bogon keepalived-1.2.24]# chkconfig –add keepalived

[root@bogon keepalived-1.2.24]# chkconfig –level 35 keepalived on

4、配置keepalived

lvs-master的配置文件如下

[root@bogon keepalived-1.2.24]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {　　　　　　#全局配置部分

# notification_email {　　　　　　#email 通知，基本不用此處所以注釋掉

# acassen@firewall.loc

# failover@firewall.loc

# sysadmin@firewall.loc

# }

# notification_email_from Alexandre.Cassen@firewall.loc

# smtp_server 192.168.200.1

# smtp_connect_timeout 30

router_id LVS_DEVEL　　　　# 設置lvs的id，在一個網路內應該是唯一的

vrrp_skip_check_adv_addr

vrrp_strict

vrrp_garp_interval 0

vrrp_gna_interval 0

}

vrrp_instance VI_1 {　　　　#vrrp實例定義部分

state MASTER　　　　 #設置lvs的狀態，報錯MASTER和BACKUP兩種，必須大寫

interface eth1　　　　#設置對外服務的介面

virtual_router_id 60　　#設置虛擬路由標示，這個標示是一個數字，同一個vrrp實例使用唯一標示

priority 100　　　　#定義優先順序，數字越大優先順序越高，在一個vrrp——instance下，master的優先順序必須大於backup

advert_int 1　　　　#設定master與backup負載均衡器之間同步檢查的時間間隔，單位是秒

authentication {　　#設置驗證類型和密碼

auth_type PASS　　#主要有PASS和AH兩種

auth_pass 1111　　#驗證密碼，同一個vrrp_instance下MASTER和BACKUP密碼必須相同

}

virtual_ipaddress {　　#設置虛擬ip地址，可以設置多個，每行一個

192.168.2.13

}

virtual_server 192.168.2.13 80 {　　#設置虛擬伺服器，需要指定虛擬ip和服務埠

delay_loop 3　　　　#健康檢查時間間隔

lb_algo rr　　　　 #負載均衡調度演算法

lb_kind DR　　　　#負載均衡轉發規則

persistence_timeout 50　　　　#設置會話保持時間，對動態網頁非常有用

protocol TCP　　　　#指定轉發協議類型，有TCP和UDP兩種

real_server 192.168.2.204 80 {　　#配置伺服器節點1，需要指定real server的真實IP地址和埠

weight 1　　　　#設置權重，數字越大權重越高

TCP_CHECK { 　　　　#realserver的狀態監測設置部分單位秒

connect_timeout 3　　　　#超時時間

nb_get_retry 3　　　　　　#重試次數

delay_before_retry 3　　　　#重試間隔

connect_port 80 　　#監測埠

}

real_server 192.168.2.205 80 {

weight 1

TCP_CHECK {

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

connect_port 80

}

LVS-backup的配置文件如下

[root@bogon keepalived-1.2.24]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

# notification_email {

# acassen@firewall.loc

# failover@firewall.loc

# sysadmin@firewall.loc

# }

# notification_email_from Alexandre.Cassen@firewall.loc

# smtp_server 192.168.200.1

smtp_connect_timeout 30

router_id LVS_DEVEL

vrrp_skip_check_adv_addr

vrrp_strict

vrrp_garp_interval 0

vrrp_gna_interval 0

}

vrrp_instance VI_1 {

state BACKUP

interface eth1

virtual_router_id 60

priority 80

advert_int 1

authentication {

auth_type PASS

auth_pass 1111

}

virtual_ipaddress {

192.168.2.13

}

virtual_server 192.168.2.13 80 {

delay_loop 3

lb_algo rr

lb_kind DR

persistence_timeout 3

protocol TCP

real_server 192.168.2.204 80 {

weight 1

TCP_CHECK {

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

connect_port 80

}

real_server 192.168.2.205 80 {

weight 1

TCP_CHECK {

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

connect_port 80

}

5、realserver的配置

兩台web伺服器都要執行下面腳本

[root@bogon www]# cat /etc/rc.d/init.d/realserver.sh

#!/bin/bash

# description: Config realserver lo and apply noarp

SNS_VIP=192.168.2.13

/etc/rc.d/init.d/functions

case “$1” in

start)

ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP

/sbin/route add -host $SNS_VIP dev lo:0

echo “1” >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo “2” >/proc/sys/net/ipv4/conf/lo/arp_announce

echo “1” >/proc/sys/net/ipv4/conf/all/arp_ignore

echo “2” >/proc/sys/net/ipv4/conf/all/arp_announce

sysctl -p >/dev/null 2>&1

echo “RealServer Start OK”

;;

stop)

ifconfig lo:0 down

route del $SNS_VIP >/dev/null 2>&1

echo “0” >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo “0” >/proc/sys/net/ipv4/conf/lo/arp_announce

echo “0” >/proc/sys/net/ipv4/conf/all/arp_ignore

echo “0” >/proc/sys/net/ipv4/conf/all/arp_announce

echo “RealServer Stoped”

;;

echo “Usage: $0 {start|stop}”

exit 1

esac

exit 0

[root@bogon www]# /etc/rc.d/init.d/realserver.sh start

/etc/rc.d/init.d/realserver.sh: line 6: /etc/rc.d/init.d/functions: 許可權不夠

RealServer Start OK

[root@bogon www]# ifconfig

eth0 Link encap:Ethernet HWaddr 00:0C:29:41:71:DF

inet addr:192.168.12.129 Bcast:192.168.12.255 Mask:255.255.255.0

inet6 addr: fe80::20c:29ff:fe41:71df/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:728 errors:0 dropped:0 overruns:0 frame:0

TX packets:98 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:137311 (134.0 KiB) TX bytes:7369 (7.1 KiB)

eth1 Link encap:Ethernet HWaddr 00:0C:29:41:71:E9

inet addr:192.168.2.204 Bcast:192.168.2.255 Mask:255.255.255.0

inet6 addr: fe80::20c:29ff:fe41:71e9/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:119838 errors:0 dropped:0 overruns:0 frame:0

TX packets:31612 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:23411786 (22.3 MiB) TX bytes:2119106 (2.0 MiB)

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:2 errors:0 dropped:0 overruns:0 frame:0

TX packets:2 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:182 (182.0 b) TX bytes:182 (182.0 b)

lo:0 Link encap:Local Loopback

inet addr:192.168.2.13 Mask:255.255.255.255

UP LOOPBACK RUNNING MTU:65536 Metric:1

6、啟動keepalived並進行測試

[root@bogon keepalived-1.2.24]# service keepalived start

正在啟動 keepalived： [確定]

lvs-master

[root@bogon keepalived-1.2.24]# tail -f /var/log/messages

Oct 21 01:19:46 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 21 01:19:46 bogon Keepalived_healthcheckers[6596]: Netlink reflector reports IP 192.168.2.13 added

Oct 21 01:19:46 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13

Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

[root@bogon keepalived-1.2.24]# ipvsadm -L -n

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.2.13:80 rr persistent 3

-> 192.168.2.204:80 Route 1 0 0

-> 192.168.2.205:80 Route 1 0 0

訪問curl http://192.168.2.13/test.txt

[root@www etc]# curl http://192.168.2.13/test.txt

it is web2

[root@www etc]# curl http://192.168.2.13/test.txt

it is web2

關掉web2再次測試

[root@www etc]# curl http://192.168.2.13/test.txt

it is web1

[root@www etc]# curl http://192.168.2.13/test.txt

it is web1

查看lvs-master

[root@bogon keepalived-1.2.24]# ipvsadm -L -n

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.2.13:80 rr persistent 3

-> 192.168.2.204:80 Route 1 0 2

[root@bogon keepalived-1.2.24]# tail -f /var/log/messages

Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 21 01:28:58 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.

Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.

Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Check on service [192.168.2.205]:80 failed after 1 retry.

Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Removing service [192.168.2.205]:80 from VS [192.168.2.13]:80

Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected.

Oct 21 01:29:31 bogon Keepalived_healthcheckers[6596]: Timeout reading data to remote SMTP server [192.168.200.1]:25.

已經自動把web2剔除

打開web2再次查看

[root@bogon keepalived-1.2.24]# ipvsadm -L -n

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.2.13:80 rr persistent 3

-> 192.168.2.204:80 Route 1 0 0

-> 192.168.2.205:80 Route 1 0 0

[root@bogon keepalived-1.2.24]# tail -f /var/log/messages

Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 21 01:28:58 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.

Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.

Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Check on service [192.168.2.205]:80 failed after 1 retry.

Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Removing service [192.168.2.205]:80 from VS [192.168.2.13]:80

Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected.

Oct 21 01:29:31 bogon Keepalived_healthcheckers[6596]: Timeout reading data to remote SMTP server [192.168.200.1]:25.

Oct 21 01:31:01 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 success.

Oct 21 01:31:01 bogon Keepalived_healthcheckers[6596]: Adding service [192.168.2.205]:80 to VS [192.168.2.13]:80

Oct 21 01:31:01 bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected.

恢復後已自動添加進來

關掉lvs master的keepalived

[root@bogon keepalived-1.2.24]# service keepalived stop

停止 keepalived： [確定]

訪問web並查看lvs backup

[root@www etc]# curl http://192.168.2.13/test.txt

it is web2

[root@www etc]# curl http://192.168.2.13/test.txt

it is web2

[root@lys2 src]# tail -f /var/log/messages

Oct 23 19:03:26 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Transition to MASTER STATE

Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Entering MASTER STATE

Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) setting protocol VIPs.

Oct 23 19:03:27 lys2 Keepalived_healthcheckers[13123]: Netlink reflector reports IP 192.168.2.13 added

Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13

Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13

Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13

[root@lys2 src]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 00:0c:29:89:0f:e3 brd ff:ff:ff:ff:ff:ff

inet 192.168.144.101/24 brd 192.168.144.255 scope global eth0

inet6 fe80::20c:29ff:fe89:fe3/64 scope link

valid_lft forever preferred_lft forever

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 00:0c:29:89:0f:ed brd ff:ff:ff:ff:ff:ff

inet 192.168.2.202/24 brd 192.168.2.255 scope global eth1

inet 192.168.2.13/32 scope global eth1

inet6 fe80::20c:29ff:fe89:fed/64 scope link

valid_lft forever preferred_lft forever

可以看到lvs backup已自動切換成master狀態並自動綁定了vip

查看lvs master vip

[root@bogon keepalived-1.2.24]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 00:0c:29:55:4d:7a brd ff:ff:ff:ff:ff:ff

inet 192.168.12.128/24 brd 192.168.12.255 scope global eth0

inet6 fe80::20c:29ff:fe55:4d7a/64 scope link

valid_lft forever preferred_lft forever

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 00:0c:29:55:4d:84 brd ff:ff:ff:ff:ff:ff

inet 192.168.2.203/24 brd 192.168.2.255 scope global eth1

inet6 fe80::20c:29ff:fe55:4d84/64 scope link

valid_lft forever preferred_lft forever

已自動解除vip

到處全部結束

以下文章點擊率最高

Loading…