Cuju: An Open Source Project for Virtualization-Based Fault Tolerance *Provide active-passive fault tolerance service*
================================================================
Virtualization technology could provide a unique benefit to protect any legacy application systems from hardware failures. The reliability of virtual machines running on virtualized servers is not only threatened by hardware failures beneath the whole virtual infrastructure, but also nosy hypervisors that essentially support virtual machines cannot be trusted.
We develop the opensource tool, Cuju, which is a virtualization based fault tolerance technique with epoch-based synchronization. There are several performance optimization technologies are applied by Cuju, including a non-stop/pipelined, continuously migration, dirty tracking for guest virtual memory/virtual device status, and eliminate data transfer between QEMU and KVM.
Cuju shows that these optimizations have greatly saved the processor usage, synchronization bandwidth and have significantly improved VM network throughput and latency at the same time.
For more information see: https://cuju-ft.github.io/cuju-web/home.html
A recommended topology below:
Another recommended topology below:
$ sudo apt-get update
ubuntu-16:
$ sudo apt-get install ssh vim gcc make gdb fakeroot build-essential \
kernel-package libncurses5 libncurses5-dev zlib1g-dev \
libglib2.0-dev qemu xorg bridge-utils openvpn vncviewer \
libssl-dev libpixman-1-dev nfs-common git
ubuntu-18:
$ sudo apt-get install ssh vim gcc make gdb fakeroot build-essential \
kernel-package libncurses5 libncurses5-dev zlib1g-dev \
libglib2.0-dev qemu xorg bridge-utils openvpn libelf-dev \
libssl-dev libpixman-1-dev nfs-common git tigervnc-viewer
/etc/network/interfaces
):auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address 192.168.11.1
netmask 255.255.255.0
gateway 192.168.11.254
dns-nameservers 8.8.8.8
eth0 is your physical NIC name, please modify it according to your actual NIC name
auto lo
iface lo inet loopback
auto br0
iface br0 inet static
bridge_ports eth0
bridge_maxwait 0
address 192.168.11.2
netmask 255.255.255.0
gateway 192.168.11.254
dns-nameservers 8.8.8.8
auto eth0
iface eth0 inet static
address 0.0.0.0
auto eth1
iface eth1 inet static
address 192.168.111.1
netmask 255.255.255.0
auto lo
iface lo inet loopback
auto br0
iface br0 inet static
bridge_ports eth0
bridge_maxwait 0
address 192.168.11.3
netmask 255.255.255.0
gateway 192.168.11.254
dns-nameservers 8.8.8.8
auto eth0
iface eth0 inet static
address 0.0.0.0
auto eth1
iface eth1 inet static
address 192.168.111.2
netmask 255.255.255.0
Build the high-speed connections (ex. 10G NIC) with Primary and Backup nodes by the eth1
After editing these network interfaces, type /etc/init.d/networking restart
or reboot
$ sudo apt-get install nfs-kernel-server
/etc/exports
to add your NFS folder: /home/[your username]/nfsfolder *(rw,no_root_squash,no_subtree_check)
After editing /etc/exports
, type /etc/init.d/nfs-kernel-server restart
or reboot
Go to your nfs folder, then download Cuju and build a VM image file (or download our Ubuntu-16.04 VM image file, the account/password
is root/root
), they will be synced with Primary and Backup node.
$ sudo mkdir /mnt/nfs
$ sudo mount -t nfs 192.168.11.1:/home/[your username]/nfsfolder /mnt/nfs
$ sudo apt-get install linux-image-4.15.0-29-generic
$ sudo apt-get install linux-headers-4.15.0-29-generic
$ sudo vim /etc/default/grub
#GRUB_DEFAULT=0
GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 4.15.0-29-generic"
$ sudo update-grub
Reboot after the above modification
$ cd /mnt/nfs
$ git clone https://github.com/Cuju-ft/Cuju.git
$ cd Cuju
$ ./configure --enable-cuju --enable-kvm --disable-pie --disable-xen --target-list=x86_64-softmmu
$ make clean
$ make -j8
*1
*2
$ cd Cuju/kvm
$ ./configure
$ make clean
$ make -j8
$ ./reinsmodkvm.sh
P.S.
*1
If you meeterror: incompatible type for argument 5 of '__get_user_pages_unlocked'
, you can use this patch:$ cd Cuju $ patch -p1 < ./patch/__get_user_pages_unlocked.patch
*2
If you meeterror: implicit declaration of function 'use_eager_fpu' [-werror=implicit-function-declaration]
, you can use this patch:$ cd Cuju $ patch -p1 < ./patch/use_eager_fpu.patch
$ cd /mnt/nfs/Cuju/kvm
$ ./reinsmodkvm.sh
runvm.sh
sudo ./x86_64-softmmu/qemu-system-x86_64 \
-drive if=none,id=drive0,cache=none,format=raw,file=/mnt/nfs/Ubuntu20G-1604.img \
-device virtio-blk,drive=drive0 \
-m 1G -enable-kvm \
-net tap,ifname=tap0 -net nic,model=virtio,vlan=0,macaddr=ae:ae:00:00:00:25 \
-cpu host \
-vga std -chardev socket,id=mon,path=/home/[your username]/vm1.monitor,server,nowait -mon chardev=mon,id=monitor,mode=readline
You need to change the guest image path (file=/mnt/nfs/Ubuntu20G-1604.img
) and monitor path (path=/home/[your username]/vm1.monitor
) for your environment
$ vncviewer :5900 &
The default account/password
is root/root
if you use we provide guest image
recv.sh
sudo x86_64-softmmu/qemu-system-x86_64 \
-drive if=none,id=drive0,cache=none,format=raw,file=/mnt/nfs/Ubuntu20G-1604.img \
-device virtio-blk,drive=drive0 \
-m 1G -enable-kvm \
-net tap,ifname=tap1 -net nic,model=virtio,vlan=0,macaddr=ae:ae:00:00:00:25 \
-vga std -chardev socket,id=mon,path=/home/[your username]/vm1r.monitor,server,nowait -mon chardev=mon,id=monitor,mode=readline \
-cpu host \
-incoming tcp:0:4441,ft_mode
recv.sh
sed -e 's/mode=readline/mode=readline -incoming tcp\:0\:4441,ft_mode/g' -e 's/vm1.monitor/vm1r.monitor/g' -e 's/tap0/tap1/g' ./runvm.sh > tmp.sh
chmod +x ./tmp.sh
./tmp.sh
ftmode.sh
ubuntu-16:
sudo echo "migrate -c tcp:192.168.111.2:4441" | sudo nc -U /home/[your username]/vm1.monitor
ubuntu-18:
sudo echo "migrate -c tcp:192.168.111.2:4441" | sudo nc -w 1 -U /home/[your username]/vm1.monitor
You need to change the ip address and port (tcp:192.168.111.2:4441
) for your environment, this is Backup Host's IP
And change the monitor path (/home/[your username]/vm1.monitor
) for your environment
If you successfully start Cuju, you will see the following message show on Primary side:
If you want to test failover
You can kill
or ctrl-c
VM on the Primary Host
You will need new session with vncviewer:
$ vncviewer :5900 &
$ vncviewer :5901 &