Ganeti

https://we.riseup.net/riseup+tech/ganeti

Usage

export INSTANCE=gl.dapple.local.example.org
export GANETI_HOST=dapple.local.example.org

Instance installation

. If there's no proper DNS record for the new host: Add new instance to /etc/hosts 2. Find an unused MAC and IP addr:

```
gnt-network info
export IP=10.27.64.132
```

Find out available disk space for new instance: vgs

Bullseye guests

If bullseye is not the default SUITE in /etc/default/ganeti-instance-debootstrap, so you need to manually set it.

Then:

gnt-instance add -t plain -o debootstrap+bullseye -B memory=8G,vcpus=2 -s 60G --net 0:mac=00:01:02:00:00:02,ip=${IP},network=varac-dmz1 -n $GANETI_HOST $INSTANCE

Watch it build:

tail -F /var/log/ganeti/os/*

When build finishes:

grep password /var/log/ganeti/os/add-debootstrap*

gnt-instance console $INSTANCE

or

ssh $INSTANCE

Remove cache:

rm /var/cache/ganeti-instance-debootstrap/cache-*

Modify instance settings

Show instance settings:

gnt-instance info plex.nucy.example.org

Modify the ganeti instances on the host:

gnt-instance modify -H initrd_path="" testjabber
gnt-instance modify -H kernel_path="" testjabber
gnt-instance reboot testjabber

Increase memory/RAM of an instance:

gnt-instance info $INSTANCE | grep memory
gnt-instance modify -B memory=2G float.example.org
gnt-instance reboot float.example.org

Add second CPU:

gnt-instance modify -B vcpus=4 $INSTANCE
gnt-instance reboot $INSTANCE

Add new template

Add new variant in /etc/ganeti/instance-debootstrap/variants
Add new variant in /etc/ganeti/instance-debootstrap/variants.list

Stop/shutdown instance

https://wiki.osuosl.org/ganeti/common_commands.html#stopping-an-instance

gnt-instance shutdown $INSTANCE

Fast shutdown:

gnt-instance shutdown --timeout=0 $INSTANCE

Removing an instance completely

gnt-instance remove --shutdown-timeout=1 emby.nucy.example.org

Storage

http://docs.ganeti.org/ganeti/current/html/design-disks.html https://wiki.osuosl.org/ganeti/common_commands.html#multi-vg-support

gnt-instance info $INSTANCE | grep -A5 disk/

gnt-instance add -t drbd --disk 0:vg=ganeti-ssd,size=10G -o image+debian \
  --no-start -I hail <instance>.osuosl.org

Add disk

https://wiki.osuosl.org/ganeti/common_commands.html#adding-disks

gnt-instance modify --disk add:vg=seagate2tb_crypt_vg,size=1.8t,name=$INSTANCE-storage $INSTANCE
gnt-instance reboot $INSTANCE

Issue with adding disks > 1tb:

https://groups.google.com/forum/#!topic/ganeti/EcclQ4-0KVY

gnt-cluster modify --ipolicy-bounds-specs min:cpu-count=1,disk-count=1,disk-size=1024,memory-size=128,nic-count=1,spindle-use=1/max:cpu-count=8,disk-count=8,disk-size=8t,memory-size=32768,nic-count=8,spindle-use=8

Remove disk

gnt-instance info $INSTANCE
gnt-instance modify --disk 1:remove --hotplug $INSTANCE
gnt-instance reboot $INSTANCE

Mount instance system disk on host

https://docs.ganeti.org/docs/ganeti/2.17/html/admin.html#accessing-an-instance-s-disks

i.e. to change /etc/fstab entries.

Install kpartx:

apt install kpartx

Map and mount:

kpartx -av /dev/dapple-vg/51a02a7d-ad66-4bfd-9f4e-75752356bc16.disk0
mount /dev/mapper/dapple--vg-51a02a7d--ad66--4bfd--9f4e--75752356bc16.disk0p1 /mnt
...
umount /mnt
kpartx -dv /dev/dapple-vg/51a02a7d-ad66-4bfd-9f4e-75752356bc16.disk0

Auto-start

autostart = yes. I need to put gnt-watcher in cron, so i will run from time to time. My question is when and how this attribute is created ? Whenever you start the instance with "gnt-instance startup", it is set to yes. When you shutdown the instance with "gnt-instance shutdown", it is set to no.

Ganeti administration

Installation

Prerequisites:

space left on vg
IPs
- Main IP for the host 192.168.178.33/24
- Ganeti Floating IP 192.168.178.3

[x] = done by ganeti ansible role

[x] apt install ganeti passwdqc debootstrap
[x] Configure br0 with systemd-networkd
[x] Enable ip forwarding: echo 'net.ipv4.ip_forward = 1' > /etc/sysctl.d/ip_forward.conf sysctl -f /etc/sysctl.d/ip_forward.conf
[ ] Fix interface enumeration (ens13 instead of ens4) ???
[x] Configure SUITE and EXTRA_PKGS in /etc/default/ganeti-instance-debootstrap
[x] Copy hooks and templates to /etc/ganeti/instance-debootstrap/
[ ] Add floating IP to /etc/hosts:
192.168.178.8 ganeti.nucy.example.org

Initialize the cluster:

gnt-cluster init --master-netdev br0 --vg-name dapple-vg --enabled-hypervisors kvm --enabled-disk-templates plain --nic-parameters link=br0 --mac-prefix 00:01:02 $GANETI_HOST
gnt-cluster modify -H kvm:kernel_path=""

gnt-cluster verify
gnt-cluster info | grep -E '(Cluster name|Master node)'

Create guest network

❯ ipcalc 10.27.64.128/29
  Address:   10.27.64.128         00001010.00011011.01000000.10000 000
  Netmask:   255.255.255.248 = 29 11111111.11111111.11111111.11111 000
  Wildcard:  0.0.0.7              00000000.00000000.00000000.00000 111
  =>
  Network:   10.27.64.128/29      00001010.00011011.01000000.10000 000
  HostMin:   10.27.64.129         00001010.00011011.01000000.10000 001
  HostMax:   10.27.64.134         00001010.00011011.01000000.10000 110
  Broadcast: 10.27.64.135         00001010.00011011.01000000.10000 111
  Hosts/Net: 6                     Class A, Private Internet

gnt-network add --network 10.27.64.128/29 --gateway 10.27.64.130 varac-dmz1
gnt-network connect -N link=br0 varac-dmz1 default
gnt-network info varac-dmz1

Change cluster IPs

Make a backup before changing IPs !

systemctl stop ganeti.service
cd /var/lib/ganeti
grep -l 192.168.1 -ir

# "Physical" IP of host
sed -i 's/192.168.1.2/192.168.178.33/g' ssconf_node_secondary_ips ssconf_node_primary_ips ssconf_master_candidates_ips config.data

# Ganeti floating/cluster IP
sed -i 's/192.168.1.8/192.168.178.3/g' ssconf_master_ip config.data

Make sure hostname -f shows the same host name which is configured in ganeti !

systemctl start ganeti.service

The remove unneeded networks:

gnt-network disconnect internal
gnt-network remove internal

Change cluster/master name

https://groups.google.com/g/ganeti/c/m6tH3DSij6w/m/BLQlnouCZbQJ

ssh $ganeti_master
old=old_node_name
new=new_node_name
cd /var/lib/ganeti/
systemctl stop ganeti
sed -i -e "s/$old/$new/g" config.data ssconf_* known_hosts
systemctl start ganeti
gnt-node list | grep -e "$old|$new"

Upgrade

Important: Update Ganeti before upgrading the OS !!!

https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bullseye#ganeti-upgrades

USB passthrough

https://bytemeta.vip/repo/ganeti/ganeti/issues/1633

$ root@nucy:~# lsusb | grep ConBee
Bus 001 Device 004: ID 1cf1:0030 Dresden Elektronik ZigBee gateway [ConBee II]

$ gnt-instance modify --hypervisor-parameters 'kvm_extra=-device usb-host\,vendorid=0x1cf1\,productid=0x0030' dock