You can easily add more swap to your system if swap is a part of rpool providing there is space available. If you do not have enough space in rpool to increase swap then you are SOL. The only option you have is to create another zfs pool with more luns (disks).
Increasing swap space using the available space in rpool
To increase swap space using the available space in rpool. You may have to delete the current swap device, resize the swap volume, then re-add the swap device.
See how much space you have available on the rpool volume
# zfs list rpool NAME USED AVAIL REFER MOUNTPOINT rpool/swap 2.06G 53.7G 16K -
Looks like we have about 55GB available.
If your swap device is in use, then you will not be able to delete it. To see if the swap area is in use. Do the following:
# swap -l swapfile swapfile dev swaplo blocks free /dev/zvol/dsk/rpool/swap 256,1 16 4194288 4194288
Review the output looking at the blocks and the free columns. If they are equal then swap is not being used for this volume. Now you can delete the swap volume.
# swap -d /dev/zbol/dsk/rpool/swap
Verify that the remove actually worked:
# swap -l No swap devices configured
Resize the swap volume to what we need.
# zfs set volsize=24G rpool/swap # zfs set refreservation=24G rpool/swap
The ZFS refreservation parameter reserves space from the pool that is guaranteed to be available to a dataset. This is important to make sure that we have enough space to use for swap after a reboot.
Now we activate our new larger swap area.
# swap -a /dev/zvol/dsk/rpool/swap # swap -l swapfile dev swaplo blocks free /dev/zvol/dsk/rpool/swap 256,1 16 50331632 50331632
Take a look at our available space now.
# zfs list rpool NAME USED AVAIL REFER MOUNTPOINT rpool 35.2G 31.7G 106K /rpool
Increasing swap space by creating a new ZFS Pool
There are times when you will need to create a new ZFS pool in order to allocate a large amount of swap space. For example if you have a system that has 128GB of RAM. Oracle DBA’s at a minimum would want to have swap space at 75% of RAM. So in this example we would need to have 96GB of swap space.
To accomplish this request we will create a ZFS pool called swappool and assign 2 64GB luns (disks) to it. I am going to give them 100GB instead of 96GB just to make them feel better.
# zpool create -m legacy swappool c0t0d4s1 # zpool add -f swappool c0t0d5s1 # zfs create -b 8192 -V 100g swappool/swap
Now we add our new swap disk to the system.
# swap -a /dev/zvol/dsk/swappool/swap # echo "/dev/zvol/dsk/swappool/swap - - swap - no -" >> /etc/vfstab # swap -l swapfile dev swaplo blocks free /dev/zvol/dsk/rpool/swap 256,1 16 4194288 4194288 /dev/zvol/dsk/swappool/swap 256,5 16 209715184 209715184
The nice thing about having your swap space in a separate pool other than rpool is that you can add and remove luns as needed.