You can easily add more swap to your system if swap is a part of rpool providing there is space available. If you do not have enough space in rpool to increase swap then you are SOL. The only option you have is to create another zfs pool with more luns (disks).

Increasing swap space using the available space in rpool

To increase swap space using the available space in rpool. You may have to delete the current swap device, resize the swap volume, then re-add the swap device.

See how much space you have available on the rpool volume

# zfs list rpool
NAME         USED  AVAIL  REFER  MOUNTPOINT
rpool/swap  2.06G  53.7G    16K  -

Looks like we have about 55GB available.

If your swap device is in use, then you will not be able to delete it. To see if the swap area is in use. Do the following:

# swap -l swapfile
swapfile                   dev  swaplo  blocks    free
/dev/zvol/dsk/rpool/swap 256,1      16 4194288 4194288

Review the output looking at the blocks and the free columns. If they are equal then swap is not being used for this volume. Now you can delete the swap volume.

# swap -d /dev/zbol/dsk/rpool/swap

Verify that the remove actually worked:

# swap -l
No swap devices configured

Resize the swap volume to what we need.

# zfs set volsize=24G rpool/swap
# zfs set refreservation=24G rpool/swap

The ZFS refreservation parameter reserves space from the pool that is guaranteed to be available to a dataset. This is important to make sure that we have enough space to use for swap after a reboot.

Now we activate our new larger swap area.

# swap -a /dev/zvol/dsk/rpool/swap

# swap -l
swapfile                   dev  swaplo   blocks     free
/dev/zvol/dsk/rpool/swap 256,1      16 50331632 50331632

Take a look at our available space now.

# zfs list rpool
NAME    USED  AVAIL  REFER  MOUNTPOINT
rpool  35.2G  31.7G   106K  /rpool

Increasing swap space by creating a new ZFS Pool

There are times when you will need to create a new ZFS pool in order to allocate a large amount of swap space. For example if you have a system that has 128GB of RAM. Oracle DBA’s at a minimum would want to have swap space at 75% of RAM. So in this example we would need to have 96GB of swap space.

To accomplish this request we will create a ZFS pool called swappool and assign 2 64GB luns (disks) to it. I am going to give them 100GB instead of 96GB just to make them feel better.

# zpool create -m legacy swappool c0t0d4s1
# zpool add -f swappool c0t0d5s1
# zfs create -b 8192 -V 100g swappool/swap

Now we add our new swap disk to the system.

# swap -a /dev/zvol/dsk/swappool/swap
# echo "/dev/zvol/dsk/swappool/swap     -       -       swap    -       no      -" >> /etc/vfstab
# swap -l
swapfile                      dev  swaplo    blocks      free
/dev/zvol/dsk/rpool/swap    256,1      16   4194288   4194288
/dev/zvol/dsk/swappool/swap 256,5      16 209715184 209715184

The nice thing about having your swap space in a separate pool other than rpool is that you can add and remove luns as needed.