Extreme Memory Usage When Using Large ipsets

https://github.com/firewalld/firewalld/issues/738

@principis

Description

@principis

What happened:
I’m using this guide to create a large country blocklist using ipsets.

Adding the ipsets takes a long time (around 5 minutes) causes high cpu usage and steadily increases the memory usage.
When reloading, the memory usage increased to 35%+ (of 8 GB).

After the reload (or restart) firewalld uses 1.5 GB memory (without the ipset it uses 26.7 MB). This seems way too much.

I tried to remove the ipset, and run firewall-cmd --reload after which the ipset was removed, but the memory usage stayed the same, 1.5 GB. This also seems like a bug. Restarting firewalld (without the blacklist) fixes the problem, after which the memory usage dropped to around 26 MB.

What you expected to happen:
Not use this much of memory, and reload faster.

How to reproduce it (as minimally and precisely as possible):

countries="af al am ar bd ca cl cn ee fr gt hk id in kp kr la pl ro ru sg ua um us uy uz ve vg vi vn vu wf ye zm zw"

# Create a temporary working directory
ipdeny_tmp_dir=$(mktemp -d -t blacklist-XXXXXXXXXX)
pushd $ipdeny_tmp_dir

# Download the latest network addresses by country file
curl -LO http://www.ipdeny.com/ipblocks/data/countries/all-zones.tar.gz
tar xf all-zones.tar.gz

firewall-cmd -q --permanent --new-ipset=blacklist --type=hash:net \
    --option=family=inet --option=hashsize=4096 --option=maxelem=200000

# Add the address ranges by country per ipdeny.com to the blacklist
for country in $countries; do
    firewall-cmd -q --permanent --ipset=blacklist \
        --add-entries-from-file=./$country.zone && \
        echo "Added $country to blacklist ipset."
done

firewall-cmd --permanent --zone=drop --add-source=ipset:blacklist
firewall-cmd -q --reload

popd
rm -rf $ipdeny_tmp_dir

Anything else we need to know?:

Environment:

  • Firewalld Version: 0.8.4-1.fc32
  • Firewalld Backend: nftables
  • OS: Fedora 32 (Server Edition)
  • Others:
    • nftables: 0.9.3

Updated: