OpenFOAM Supports Apptainer v1.2.0 Template Definition File Feature

https://ciq.com/blog/openfoam-supports-apptainer-v1-2-0-template-definition-file-feature/

OpenFOAM Supports Apptainer v1.2.0 Template Definition File Feature July 25, 2023 OpenFOAM Supports Apptainer v1.2.0 Template Definition File Feature Table of contents Introduction Apptainer Template Definition File Basics Template variables %arguments section –build-arg option –build-arg-file option Hands-on Guide to OpenFOAM Official Apptainer Definition File Clone OpenFOAM containers repository Apptainer Template Definition File for OpenFOAM Build with Default Value (OS_VER=latest, FOAM_VERSION=2306) Build OpenFOAM v2212 Developer Image on Rocky Linux 9.2 (OS_VER=9.2, FOAM_VERSION=2212) Build OpenFOAM v2306 Default Image on Rocky Linux 8.8 (OS_VER=8.8, PACKAGE=openfoam2306-default) Run motorBike Example using OpenFOAM v2306 Default Apptainer Image Customise Definition File for Your OpenFOAM Development Project Your own OpenFOAM Apptainer Image on Docker Hub Acknowledgement Appendix Available tags for Rocky Linux images OpenFOAM packages Contributors Yoshiaki Senda Subscribe to our newsletter Enter your email Subscribe

Introduction We are happy to announce that OpenFOAM® (OpenCFD Ltd.) is the very first project that officially supports the brand-new Apptainer v1.2.0 feature “template definition file” just released on July 18, 2023. OpenFOAM officially supports Apptainer since OpenFOAM v2212 and since then we have closely worked together on the OpenFOAM container repository.

Screenshot 2023 07 04 at 22 57 39 1024x779

In this blog post, we will explain (1) Apptainer “template definition file” basics, (2) how to use OpenFOAM’s official Apptainer template definition file to build your own brand-new OpenFOAM v2306 container, (3) how to create your own OpenFOAM development environment based on the official template, and (4) finally, we will show you how to share your own OpenFOAM development environment with your collaborators using Docker Hub. Apptainer helps drive researchers,’ developers,’ and engineers’ collaboration by providing “Mobility of Compute.”

Apptainer Template Definition File Basics The new “template definition file” feature introduces template variables to every section of the Apptainer definition file. Before we introduced these template variables, environment variables were used for similar purposes and still can be used, but it only works for limited sections of the Apptainer definition file (e.g., environment variables can not be used for bootstrap OS image names).

Template variables With template variables, for example, we can switch bootstrap OS image versions like the following:

Bootstrap: docker From: rockylinux/rockylinux:{{ OS_VER }} %arguments section To support default values for template variables, an %arguments section is added to the Apptainer definition file. Let’s make a file called rockylinux-template.def with the following contents.

Bootstrap: docker From: rockylinux/rockylinux:{{ OS_VER }}

%arguments OS_VER=8.8 With a default value, you can build this definition file as usual:

apptainer build rockylinux8.sif rockylinux-template.def Now a Rocky Linux 8.8 container called rockylinux8.sif is built.

./rockylinux8.sif cat /etc/os-release NAME=”Rocky Linux” VERSION=”8.8 (Green Obsidian)” ID=”rocky” ID_LIKE=”rhel centos fedora” VERSION_ID=”8.8” PLATFORM_ID=”platform:el8” PRETTY_NAME=”Rocky Linux 8.8 (Green Obsidian)” ANSI_COLOR=”0;32” LOGO=”fedora-logo-icon” CPE_NAME=”cpe:/o:rocky:rocky:8:GA” HOME_URL=”https://rockylinux.org/” BUG_REPORT_URL=”https://bugs.rockylinux.org/” SUPPORT_END=”2029-05-31” ROCKY_SUPPORT_PRODUCT=”Rocky-Linux-8” ROCKY_SUPPORT_PRODUCT_VERSION=”8.8” REDHAT_SUPPORT_PRODUCT=”Rocky Linux” REDHAT_SUPPORT_PRODUCT_VERSION=”8.8” –build-arg option The new option is added to apptainer build command, –build-arg, and –build-arg-file. When you want to bump up the OS version to 9.2, here is the time to use that new option –build-arg.

apptainer build –build-arg OS_VER=9.2 rockylinux9.sif rockylinux-template.def This time, a Rocky Linux 9.2 container called rockylinux9.sif is built.

./rockylinux9.sif cat /etc/os-release NAME=”Rocky Linux” VERSION=”9.2 (Blue Onyx)” ID=”rocky” ID_LIKE=”rhel centos fedora” VERSION_ID=”9.2” PLATFORM_ID=”platform:el9” PRETTY_NAME=”Rocky Linux 9.2 (Blue Onyx)” ANSI_COLOR=”0;32” LOGO=”fedora-logo-icon” CPE_NAME=”cpe:/o:rocky:rocky:9::baseos” HOME_URL=”https://rockylinux.org/” BUG_REPORT_URL=”https://bugs.rockylinux.org/” SUPPORT_END=”2032-05-31” ROCKY_SUPPORT_PRODUCT=”Rocky-Linux-9” ROCKY_SUPPORT_PRODUCT_VERSION=”9.2” REDHAT_SUPPORT_PRODUCT=”Rocky Linux” REDHAT_SUPPORT_PRODUCT_VERSION=”9.2” –build-arg-file option Create the file called args.list(or whatever name and extension):

cat « EOF > args.list OS_VER=9.2 EOF You can build the same Apptainer image as above with –build-arg-file option. The file-based approach is convenient when you want to keep the build command the same but change the build settings by just changing the arguments list (e.g., CI/CD pipelines) and/or when you want to share the build settings.

apptainer build –build-arg-file args.list rockylinux9.2.sif rockylinux-template.def Again, a Rocky Linux 9.2 container called rockylinux9.2.sif is built. Let’s move on to the hands-on guide to OpenFOAM official Apptainer definition file 🙂.

Hands-on Guide to OpenFOAM Official Apptainer Definition File The OpenFOAM official container repository is here https://develop.openfoam.com/packaging/containers .

Screenshot 2023 07 20 at 14 54 46

Clone OpenFOAM containers repository First, clone the container repository and change the directory to container/docker.

git clone https://develop.openfoam.com/packaging/containers.git cd containers/docker Apptainer Template Definition File for OpenFOAM The following is the content of openfoam-run_rocky-template.def, the OpenFOAM Apptainer template definition file based on a Rocky Linux image. We can switch OS versions by using the OS_VER template variable, can change the OpenFOAM package using the PACKAGE template variable, and can change the OpenFOAM version by using the FOAM_VERSION template variable when we build images on the fly. Let’s see examples.

———————————-sh-————————————

Copyright (C) 2021-2022 OpenCFD Ltd.

SPDX-licence-Identifier: (GPL-3.0+)

Create openfoam ‘-run’ image for Rocky Linux using copr repo.

Example

apptainer build openfoam2306.sif openfoam-run_rocky-template.def

apptainer build –build-arg OS_VER=8 –build-arg FOAM_VERSION=2306 …

—————————————————————————

Bootstrap: docker From: rockylinux/rockylinux:{{ OS_VER }}

%files openfoam-files.rc/* /openfoam/

%arguments OS_VER=latest FOAM_VERSION=2306 PACKAGE=openfoam{{ FOAM_VERSION }}-devel

%post dnf -y install wget rsync dnf -y install ‘dnf-command(config-manager)’ dnf -y install epel-release crb enable dnf -y install ‘dnf-command(copr)’ dnf -y copr enable openfoam/openfoam dnf -y install {{ PACKAGE }} dnf -y clean all

%post /bin/sh /openfoam/assets/post-install.sh -fix-perms -no-sudo

%runscript exec /openfoam/run “$@”

%labels Author OpenCFD Ltd. Author CIQ Inc.

—————————————————————————

Build with Default Value (OS_VER=latest, FOAM_VERSION=2306) This is the simplest example using default values specified in the %arguments section of the template definition file. rockylinux/rockylinux:latest points to the latest version of Rocky Linux 8.x, such as 8.8 here today.

apptainer build openfoam2306.sif openfoam-run_rocky-template.def This command builds the image for an OpenFOAM v2306 development environment on Rocky Linux 8.8 without OpenFOAM tutorials.

Build OpenFOAM v2212 Developer Image on Rocky Linux 9.2 (OS_VER=9.2, FOAM_VERSION=2212) This one builds the image for an OpenFOAM v2212 development environment on Rocky Linux 9.2 without OpenFOAM tutorials.

apptainer build –build-arg OS_VER=9.2 –build-arg FOAM_VERSION=2306
openfoam2212-dev.sif openfoam-run_rocky-template.def Build OpenFOAM v2306 Default Image on Rocky Linux 8.8 (OS_VER=8.8, PACKAGE=openfoam2306-default) This one builds the image for an OpenFOAM v2306 development environment on Rocky Linux 8.8 that includes OpenFOAM tutorials inside an image. Let’s use this image at the next step 🙂:

apptainer build –build-arg OS_VER=8.8 –build-arg PACKAGE=openfoam2306-default
openfoam2306-default.sif openfoam-run_rocky-template.def Before we move on to the next step, let’s deploy this OpenFOAM Apptainer image to your PATH.

If you don’t have an ~/bin directory, create the directory first. If you are working on Rocky Linux or equivalent derivatives, you can skip this step.

mkdir -p ~/bin If you are working on Ubuntu or equivalent derivatives, make sure ~/bin is set to your PATH. If you are working on Rocky Linux or equivalent derivatives, you can skip this step.

export PATH=~/bin:$PATH echo ‘export PATH=~/bin:$PATH’ » ~/.bashrc Move openfoam2306-default.sif to ~/bin .

mv openfoam2306-default.sif ~/bin Now you are ready to go to the next step!

Run motorBike Example using OpenFOAM v2306 Default Apptainer Image Make sure you are on your ~/ home directory. Create an openfoam2306 directory and change the directory there.

cd ~/ mkdir openfoam2306 && cd $_ Execute the container image you just created at the previous step:

openfoam2306-default.sif This gives you an OpenFOAM shell that has all the necessary environment variables already set and ready to roll.

========= | \ / F ield | OpenFOAM in a container [from OpenCFD Ltd.] \ / O peration | \ / A nd | www.openfoam.com \/ M anipulation | ————————————————————————— Release notes: https://www.openfoam.com/news/main-news/openfoam-v2306 Documentation: https://www.openfoam.com/documentation/ Issue Tracker: https://develop.openfoam.com/Development/openfoam/issues/ Local Help: more /openfoam/README ————————————————————————— System : Rocky Linux 8.6 (Green Obsidian) OpenFOAM : /usr/lib/openfoam/openfoam2306 Build : _fbf00d6b-20230626 OPENFOAM=2306 patch=0

Note Different OpenFOAM components and modules may be present (or missing) on any particular container installation. Eg, source code, tutorials, in-situ visualisation, paraview plugins, external linear-solver interfaces etc. ————————————————————————— apptainer-openfoam2306:~/openfoam2306 ysenda$ Check that the tutorials path exists on your container.

apptainer-openfoam2306:~/openfoam2306 ysenda$ echo $FOAM_TUTORIALS/ /usr/lib/openfoam/openfoam2306/tutorials/ Copy the OpenFOAM tutorials to your home directory. This home directory is the home directory on your host machine bind mounted to your container home directory. Apptainer is automatically bind mounted to your home directory on the container filesystem, so you can access your data under the home directory without explicitly mounting to the container like the other container solutions.

apptainer-openfoam2306:~/openfoam2306 ysenda$ cp -r $FOAM_TUTORIALS . Change the directory to motorBike tutorial:

apptainer-openfoam2306:~/openfoam2306 ysenda$ cd tutorials/multiphase/interFoam/RAS/motorBike If you are working on 64 cores system, you can change parallel run settings by doing the following. The default settings of number of parallel processes is 5. You can skip this step if you are good with default settings.

cat « EOF > system/decomposeParDict /——————————–- C++ -———————————-
| ========= | | | \ / F ield | OpenFOAM: The Open Source CFD Toolbox | | \ / O peration | Version: 4.x | | \ / A nd | Web: www.OpenFOAM.org | | \/ M anipulation | | *—————————————————————————*/ FoamFile { version 2.0; format ascii; class dictionary; note “mesh decomposition control dictionary”; object decomposeParDict; } // * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 64; method scotch; EOF Run the tutorial using the Allrun script. This will run the motorBike example using MPI (default: 5 MPI parallel processes). The motorBike example is a bit heavy workload, so you can quit simulation by Ctrl + c .

apptainer-openfoam2306:~/openfoam2306/tutorials/multiphase/interFoam/RAS/motorBike ysenda$ ./Allrun Customise Definition File for Your OpenFOAM Development Project Does your OpenFOAM development project need additional libraries? YES? No problem, we can customise the official definition file to fit your needs.

For example, for the “drlfoam project” (deep reinforcement learning with OpenFOAM) at https://github.com/OFDataCommittee/drlfoam, it seems like the project requires Python 3.8, PyTorch 1.12.1 (CPU only version), pandas, numpy, and PyTorch API for C++. Let’s add those dependencies to the definition file. See line 36 to line 51, as you may notice customising the definition file is pretty straight-forward. You don’t need to learn tool specific grammar; it’s just the same as what you do on a Linux system.

We are not going to verify this definition file works for “drlfoam” in this article. This is intended to show you how easy it is to customise the definition file for your project.

———————————-sh-————————————

Copyright (C) 2021-2022 OpenCFD Ltd.

SPDX-licence-Identifier: (GPL-3.0+)

Create openfoam ‘-run’ image for Rocky Linux using copr repo.

Example

apptainer build openfoam2306.sif openfoam-run_rocky-template.def

apptainer build –build-arg OS_VER=8 –build-arg FOAM_VERSION=2306 …

—————————————————————————

Bootstrap: docker From: rockylinux/rockylinux:{{ OS_VER }}

%files openfoam-files.rc/* /openfoam/

%arguments OS_VER=8.8 FOAM_VERSION=2206 PACKAGE=openfoam{{ FOAM_VERSION }}-devel

%post /bin/sh /openfoam/assets/post-install.sh -fix-perms -no-sudo

%post dnf -y install unzip dnf install -y python38 python38-pip dnf clean all cat « EOF > requirements.txt –find-links https://download.pytorch.org/whl/torch_stable.html torch==1.12.1+cpu pandas numpy EOF pip3 install -r requirements.txt rm requirements.txt

wget -q -O libtorch.zip https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.6.0%2Bcpu.zip
unzip libtorch.zip -d opt/
rm libtorch.zip

%runscript exec /openfoam/run “$@”

%labels Author OpenCFD Ltd. Author CIQ Inc.

—————————————————————————

You can build the above example definition file with the following command:

apptainer build drlfoam.sif drlfoam-openfoam-run_rocky-template.def Your own OpenFOAM Apptainer Image on Docker Hub Reproducibility is important in many cases. When you work with collaborators, sharing your development environment with those collaborators is kind of pain point. The combination of Apptainer (formerly Singularity) and container registries can help with that part.

Once you build your own development environment as an Apptainer container, you can push that Apptainer container image to OCI registries (see more details here: https://ciq.com/blog/using-apptainers-with-public-cloud-container-artifact-registry-offerings/).

The best well-known OCI registry is Docker Hub. Docker Hub usually hosts Docker container images like the following:

Screenshot 2023 07 05 at 11 47 53 1024x671

But Docker Hub is also capable of hosting Apptainer container images because they supports ORAS protocol. We can push an Apptainer container image to Docker Hub using the apptainer command line tool that also supports ORAS protocol.

First, login to Docker Hub. Please replace USERNAME to fit your case. You will be asked for a Password/Token; please enter your token here.

apptainer remote login –username USERNAME oras://docker.io Once you have signed into Docker Hub, you can create a personal access token here.

After successful login, push your image to Docker Hub.

apptainer push awesomefoam-dev.sif oras://docker.io/USERNAME/awesomefoam-dev:2306 If your repository is the public repository, your collaborators can pull this image without authentication.

apptainer pull oras://docker.io/USERNAME/awesomefoam-dev:2306 Now your collaborators can start developing the project using exactly the same environment as you do 🎉.

./awesomefoam-dev_2306.sif

Acknowledgement We would like to say thank you to those who participated in the feature request discussions.

Mark Olesen (OpenCFD Ltd.)

Xu Yang

Dave Dykstra

Dave Trudgian

Dave Godlove

Davis Catherman

Gregory M. Kurtzer

Jonathon Anderson

Yoshiaki Senda

Appendix Available tags for Rocky Linux images You can find tags here https://hub.docker.com/r/rockylinux/rockylinux/tags .

Screenshot 2023 07 05 at 12 29 38 1024x876

OpenFOAM packages Currently available OpenFOAM packages on the Rocky Linux (EL8/EL9) environment are the following. The same packages are also available on Debian, Ubuntu, and openSUSE.

$ dnf search openfoam Last metadata expiration check: 0:01:01 ago on Wed Jul 5 03:41:56 2023. ================================================ Name Exactly Matched: openfoam ================================================ openfoam.noarch : Free, Open Source, Computational Fluid Dynamics Package =============================================== Name & Summary Matched: openfoam =============================================== openfoam-default.x86_64 : OpenFOAM default installation bundle openfoam-devel.noarch : OpenFOAM source code headers and wmake build chain openfoam-selector.noarch : Tool to manage which OpenFOAM version to use openfoam-selector.src : Tool to manage which OpenFOAM version to use openfoam1912-common.noarch : OpenFOAM common files openfoam1912-default.x86_64 : OpenFOAM default installation bundle openfoam1912-devel.noarch : OpenFOAM source code headers and wmake build chain openfoam1912-doc.noarch : OpenFOAM documentation openfoam1912-tools.x86_64 : OpenFOAM-specific build tools openfoam1912-tutorials.noarch : OpenFOAM tutorials openfoam2006-common.noarch : OpenFOAM common files openfoam2006-default.x86_64 : OpenFOAM default installation bundle openfoam2006-devel.noarch : OpenFOAM source code headers and wmake build chain openfoam2006-doc.noarch : OpenFOAM documentation openfoam2006-tools.x86_64 : OpenFOAM-specific build tools openfoam2006-tutorials.noarch : OpenFOAM tutorials openfoam2012-common.noarch : OpenFOAM common files openfoam2012-default.x86_64 : OpenFOAM default installation bundle openfoam2012-devel.noarch : OpenFOAM source code headers and wmake build chain openfoam2012-doc.noarch : OpenFOAM documentation openfoam2012-tools.x86_64 : OpenFOAM-specific build tools openfoam2012-tutorials.noarch : OpenFOAM tutorials openfoam2106-common.noarch : OpenFOAM common files openfoam2106-default.x86_64 : OpenFOAM default installation bundle openfoam2106-devel.noarch : OpenFOAM source code headers and wmake build chain openfoam2106-doc.noarch : OpenFOAM documentation openfoam2106-tools.x86_64 : OpenFOAM-specific build tools openfoam2106-tutorials.noarch : OpenFOAM tutorials openfoam2112-common.noarch : OpenFOAM common files openfoam2112-default.x86_64 : OpenFOAM default installation bundle openfoam2112-devel.noarch : OpenFOAM source code headers and wmake build chain openfoam2112-doc.noarch : OpenFOAM documentation openfoam2112-tools.x86_64 : OpenFOAM-specific build tools openfoam2112-tutorials.noarch : OpenFOAM tutorials openfoam2206-common.noarch : OpenFOAM common files openfoam2206-default.x86_64 : OpenFOAM default installation bundle openfoam2206-devel.noarch : OpenFOAM source code headers and wmake build chain openfoam2206-doc.noarch : OpenFOAM documentation openfoam2206-tools.x86_64 : OpenFOAM-specific build tools openfoam2206-tutorials.noarch : OpenFOAM tutorials openfoam2212-common.noarch : OpenFOAM common files openfoam2212-default.x86_64 : OpenFOAM default installation bundle openfoam2212-devel.noarch : OpenFOAM source code headers and wmake build chain openfoam2212-doc.noarch : OpenFOAM documentation openfoam2212-tools.x86_64 : OpenFOAM-specific build tools openfoam2212-tutorials.noarch : OpenFOAM tutorials openfoam2306-common.noarch : OpenFOAM common files openfoam2306-default.x86_64 : OpenFOAM default installation bundle openfoam2306-devel.noarch : OpenFOAM source code headers and wmake build chain openfoam2306-doc.noarch : OpenFOAM documentation openfoam2306-tools.x86_64 : OpenFOAM-specific build tools openfoam2306-tutorials.noarch : OpenFOAM tutorials ==================================================== Name Matched: openfoam ==================================================== openfoam1912.src : Free, Open Source, Computational Fluid Dynamics Package openfoam1912.x86_64 : Free, Open Source, Computational Fluid Dynamics Package openfoam2006.src : Free, Open Source, Computational Fluid Dynamics Package openfoam2006.x86_64 : Free, Open Source, Computational Fluid Dynamics Package openfoam2012.src : Free, Open Source, Computational Fluid Dynamics Package openfoam2012.x86_64 : Free, Open Source, Computational Fluid Dynamics Package openfoam2106.src : Free, Open Source, Computational Fluid Dynamics Package openfoam2106.x86_64 : Free, Open Source, Computational Fluid Dynamics Package openfoam2112.src : Free, Open Source, Computational Fluid Dynamics Package openfoam2112.x86_64 : Free, Open Source, Computational Fluid Dynamics Package openfoam2206.src : Free, Open Source, Computational Fluid Dynamics Package openfoam2206.x86_64 : Free, Open Source, Computational Fluid Dynamics Package openfoam2212.src : Free, Open Source, Computational Fluid Dynamics Package openfoam2212.x86_64 : Free, Open Source, Computational Fluid Dynamics Package openfoam2306.x86_64 : Free, Open Source, Computational Fluid Dynamics Package openfoam2306.src : Free, Open Source, Computational Fluid Dynamics Package