Singularity v3.3.0-rc.3 Checklist #3963

https://github.com/apptainer/singularity/issues/3963

jscook2345 opened on Jul 22, 2019 · edited by jscook2345 Contributor This is the QA Checklist for Singularity v3.3.0-rc.3 Download: https://github.com/sylabs/singularity/releases/tag/v3.3.0-rc.3

Each comment should include:

A description of the test done Some example output A link to an issue, if one was found Each issue should include:

The Release 3.3.0 label Changes since v3.3.0-rc.1

Oras context fix #3791

Update help docs to include oras transport #3749

Update the help for remote commands that use a default value for the remote name #3796

Fix user namespace mappings and build section execution context #3759

Fix parent process signal propagation #3696

Allow execution from a parent (rootless) user namespace #3780

Fix bad pull library URI variable introduces during CLI refactoring #3825

Revisit the implementation of image caches #3758

Avoid having the cache related info message displayed all the time #3782

Fix user namespace bind source located on a read-only filesystem #3803

Fixed panic when export-ing a key when keyring is empty #3889

Build a sif from a sif without root #3897

Add a fakeroot engine for build #3926

Use newuidmap/newgidmap as fakeroot fallback #3943 Changed behaviours / defaults

singularity remote login will now use the default remote if one is not supplied

singularity remote status will now use the default remote if one is not supplied

singularity pull with the shub transport now caches the image

singularity cache clean will now delete only sub-directories of SINGULARITY_CACHEDIR instead of the directory itself New feature: oras

singularity pull understands a new transport, oras, which works with the Azure Container Registry and docker/distribution

singularity push understands a new transport, oras, which works with the Azure Container Registry and docker/distribution New feature: fakeroot

singularity build understands a new option, –fakeroot

singularity exec understands a new option, –fakeroot

singularity exec –fakeroot requires –net to access the network

singularity instance start understands a new option, –fakeroot

singularity instance start –fakeroot requires –net to access the network

singularity run understands a new option, –fakeroot

singularity run –fakeroot requires –net to access the network

singularity shell understands a new option, –fakeroot

singularity shell –fakeroot requires –net to access the network

singularity test understands a new option, –fakeroot

singularity test –fakeroot requires –net to access the network New command: sif

singularity sif is a new command that allows you to work with SIF files.

singularity sif add Add a data object to a SIF file

singularity sif del Delete a specified object descriptor and data from SIF file

singularity sif dump Extract and output data objects from SIF files

singularity sif header Display SIF global headers

singularity sif info Display detailed information of object descriptors

singularity sif list List object descriptors from SIF files

singularity sif new Create a new empty SIF image file

singularity sif setprim Set primary system partition Activity

jscook2345 added Testing Anything surrounding testing of the repository

Release 3.3.0 on Jul 22, 2019

jscook2345 assigned jchavezlavalle , sushma-4 and jscook2345 on Jul 22, 2019 jscook2345 jscook2345 commented on Jul 25, 2019 jscook2345 on Jul 25, 2019 Contributor Author centos 7.6.1810 version @jscook2345

From the release tarball

$ cat /etc/centos-release CentOS Linux release 7.6.1810 (Core)

$ singularity version 3.3.0-rc.3

$ singularity –version singularity version 3.3.0-rc.3 jscook2345 jscook2345 commented on Jul 25, 2019 jscook2345 on Jul 25, 2019 Contributor Author centos 7.6.1810 build help @jscook2345

$ singularity help build … -f, –fakeroot build using user namespace to fake root user (requires a privileged installation … jscook2345 jscook2345 commented on Jul 25, 2019 jscook2345 on Jul 25, 2019 · edited by jscook2345 Contributor Author centos 7.6.1810 build –fakeroot from definition with docker bootstrap @jscook2345

$ singularity build bbFromDocker.sif docker.def FATAL: You must be the root user, however you can use –remote or –fakeroot to build from a Singularity recipe file

$ singularity build –fakeroot bbFromDocker.sif docker.def FATAL: could not use fakeroot: user vagrant not found in /etc/subuid

$ sudo vim /etc/subuid $ cat /etc/subuid vagrant:100000:65536

$ singularity build –fakeroot bbFromDocker.sif docker.def FATAL: could not use fakeroot: user vagrant not found in /etc/subgid

$ sudo vim /etc/subgid $ cat /etc/subgid vagrant:100000:65536

[vagrant@localhost tests]$ singularity build –fakeroot bbFromDocker.sif docker.def INFO: Starting build… ….. INFO: Running post scriptlet

id uid=0(root) gid=0(root) groups=0(root) INFO: Creating SIF file… INFO: Build complete: bbFromDocker.sif jscook2345 jscook2345 commented on Jul 25, 2019 jscook2345 on Jul 25, 2019 Contributor Author centos 7.6.1810 build –fakeroot from definition with library bootstrap @jscook2345

$ singularity build bbFromLibrary.sif library.def FATAL: You must be the root user, however you can use –remote or –fakeroot to build from a Singularity recipe file

$ singularity build –fakeroot bbFromLibrary.sif library.def INFO: Starting build… INFO: Running post scriptlet

id uid=0(root) gid=0(root) groups=0(root) INFO: Creating SIF file… INFO: Build complete: bbFromLibrary.sif jscook2345 jscook2345 commented on Jul 25, 2019 jscook2345 on Jul 25, 2019 Contributor Author centos 7.6.1810 build –fakeroot from definition with shub bootstrap @jscook2345

$ singularity build bbFromShub.sif shub.def FATAL: You must be the root user, however you can use –remote or –fakeroot to build from a Singularity recipe file

$ singularity build –fakeroot bbFromShub.sif shub.def INFO: Starting build… 680.03 KiB / 680.03 KiB [===========================================================================================] 100.00% 1.30 MiB/s 0s INFO: Running post scriptlet

id uid=0(root) gid=0(root) groups=0(root) INFO: Creating SIF file… INFO: Build complete: bbFromShub.sif jscook2345 jscook2345 commented on Jul 25, 2019 jscook2345 on Jul 25, 2019 Contributor Author centos 7.6.1810 build –fakeroot –sandbox from definition with scratch bootstrap @jscook2345

$ singularity build –sandbox fromScratch scratch.def FATAL: You must be the root user, however you can use –remote or –fakeroot to build from a Singularity recipe file

$ singularity build –fakeroot –sandbox fromScratch scratch.def INFO: Starting build… INFO: Creating sandbox directory… INFO: Build complete: fromScratch

$ ls fromScratch/ dev environment etc home proc root singularity sys tmp var jscook2345 jscook2345 commented on Jul 25, 2019 jscook2345 on Jul 25, 2019 Contributor Author centos 7.6.1810 build –fakeroot from definition with localimage bootstrap @jscook2345

$ singularity build bbFromLocalImage.sif localimage.def FATAL: You must be the root user, however you can use –remote or –fakeroot to build from a Singularity recipe file

$ singularity build –fakeroot bbFromLocalImage.sif localimage.def INFO: Starting build… INFO: Running post scriptlet

id uid=0(root) gid=0(root) groups=0(root) INFO: Creating SIF file… INFO: Build complete: bbFromLocalImage.sif jscook2345 jscook2345 commented on Jul 25, 2019 jscook2345 on Jul 25, 2019 · edited by jscook2345 Contributor Author centos 7.6.1810 build –fakeroot from definition with yum bootstrap @jscook2345

https://github.com/sylabs/singularity/issues/3990

This feature works as expected if you include seccomp in your build of singularity.

$ singularity build –fakeroot yum.sif yum.def INFO: Starting build… INFO: Skipping GPG Key Import INFO: Running post scriptlet

id uid=0(root) gid=0(root) groups=0(root) INFO: Creating SIF file… INFO: Build complete: yum.sif jscook2345 jscook2345 commented on Jul 25, 2019 jscook2345 on Jul 25, 2019 · edited by jscook2345 Contributor Author centos 7.6.1810 build –fakeroot from definition with debootstrap bootstrap @jscook2345

#3990

This feature does not work with debootstrap. See: https://sylabs.io/guides/3.3/user-guide/fakeroot.html#build

jscook2345 jscook2345 commented on Jul 25, 2019 jscook2345 on Jul 25, 2019 · edited by jscook2345 Contributor Author centos 7.6.1810 build –fakeroot –sandbox with definition from docker bootstrap @jscook2345

$ singularity build –sandbox bbsbFromDocker docker.def FATAL: You must be the root user, however you can use –remote or –fakeroot to build from a Singularity recipe file

$ singularity build –fakeroot –sandbox bbsbFromDocker docker.def INFO: Starting build… Getting image source signatures Skipping fetch of repeat blob sha256:ee153a04d6837058642958836062f20badf39f558be3e6c7c7773ef7d8301d90 Copying config sha256:d068e40b116fa97769e99457effd4f2dcd5d07556b5ae6829e71e0508a79ddf7 575 B / 575 B [============================================================] 0s Writing manifest to image destination Storing signatures INFO: Running post scriptlet

id uid=0(root) gid=0(root) groups=0(root) INFO: Creating sandbox directory… INFO: Build complete: bbsbFromDocker 76 remaining items jscook2345 jscook2345 commented on Jul 29, 2019 jscook2345 on Jul 29, 2019 Contributor Author centos 7.6 v1905.1 instance start –fakeroot shub:// @jscook2345

$ singularity instance start shub://busybox:latest bb INFO: instance started successfully

$ singularity exec instance://bb id uid=1000(vagrant) gid=1000(vagrant) groups=1000(vagrant)

$ singularity exec –fakeroot instance://bb id uid=1000(vagrant) gid=1000(vagrant) groups=1000(vagrant)

$ singularity instance stop bb Stopping bb instance of /home/vagrant/.singularity/cache/shub/hash/busybox_latest.sif (PID=21394)

$ singularity instance start –fakeroot shub://busybox:latest bb INFO: Convert SIF file to sandbox… INFO: instance started successfully

$ singularity exec instance://bb id uid=0(root) gid=0(root) groups=0(root)

$ singularity exec –fakeroot instance://bb id uid=0(root) gid=0(root) groups=0(root)

$ singularity instance stop bb Stopping bb instance of /tmp/rootfs-794045910 (PID=21466) jscook2345 jscook2345 commented on Jul 29, 2019 jscook2345 on Jul 29, 2019 Contributor Author centos 7.6 v1905.1 instance start –fakeroot @jscook2345

$ singularity instance start busybox_latest.sif bb INFO: instance started successfully

$ singularity instance list INSTANCE NAME PID IMAGE bb 21546 /home/vagrant/examples/instance_start/sif/busybox_latest.sif

$ singularity exec instance://bb id uid=1000(vagrant) gid=1000(vagrant) groups=1000(vagrant)

$ singularity exec –fakeroot instance://bb id uid=1000(vagrant) gid=1000(vagrant) groups=1000(vagrant)

$ singularity instance stop bb Stopping bb instance of /home/vagrant/examples/instance_start/sif/busybox_latest.sif (PID=21546)

$ singularity instance start –fakeroot busybox_latest.sif bb INFO: Convert SIF file to sandbox… INFO: instance started successfully

$ singularity instance list INSTANCE NAME PID IMAGE bb 21622 /tmp/rootfs-063318723

$ singularity exec instance://bb id uid=0(root) gid=0(root) groups=0(root)

$ singularity exec –fakeroot instance://bb id uid=0(root) gid=0(root) groups=0(root)

$ singularity instance stop bb Stopping bb instance of /tmp/rootfs-063318723 (PID=21622) jscook2345 jscook2345 commented on Jul 29, 2019 jscook2345 on Jul 29, 2019 Contributor Author centos 7.6 v1905.1 instance start –fakeroot @jscook2345

$ singularity instance start busybox_latest bb INFO: instance started successfully

$ singularity instance list INSTANCE NAME PID IMAGE bb 21728 /home/vagrant/examples/instance_start/sandbox/busybox_latest

$ singularity exec instance://bb id uid=1000(vagrant) gid=1000(vagrant) groups=1000(vagrant)

$ singularity exec –fakeroot instance://bb id uid=1000(vagrant) gid=1000(vagrant) groups=1000(vagrant)

$ singularity instance stop bb Stopping bb instance of /home/vagrant/examples/instance_start/sandbox/busybox_latest (PID=21728)

]$ singularity instance start –fakeroot busybox_latest bb INFO: instance started successfully

$ singularity instance list INSTANCE NAME PID IMAGE bb 21795 /home/vagrant/examples/instance_start/sandbox/busybox_latest

$ singularity exec instance://bb id uid=0(root) gid=0(root) groups=0(root)

$ singularity exec –fakeroot instance://bb id uid=0(root) gid=0(root) groups=0(root)

$ singularity instance stop bb Stopping bb instance of /home/vagrant/examples/instance_start/sandbox/busybox_latest (PID=21795) jscook2345 jscook2345 commented on Jul 29, 2019 jscook2345 on Jul 29, 2019 Contributor Author centos 7.6 v1905.1 instance start –fakeroot oras:// @jscook2345

$ singularity instance start oras://localhost:5000/busybox:latest bb INFO: instance started successfully

$ singularity exec instance://bb id uid=1000(vagrant) gid=1000(vagrant) groups=1000(vagrant)

$ singularity exec –fakeroot instance://bb id uid=1000(vagrant) gid=1000(vagrant) groups=1000(vagrant)

$ singularity instance stop bb Stopping bb instance of /home/vagrant/.singularity/cache/oras/sha256:6a164fa666c18dd554ed7a50b83824d42071049c4a3102abea415b963b74766a/busybox_latest.sif (PID=21873)

$ singularity instance start –fakeroot oras://localhost:5000/busybox:latest bb INFO: Convert SIF file to sandbox… INFO: instance started successfully

$ singularity exec instance://bb id uid=0(root) gid=0(root) groups=0(root)

$ singularity exec –fakeroot instance://bb id uid=0(root) gid=0(root) groups=0(root)

$ singularity instance stop bb Stopping bb instance of /tmp/rootfs-327052318 (PID=21947) jscook2345 jscook2345 commented on Jul 29, 2019 jscook2345 on Jul 29, 2019 · edited by jscook2345 Contributor Author centos 7.6 v1905.1 instance start –fakeroot instance:// @jscook2345

$ singularity instance start busybox_latest.sif bb1 INFO: instance started successfully

$ singularity instance start instance://bb1 bb2 FATAL: Starting an instance from another is not allowed

$ singularity instance start –fakeroot instance://bb1 bb2 FATAL: Starting an instance from another is not allowed

$ singularity instance stop bb1 Stopping bb1 instance of /home/vagrant/examples/instance_start/instance/busybox_latest.sif (PID=22021) jscook2345 jscook2345 commented on Jul 29, 2019 jscook2345 on Jul 29, 2019 Contributor Author centos 7.6 v1905.1 instance start –fakeroot cannot modify files it does not have permission to on host system @jscook2345

$ singularity instance start –bind /etc:/etc –fakeroot busybox_latest.sif bb INFO: Convert SIF file to sandbox… INFO: instance started successfully

$ singularity instance list INSTANCE NAME PID IMAGE bb 22118 /tmp/rootfs-641164762

$ singularity exec instance://bb touch /etc/passwd touch: /etc/passwd: Permission denied

$ singularity exec –fakeroot instance://bb touch /etc/passwd touch: /etc/passwd: Permission denied

$ singularity instance stop bb Stopping bb instance of /tmp/rootfs-641164762 (PID=22118) jscook2345 jscook2345 commented on Jul 29, 2019 jscook2345 on Jul 29, 2019 Contributor Author centos 7.v v1905.1 instance start –fakeroot cannot access the network without the –net flag @jscook2345

$ singularity instance start –fakeroot busybox_latest.sif bb INFO: Convert SIF file to sandbox… INFO: instance started successfully

$ singularity exec instance://bb ping -c1 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes ping: permission denied (are you root?)

$ singularity exec –fakeroot instance://bb ping -c1 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes ping: permission denied (are you root?)

$ singularity exec –fakeroot –net instance://bb ping -c1 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes ping: permission denied (are you root?)

$ singularity instance stop bb Stopping bb instance of /tmp/rootfs-413798563 (PID=22191) $ singularity instance start –fakeroot –net busybox_latest.sif bb INFO: Convert SIF file to sandbox… INFO: instance started successfully

$ singularity instance list bb INSTANCE NAME PID IMAGE bb 22272 /tmp/rootfs-712555440

$ singularity exec instance://bb ping -c1 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: seq=0 ttl=61 time=36.059 ms

— 8.8.8.8 ping statistics — 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max = 36.059/36.059/36.059 ms

$ singularity exec –fakeroot instance://bb ping -c1 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: seq=0 ttl=61 time=33.934 ms

— 8.8.8.8 ping statistics — 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max = 33.934/33.934/33.934 ms

$ singularity exec –fakeroot –net instance://bb ping -c1 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: seq=0 ttl=61 time=36.697 ms

— 8.8.8.8 ping statistics — 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max = 36.697/36.697/36.697 ms

$ singularity instance stop bb Stopping bb instance of /tmp/rootfs-712555440 (PID=22272) jscook2345 jscook2345 commented on Jul 29, 2019 jscook2345 on Jul 29, 2019 Contributor Author centos 7.6 v1905.1 instance start –fakeroot cannot bind to ports below 1024 without the –net flag @jscook2345

$ singularity instance start –fakeroot python_latest.sif py INFO: Convert SIF file to sandbox… INFO: instance started successfully

$ singularity exec instance://py python -m http.server 80 Traceback (most recent call last): ….. PermissionError: [Errno 13] Permission denied

$ singularity exec instance://py python -m http.server 8080 Serving HTTP on 0.0.0.0 port 8080 (http://0.0.0.0:8080/) … ^C

$ singularity exec –fakeroot instance://py python -m http.server 80 Traceback (most recent call last): ….. PermissionError: [Errno 13] Permission denied

$ singularity exec –fakeroot instance://py python -m http.server 8080 Serving HTTP on 0.0.0.0 port 8080 (http://0.0.0.0:8080/) … ^C

$ singularity instance stop py Stopping py instance of /tmp/rootfs-713549518 (PID=22442) $ singularity instance start –fakeroot –net python_latest.sif py INFO: Convert SIF file to sandbox… INFO: instance started successfully

$ singularity exec instance://py python -m http.server 80 Serving HTTP on 0.0.0.0 port 80 (http://0.0.0.0:80/) … ^C

$ singularity exec –fakeroot instance://py python -m http.server 80 Serving HTTP on 0.0.0.0 port 80 (http://0.0.0.0:80/) … ^C

$ singularity exec –fakeroot –net instance://py python -m http.server 80 Serving HTTP on 0.0.0.0 port 80 (http://0.0.0.0:80/) … ^C

$ singularity instance stop py Stopping py instance of /tmp/rootfs-909064595 (PID=22567) jscook2345 jscook2345 commented on Jul 29, 2019 jscook2345 on Jul 29, 2019 · edited by jscook2345 Contributor Author centos 7.6 v1905.1 verify #3926 patched @jscook2345

https://github.com/sylabs/singularity/issues/4033

Fixes #3892

[vagrant@localhost build]$ singularity build –fakeroot –sandbox sb/ 1804.def INFO: Starting build… ….. INFO: Running post scriptlet ….. INFO: Creating sandbox directory… INFO: Build complete: sb/ Fixes #3927

$ singularity build –fakeroot c7.sif docker://centos FATAL: fakeroot requires to set ‘allow setuid = yes’ in /usr/local/singularity/etc/singularity/singularity.conf Fix yum bootstrap with fakeroot.

$ singularity build –fakeroot –sandbox c7/ yum.def INFO: Starting build… INFO: Skipping GPG Key Import INFO: Creating sandbox directory… INFO: Build complete: c7/

$ singularity build –fakeroot –sandbox c7.sif yum.def INFO: Starting build… INFO: Skipping GPG Key Import INFO: Creating sandbox directory… INFO: Build complete: c7.sif jscook2345 jscook2345 commented on Jul 29, 2019 jscook2345 on Jul 29, 2019 · edited by jscook2345 Contributor Author centos 7.6 v1905.1 verify #3943 patched @jscook2345

Use newuidmap/newgidmap as fakeroot fallback for unprivileged installation Fakeroot configuration is now handled by /etc/subxid files $ singularity exec centos_latest.sif id ERROR : Failed to set effective UID to 0

$ singularity exec –fakeroot centos_latest.sif id INFO: Convert SIF file to sandbox… FATAL: newuidmap was not found in PATH (/usr/bin:/usr/sbin:/bin:/sbin:/usr/local/bin:/usr/local/sbin), required with fakeroot and unprivileged installation

$ singularity exec –fakeroot centos_latest.sif id INFO: Convert SIF file to sandbox… FATAL: could not use fakeroot: user vagrant not found in /etc/subuid

$ singularity exec –fakeroot centos_latest.sif id INFO: Convert SIF file to sandbox… FATAL: could not use fakeroot: user vagrant not found in /etc/subgid

$ singularity exec –fakeroot centos_latest.sif id INFO: Convert SIF file to sandbox… uid=0(root) gid=0(root) groups=0(root) INFO: Cleaning up image… Introduce new SINGULARITY_SUID_INSTALL constant $ grep ‘SUID_INSTALL’ internal/pkg/buildcfg/config.go const SINGULARITY_SUID_INSTALL = 0 Fakeroot: fix home directory (previously used user home directory) Fakeroot: HOME environment variable while joining an instance $ singularity instance start –fakeroot centos_latest.sif c7 INFO: Convert SIF file to sandbox… INFO: instance started successfully

$ singularity instance list INSTANCE NAME PID IMAGE c7 26033 /tmp/rootfs-256284538

[vagrant@localhost sif]$ singularity exec instance://c7 printenv HOME /root

$ singularity exec instance://c7 touch /root/c7

$ sudo ls /root/c7 ls: cannot access /root/c7: No such file or directory

$ ls $HOME/c7 /home/vagrant/c7

$ singularity instance stop c7 Stopping c7 instance of /tmp/rootfs-256284538 (PID=26033) GodloveD GodloveD commented on Jul 30, 2019 GodloveD on Jul 30, 2019 Collaborator Ubuntu 18.04.2 LTS singularity test understands a new option, –fakeroot @jscook2345

Screen Shot 2019-07-29 at 4 20 25 PM

Just had to screenshot for the colours. lol

GodloveD GodloveD commented on Jul 30, 2019 GodloveD on Jul 30, 2019 · edited by GodloveD Collaborator Ubuntu 18.04.2 LTS singularity test –fakeroot requires –net to access the network @jscook2345

$ singularity test –fakeroot test.sif INFO: Convert SIF file to sandbox… ping: socket: Operation not permitted INFO: Cleaning up image… ERROR: failed to delete container image /tmp/rootfs-199334232: permission denied

$ singularity test –fakeroot –net test.sif INFO: Convert SIF file to sandbox… PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=61 time=35.8 ms

— 8.8.8.8 ping statistics — 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 35.850/35.850/35.850/0.000 ms INFO: Cleaning up image… ERROR: failed to delete container image /tmp/rootfs-721709052: permission denied Although this obviously identifies another issue with failing to delete the root file system in /tmp

Issue refed by #4049

jscook2345 jscook2345 commented on Jul 30, 2019 jscook2345 on Jul 30, 2019 Contributor Author centos 7.6 v1905.1 verify #3696 patched @jscook2345

$ singularity –version singularity version 3.3.0-rc.4

$ singularity exec ./centos_latest.sif sleep 1000 & [1] 17598

$ jobs [1]+ Running singularity exec ./centos_latest.sif sleep 1000 &

$ fg 1 singularity exec ./centos_latest.sif sleep 1000 ^C

$ ps wauxf

grep sleep

jscook2345 closed this as completedon Aug 8, 2019

dtrudg unassigned sushma-4 on May 13, 2020