podman mem_limit failing start of the container #16921

https://github.com/containers/podman/issues/16921

rsarpal opened on Dec 22, 2022 Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

Using mem_limit: 15g in docker-compose file for inlfuxdb container.

Describe the results you received:

podman start -a influxdb_rs ERRO[0000] failed to move the rootless netns slirp4netns process to the systemd user.slice: exec: “dbus-launch”: executable file not found in $PATH Error: OCI runtime error: unable to start container fdeb916d15deb277216432f3dad244e3fc5a6416c73d9dfaf68035e9e0f75bd0: runc: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: cannot set memory limit: container could not join or create cgroup exit code: 125

Describe the results you expected:

I expected the podman container to start successfully

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client: Podman Engine Version: 4.2.0 API Version: 4.2.0 Go Version: go1.18.7 Built: Wed Oct 26 22:23:47 2022 OS/Arch: linux/amd64 Output of podman info:

host: arch: amd64 buildahVersion: 1.27.1 cgroupControllers: [] cgroupManager: cgroupfs cgroupVersion: v1 conmon: package: conmon-2.1.4-1.module+el8.7.0+17064+3b31f55c.x86_64 path: /usr/bin/conmon version: ‘conmon version 2.1.4, commit: 64e1fe3ac604668d46b6efda338a9ba5a9b91b98’ cpuUtilisation: idlePercent: 99.68 systemPercent: 0.06 userPercent: 0.27 cpus: 128 distribution: distribution: ‘“rhel”’ version: “8.5” eventLogger: file hostname: abc.r.fi idMappings: gidmap: - container_id: 0 host_id: 1001 size: 1 - container_id: 1 host_id: 165536 size: 65536 uidmap: - container_id: 0 host_id: 1001 size: 1 - container_id: 1 host_id: 165536 size: 65536 kernel: 4.18.0-372.26.1.el8_6.x86_64 linkmode: dynamic logDriver: k8s-file memFree: 1015374880768 memTotal: 1077257314304 networkBackend: cni ociRuntime: name: runc package: runc-1.1.4-1.module+el8.7.0+17064+3b31f55c.x86_64 path: /usr/bin/runc version: |- runc version 1.1.4 spec: 1.0.2-dev go: go1.18.7 libseccomp: 2.5.1 os: linux remoteSocket: path: /tmp/podman-run-1001/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: true serviceIsRemote: false slirp4netns: executable: /bin/slirp4netns package: slirp4netns-1.2.0-2.module+el8.7.0+17064+3b31f55c.x86_64 version: |- slirp4netns version 1.2.0 commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383 libslirp: 4.4.0 SLIRP_CONFIG_VERSION_MAX: 3 libseccomp: 2.5.1 swapFree: 17179865088 swapTotal: 17179865088 uptime: 1514h 41m 22.00s (Approximately 63.08 days) plugins: authorisation: null log:

  • k8s-file
  • none
  • passthrough
  • journald network:
  • bridge
  • macvlan
  • ipvlan volume:
  • local registries: search:
  • registry.access.redhat.com
  • registry.redhat.io
  • docker.io store: configFile: /home/r/.config/containers/storage.conf containerStore: number: 1 paused: 0 running: 0 stopped: 1 graphDriverName: overlay graphOptions: {} graphRoot: /home/r/.local/share/containers/storage graphRootAllocated: 1008015245312 graphRootUsed: 29270917120 graphStatus: Backing Filesystem: xfs Native Overlay Diff: “true” Supports d_type: “true” Using metacopy: “false” imageCopyTmpDir: /var/tmp imageStore: number: 5 runRoot: /tmp/podman-run-1001/containers volumePath: /home/r/.local/share/containers/storage/volumes version: APIVersion: 4.2.0 Built: 1666812227 BuiltTime: Wed Oct 26 22:23:47 2022 GitCommit: “” GoVersion: go1.18.7 Os: linux OsArch: linux/amd64 Version: 4.2.0 Package info (e.g. output of rpm -q podman or apt list podman or brew info podman):

Installed Packages Name : podman Epoch : 3 Version : 4.2.0 Release : 4.module+el8.7.0+17064+3b31f55c Architecture : x86_64 Sise : 41 M Source : podman-4.2.0-4.module+el8.7.0+17064+3b31f55c.src.rpm Repository : @System From repo : rhel-8-for-x86_64-appstream-rpms Summary : Manage Pods, Containers and Container Images URL : https://podman.io/ licence : ASL 2.0 and GPLv3+

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Activity

openshift-ci added kind/bug Categorizes issue or PR as related to a bug. on Dec 22, 2022 rhatdan rhatdan commented on Dec 22, 2022 rhatdan on Dec 22, 2022 Member I believe this is a cgroup v1 issue, most likely this will only work with V2 in rootless mode.

rhatdan rhatdan commented on Dec 22, 2022 rhatdan on Dec 22, 2022 Member @giuseppe Do you agree?

giuseppe giuseppe commented on Dec 23, 2022 giuseppe on Dec 23, 2022 Member Is dbus-launch installed? Anyway I think it will just bring you one step forward and it will fail to apply the limit as rootless. You need to switch to cgroup v2 or use root if you want to specify a memory limit for your container

rsarpal rsarpal commented on Dec 23, 2022 rsarpal on Dec 23, 2022 Author Ok, we have now installed dbus-launch and have also moved to cgroupsv2. Now while running the container there is another Error coming up

ERRO[0000] failed to move the rootless netns slirp4netns process to the systemd user.slice: Process org.freedesktop.systemd1 exited with status 1 Error: runc: runc create failed: unable to start container process: unable to apply cgroup configuration: rootless needs no limits + no cgrouppath when no permission is granted for cgroups: mkdir /sys/fs/cgroup/user.slice/user-1572600290.slice/2a7cad2179e7f824ccab3e25df55c9cd60e49445c740f49ab3775f800ef57241: permission denied: OCI permission denied exit code: 126

giuseppe giuseppe commented on Jan 3, 2023 giuseppe on Jan 3, 2023 Member can you show me the output for systemd-run –scope –user -p MemoryMax=1G cat /proc/self/cgroup?

rsarpal rsarpal commented on Jan 9, 2023 rsarpal on Jan 9, 2023 Author I am running this as root now as my own user got another error

[root@rocky-perch ~]# systemd-run –scope –user -p MemoryMax=1G cat /proc/self/cgroup Failed to create bus connection: No such file or directory

From my own user everything podman related returns

podman info Error: cannot re-exec process to join the existing user namespace

rsarpal rsarpal commented on Jan 9, 2023 rsarpal on Jan 9, 2023 Author Also

podman info –log-level=debug INFO[0000] podman filtering at log level debug DEBU[0000] Called info.PersistentPreRunE(podman info –log-level=debug) DEBU[0000] Merged system config “/usr/share/containers/containers.conf” DEBU[0000] Merged system config “/home/r/.config/containers/containers.conf” DEBU[0000] Using conmon: “/usr/bin/conmon” DEBU[0000] Initialising boltdb state at /home/r/.local/share/containers/storage/libpod/bolt_state.db DEBU[0000] Overriding run root “/run/user/1572600290/containers” with “/tmp/podman-run-1572600290/containers” from database DEBU[0000] Overriding tmp dir “/run/user/1572600290/libpod/tmp” with “/tmp/podman-run-1572600290/libpod/tmp” from database DEBU[0000] systemd-logind: Unknown object ‘/’. DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /home/r/.local/share/containers/storage DEBU[0000] Using run root /tmp/podman-run-1572600290/containers DEBU[0000] Using static dir /home/r/.local/share/containers/storage/libpod DEBU[0000] Using tmp dir /tmp/podman-run-1572600290/libpod/tmp DEBU[0000] Using volume path /home/r/.local/share/containers/storage/volumes DEBU[0000] Set libpod namespace to “” DEBU[0000] Not configuring container store DEBU[0000] Initialising event backend file DEBU[0000] Configured OCI runtime runsc initialisation failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Configured OCI runtime krun initialisation failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime runj initialisation failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialisation failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Using OCI runtime “/usr/bin/runc” Error: cannot re-exec process to join the existing user namespace

rsarpal rsarpal commented on Jan 9, 2023 rsarpal on Jan 9, 2023 Author ok i removed the pause.pid file from /tmp/podman-run-1572600290/libpod/tmp and now that Error: cannot re-exec process to join the existing user namespace is gone

Now we also have cgroupsv2 enabled

cgroupManager: cgroupfs cgroupVersion: v2 conmon: package: conmon-2.1.4-1.module+el8.7.0+17064+3b31f55c.x86_64 path: /usr/bin/conmon version: ‘conmon version 2.1.4, commit: 64e1fe3ac604668d46b6efda338a9ba5a

but I continue to get the errors

ERRO[0000] failed to move the rootless netns slirp4netns process to the systemd user.slice: Process org.freedesktop.systemd1 exited with status 1 Failed to create bus connection: Connection refused Error: runc: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/user.slice/user-1572600290.slice/e5c2ab07e80e28e77218f0cfb6e9badc1d498c938e6ffea38fc086745fad640f/memory.swap.max: no such file or directory: OCI runtime attempted to invoke a command that was not found exit code: 127 podman start rel_influxdb_rs ERRO[0000] failed to move the rootless netns slirp4netns process to the systemd user.slice: Process org.freedesktop.systemd1 exited with status 1 Failed to create bus connection: Connection refused Error: unable to start container “e5c2ab07e80e28e77218f0cfb6e9badc1d498c938e6ffea38fc086745fad640f”: runc: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: openat2 /sys/fs/cgroup/user.slice/user-1572600290.slice/e5c2ab07e80e28e77218f0cfb6e9badc1d498c938e6ffea38fc086745fad640f/memory.swap.max: no such file or directory: OCI runtime attempted to invoke a command that was not found exit code: 125

rsarpal rsarpal commented on Jan 9, 2023 rsarpal on Jan 9, 2023 Author I ran chmod and this has fixed the error about memory.swap.max: no such file or directoy

sudo chmod o+w /sys/fs/cgroup/user.slice/user-1572600290.slice/memory.swap.max

and with this now the container is running after using chmod , however it still throws one error:

ERRO[0000] failed to move the rootless netns slirp4netns process to the systemd user.slice: Process org.freedesktop.systemd1 exited with status 1 Failed to create bus connection: Connection refused

giuseppe giuseppe commented on Jan 10, 2023 giuseppe on Jan 10, 2023 Member if you cannot run systemd-run –scope –user then something else seems to be broken on your system; it doesn’t look like a podman issue.

Also please run the command as your unprivileged user, not root.

I am closing the issue since I don’t see anything wrong from Podman that must be addressed, but feel free to comment further

Updated: