Helm chart frontend pod does not start because of permission issues in raspberry pie 4 k3s

Bug
1

Hello, I’m trying to run vikunja from the helm chart there on a local Raspberry pie running k3s.

The frontend cannot start with the following error

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
/docker-entrypoint.d/20-envsubst-on-templates.sh: line 33: can't create /etc/nginx/conf.d/default.conf: Permission denied

The api started fine, so I’m guessing this is a problem with the chart config ?

I have tried the following values

frontend:
  securityContext:
    capabilities:
      drop:
      - ALL
    readOnlyRootFilesystem: false
    runAsNonRoot: true
    runAsUser: 1000

And now the error I get is

20-envsubst-on-templates.sh: ERROR: /etc/nginx/templates exists, but /etc/nginx/conf.d is not writable
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/50-injector.sh
info: API URL is [redacted]
info: Sentry enabled: false
sed: can't create temp file '/usr/share/nginx/html/index.htmlXXXXXX': Permission denied

Any idea what I should do ?

2

What’s interesting is how this is only a problem on raspberry pi. Other hosts are fine as far as I know.

That “permission denied” error sounds like a readonly file system is not a problem. Does it work if you drop all security config? The container does write any values in a volume or similar and drops all privileges once it started. In theory, it should not need any other security config.

3

I can reproduce this issue on an x86_64 k3s node. Any ideas on troubleshooting?

1 Like
4

Does it work if you run the docker image directly? (Without kubernetes)
Maybe that way we can figure out if it’s a but in the helm chart or in the image.

5

Seems like this might be a containerd vs. docker thing. When I run the example docker-compose.yml with straight docker as the runtime, its fine. When I run it via nerdctl/containerd things break. More specifically on the frontend container: setrlimit(RLIMIT_NOFILE, 65535) failed (1: Operation not permitted). Unless you jump through hoops, kubernetes uses containerd as its runtime, not docker.

6

Can you run a bare nginx container with containerd? Just to make sure it’s a customization of us which is causing the problem.

7

Yep, “sudo nerdctl run --name nerdctl-nginx -p 8080:80 -d nginx:alpine” spins up without issue.

8

Any update about this issue? just used the latest unstable frontend image in kubernetes its giving me the same error.

9

@r1cebank Are you running this on a raspberry pi?

Here’s an issue where people have the same problem: [Nginx ingress controller] file open limit · Issue #1966 · kubernetes-retired/contrib · GitHub

I did some digging and it looks like this is not caused by any of our scripts but by the custom nginx config. This fails with the mentioned error message:

sudo nerdctl run --address /var/run/docker/containerd/containerd.sock -p 8080:80 -v $PWD/docker/nginx.conf:/etc/nginx/nginx.conf nginx:alpine
10

Okay so it looks like this line is the problem: frontend/nginx.conf at main - frontend - Gitea

11

Pushed a fix in 2f18d0cbad, please try with the next unstable build (~45 min).

12

I am having this issue with x86 with k3s. Thank you for the fix, I was just about to ask if there is anything I can test on my side to help fix this issue. :laughing:

13

I am still getting the same error with latest unstable, but based on a commit in nginx-kubernetes (Run IC as non-root · nginxinc/kubernetes-ingress@6b6ca41 · GitHub) the correct security context for frontend should be:

securityContext:
  allowPrivilegeEscalation: true
  runAsUser: 101 #nginx
  capabilities:
    drop:
    - ALL
    add:
    - NET_BIND_SERVICE

@kolaente after the update on my side, I am able to successfully run frontend image without the issue above on x86 (k3s). Thanks

14

Glad it works now! Do you want to send a PR to the helm chart?

15

Done :grinning: #6 - Update security context to allow frontend pod to start - helm-chart - Gitea

16

Hello @koalente, what is the process to start a release ? it seems like the helm repo chart still has the v0.1.2 can we make a v0.1.3 release ?

17

I’ve just released 0.1.4 with the changes.