Does that mean you get the error in phpmyadmin? Does it work if you create a new user and database?
I’ve just noticed another thing: In your docker-compose config you specified the database as VIKUNJA_DATABASE=vikunja but that should be VIKUNJA_DATABASE_DATABASE=vikunja. Does it work with that?
Still, that doesn’t explain why it did not work with the config file.