That should be docker run -e VIKUNJA_API_URL=192.168.5.6:3456/api/v1/ vikunja/frontend. Environnement variables are always set without the $ at the beginning in docker.
Definitely try docker-compose though. It will make your live much easier. This may be helpful for you: Docker Walkthrough | Vikunja