As far as I can tell, this is currently as it is designed. A task if part of a project, if you want a member to work with this, they need permissions on the project. It’s that easy (and therefore not very flexible) at the moment.
There are thoughts to put in a more fine grained permission system, not sure if that use case would be covered and if that would be available for everyone: Vikunja as a business