Is everything ok with quaternions?

We have made a very little progress making rotation work in the difficulty=4 task after trying several things. Our initial believe was that it might be hard for a NN to read in quaternions, but after tryin various other representations we are still unable to move the needle (even on a simplified task where you need to rotate the object that lies on the ground).

We just want to double check if everything is ok with the quaternions that we are getting from the environment? Or if there is anyone who has managed to successfully perform rotations, so we have a confirmation that the problem is on our side.