Step() function behavior on the real robot

Dear Organizers,

I have a question about the implementation of the step() function. I am not 100% sure whether I am correct, but when I run experiments on the real platform and log how long the execution of the step() function takes, I get a strange behavior. E.g. if a frameskip of 10 is specified, the step() function takes 10ms. This is problematic as this basically means that our own code will never run at the specified frequency (at least if interpreting that setting frameskip defines the frequency).

I think the reason for this behavior is that in the step() function one is not only appending the action into the queue, but right after this action, directly the observation is retrieved, which is a blocking action and waits until the appended action is executed.

I am wondering whether you also experience this behavior and whether this is intended.

I think to circumvent this issue, one idea would be to only once retrieve the observation after having appended all of the actions. But then retrieve the observation with respect to the current time index (e.g. t = self.platform.get_current_timeindex()). This way, the step function would run as quickly as possible,…

Thanks and best,
Niklas

You are right, the implementation in the example environment will block due to waiting for the observation after each action. I think the reason we did it like this is that we need all the intermediate observations if we want to compute the reward in the same way as it is done during the evaluation (i.e. accumulated over all steps). But I see that this doesn’t make much sense if the purpose of the frameskip is to have more computation time between the steps. And you may anyway not need the reward like this at runtime.

I think your suggestion makes sense. I would just change one thing: Instead of using get_current_timeindex() it might be better to use the t of the first action appended in this step. This way, step() would wait until all actions of the previous step are executed, resulting in a constant step rate.