Timing / runtime of the code

Hi everyone,

First of all, thanks a lot for organizing everything.
I wanted to ask whether some people also have problems concerning the runtime of the code on the real system. In our case, it seems like the execution speed of our code varies significantly, see e.g. logs 18682 and 18688. The exact same control policy just results in very different behavior unfortunately. We think that the different behavior might be due to timing issues. Is there any way to check the load / computational resources while the code is running? Do other teams also have issues in the reproducability of their code on the real system?

Thanks and best.

1 Like

Were the jobs with timing issues all running on roboch6?

I just checked and noticed that on this robot, a background service was running that we use for maintenance purposes. This service unfortunately consumes a significant amount of CPU resources which is why it is normally disabled on the robots used for the challenge. It seems that we forgot to disable it on roboch6. I’m very sorry that this caused problems.

It is disabled now, so if this was indeed the cause of the problem, it should be fixed now.

1 Like

No, unfortunately not. Some jobs have also been running on roboch4.

Is it possible that the same problem arises on different robots?

Do you see any way to find out whether our code together what is running together with your parts is overloading the computational resources of the platform?

On roboch4 there was a zombie process of some previous test run, which apparently did not terminate properly and was still running in the background. I’m not sure if this caused any problems but apart form this I did not find anything suspicious.

You could try monitoring the CPU load during your runs. From Python this can easily be done with the psutil package. You’ll need to install it in your Singularity image, though (apt-get install python3-psutil). I did a quick test, running print(psutil.cpu_percent(0.1)) in a loop and it seems to work also inside the Singularity container.
It also has functions for showing memory usage and some other things.

1 Like

Perfect. Thanks a lot for your quick help.

So unfortunately I could not yet resolve the issue. See again, jobs 20034 where again everything works smooth and 20066 where the exact same controller seems to behave entirely different.

Do you have any idea what could be going wrong? I have reassured that we run at a constant frequency without overloading the platform and also in simulation everything works fine. Also, we are doing torque control, do you think this could be an issue?

Thanks a lot in advance and sorry for being maybe a little bit unspecific. It is just that we also have a very tough time of isolating the issue, that is why I am asking because you maybe might have more experience with the platform,…

Sorry, I again had a look into the logs and it seems like being a problem on my side,…