There are motion sensors in the Oval Office’s floor.
One of the Secret Service’s rules is that the president shouldn’t be left alone. And if the president wants to stay alone in the Oval Office without agents, they can easily trace any movement and any potential threat with the help of sensors in the floor.