As Patrick said, yes, if they don't they are broken. Especially so if Xvfb does not represent a modern X anymore, as that would mean that tests will behave differently depending on where you launch them.

And non-deterministic tests are stupid, useless, broken tests.