Efficient rollbacks to support predict-rollback netcode

[bfelbo]

Jolt seems like a perfect fit as a physics engine for multiplayer games using predict-rollback netcode architecture as it has built-in support for rolling back simulations.

However, the docs on this mention that you need to yourself manage any changes happening outside of the update step:

SaveState only records the state that the physics engine modifies during its update step (positions, velocities etc.), so if you change anything else you need to restore this yourself. E.g. if you did a SetFriction on frame N + 2 then, when rewinding, you need to restore the friction to what is was on frame N and update it again on frame N + 2 when you replay. If you start adding/removing objects (e.g. bodies or constraints) during these frames, the RestoreState function will not work. If you added a body on frame N + 1, you'll need to remove it when rewinding and then add it back on frame N + 1 again (with the proper initial position/velocity etc. because it won't be contained in the snapshot at frame N).

This adds a lot of additional complexity. Every multiplayer game will need to implement and manage this do/undo tracking themselves, which is highly prone to errors.

If Jolt added support for saving state + rolling back other changes, it would be much easier to make multiplayer games using it. For instance, it'd be great if Jolt supported save + roll back for adding/removing objects. Would you consider adding support in Jolt for this?

[jrouwe]

The reason I didn't do this is that it would create a lot of state to be sent over the wire every frame. It is unlikely that for every object you modify all of the properties every frame. E.g. most objects would have a constant friction for their entire lifetime, but since I don't know if this will be the case it is hard for me to send sparse information. Also I don't know anything about the state of the clients, so maybe client A and B are in sync, friction of object X changes so I'll send the friction update, but then later client C joins and needs the full state even though the friction didn't change.

So the rule is: I send all object information that is changed by the engine itself (and a lot of that state is stored in private members), so that you have the minimum set that you need.

Everything else is accessible through getters and setters and if you call the setter, you need to make sure that it gets called on the remote client as well.

[bfelbo]

Thanks for the quick answer! I'm not sure I follow though. In the predict-rollback netcode architecture, the only thing sent over the wire is the client inputs. Then the server and other clients are simulating the world based on those inputs.

For context, we've built a JS-based predict-rollback netcode solution. There we only send player inputs over the networking socket. All game logic is rolled back when receiving an incoming message and then clients resimulate on top of that to get back to the current tick.

I'm looking into whether we can integrate Jolt for physics. For that, we'd need to be able to efficiently restore Jolt to a previous state a few frames back w/o manually keeping track of friction etc. Ideally, Jolt should store the changes within the physics engine itself so that it efficiently only stores changes and doesn't need to do a full copy of the world every frame.

[jrouwe]

Yes, sorry, I was thinking about another discussion I had here where the resulting state is sent over the wire and some effort was made to reduce the state to a minimum.

Keeping track of changes to bodies and storing a minimal diff is something I don't think belongs in a physics engine (it would use memory to track the changes and cost performance for everyone, not just for people who use the feature) so I think this is most easily added as a small layer on top of the physics engine. It wouldn't be much more than a wrapper around a Body that implements e.g. a SetFriction call that serializes the change and then passes it on to the underlying Body. I'm not sure what kind of physics engine you're currently using, but I don't know of any that implements this kind of change tracking out of the box.

[bfelbo]

Keeping track of changes to bodies and storing a minimal diff (...) would use memory to track the changes and cost performance for everyone, not just for people who use the feature

Yes, if added, I'd imagine if the tracking of changes would be disabled by default. That way, it'd only have a cost performance for people using the feature.

I think this is most easily added as a small layer on top of the physics engine. It wouldn't be much more than a wrapper around a Body that implements e.g. a SetFriction call that serializes the change and then passes it on to the underlying Body

We could easily add a wrapper on top for SetFriction. However, for a world with many bodies, doing SaveState every frame could be quite expensive if the physics engine saves the entire world. It'd be much much more efficient if the physics engine was tracking differences for each body and that's not something that can be easily or efficiently done with a wrapper. Just calling SaveState will be expensive in itself if there's many bodies if it saves the entire world instead of tracking differences.

I'm not sure what kind of physics engine you're currently using, but I don't know of any that implements this kind of change tracking out of the box.

E.g. Photon Quantum implements this. It's a combined multiplayer + physics engine so it was deliberately designed for predict-rollback netcode with efficient change tracking. It's used for AAA games like Stumble Guys with 25M DAU.

Most multiplayer games these days use predict-rollback netcode regardless of genre and platform as it's so efficient and has lots of benefits such as client-side simulation, replays, and bots. Adding efficient change tracking would make Jolt the obvious choice for making multiplayer games. I know it's a big undertaking, but perhaps worth it? :)

[bfelbo]

Jolt is in a great position to be the top multiplayer physics engine as it already has solid determinism across platforms 🥳

[jrouwe]

However, for a world with many bodies, doing SaveState every frame could be quite expensive if the physics engine saves the entire world. It'd be much much more efficient if the physics engine was tracking differences for each body and that's not something that can be easily or efficiently done with a wrapper.

If you go with what I suggested, you don't need to iterate over all the bodies. If you call SetFriction on your Body wrapper class, it will internally call SetFriction on the actual Jolt Body and then register 'friction' as changed in a bit field and add the Body wrapper to a list to be scanned when the state needs to be saved. That way you only send the minimal amount of state and iterate the minimum number of bodies. I can't make it more efficient than that as I wouldn't want to store the 'property X is changed' flags inside the Body class and also wouldn't want to store the 'list of changed bodies' (this costs memory, requires synchronization in a multithreaded scenario and costs performance for the extra tracking that all users will pay for). The number of properties that can be changed is very high (e.g. you can change an entire shape). If you know which properties you allow to be changed at run-time you can do it far more efficient than I could ever do it. I'm not saying that I will never provide such a wrapper, but for now I will leave this to be implemented on the caller side.

[bfelbo]

Thanks, that makes sense. It might end up a bit complex to also track added/removed bodies, especially as everything needs to be reapplied in the exact same order. That could be fine though. I completely understand why you think this should be the responsibility of an external wrapper.

As part of this discussion, I’ve realized though that a more important issue for us to use Jolt for a predict-rollback multiplayer game is that the engine doesn't easily allow for efficient rollbacks of actual physics system state such as position and velocity. Apologies if this wasn’t clear in my previous comment. I've now updated the issue title.

I'll take a step back to try to give the proper context. Imagine that I want to use Jolt for building a multiplayer GTA game. For such a game, a predict-rollback architecture is ideal as the world state is too large to send over the wire continuously so you’d want each client to simulate the world independently and deterministically. If useful, here’s a great GDC talk for more details on predict-rollback networking.

In this GTA game, there's a bunch of static and dynamic physics bodies. Based on the code/docs, Jolt’s SaveState would save everything into a snapshot and Jolt’s RestoreState would restore everything + recompute bounding boxes etc. This would include all the static bodies (e.g. buildings and parked cars), which wouldn’t have changed at all and thus be a lot of unnecessary CPU overhead to save + restore. This would happen many times per second as part of predict-rollback.

SaveBodyState and RestoreBodyState seem useful, but it’s unclear to me how I’d use them for this GTA game as it’d be hard to know in an external wrapper which bodies to save/restore. I wouldn't be able to just track the players' cars as a moving car might collide with a parked car etc.

I'm still new to Jolt and might be missing something. How would you save + restore world state with Jolt to implement a multiplayer GTA? :)

[bfelbo]

wouldn't want to store the 'list of changed bodies' (this costs memory, requires synchronization in a multithreaded scenario and costs performance for the extra tracking that all users will pay for)

This could be an option with a performance penalty similar to the CROSS_PLATFORM_DETERMINISTIC option in CMake. It'd be fine if the new flag would enforce running Jolt with a single thread. I know that Rocket League deliberately used single-threaded physics for simplicity with the predict-rollback netcode. We would also only use a single thread and I'd expect many other multiplayer games would do the same.

[jrouwe]

Thanks, that makes sense. It might end up a bit complex to also track added/removed bodies, especially as everything needs to be reapplied in the exact same order.

The documentation oversimplifies things a bit. If you do things in the same order it is guaranteed to be deterministic, but if you know what you're doing you can take some liberties. E.g. doing BodyA.SetFriction(...); BodyB.SetFriction(...); or BodyB.SetFriction(...); BodyA.SetFriction(...); doesn't matter for determinism,
the main thing you need to ensure is that when you do a PhysicsSystem::Update that the binary state is the same.
Also adding body A then B is the same as B then A as long as the BodyIDs of A and B are consistent.
For constraints, adding A then B or B then A is equivalent as long as ConstraintSettings::mConstraintPriority is unique per constraint so that it defines a consistent ordering (in this case all constraints in the system must have a unique number).
Note though that PhysicsSystem::SaveState relies on the ordering of constraints, so you'll have to call Constraint::SaveState / Constraint::RestoreState directly yourself.

This would include all the static bodies (e.g. buildings and parked cars), which wouldn’t have changed at all and thus be a lot of unnecessary CPU overhead to save + restore

The SaveState function comes with a StateRecorderFilter::ShouldSaveBody interface which could simply return false for all static bodies + inactive bodies (you'd have to be a bit careful and track active bodies before the physics step that may have gone to sleep during the step and consider them active).

This could be an option with a performance penalty similar to the CROSS_PLATFORM_DETERMINISTIC option in CMake.

I'm trying to limit the amount of defines as they explode the number of builds to test. As said before, I don't think it's needed as it is easily tracked outside of the physics engine.

It'd be fine if the new flag would enforce running Jolt with a single thread. I know that Rocket League deliberately used single-threaded physics for simplicity with the predict-rollback netcode. We would also only use a single thread and I'd expect many other multiplayer games would do the same.

I think this would be a big shame and totally not needed as I put a lot of effort in making the sim deterministic while still using multiple threads.

[bfelbo]

Super helpful with all the details on ordering and StateRecorderFilter::ShouldSaveBody interface!

Appreciate you really going into details to answer my questiona! Sounds like an external tracking system is a more viable approach than I thought. Thanks!

[jrouwe]

Re-opening the discussion as otherwise it becomes hidden by default and I think this thread may contain useful information for other people.

[TIHan]

Jolt is very viable for prediction and rollback corrections, just based purely on my experience with it.

Also adding body A then B is the same as B then A as long as the BodyIDs of A and B are consistent.

This is exactly what I did. I ensure the BodyIDs are identical on client+server.