[Raylib 6.0] Model Animation Discussion

[JeffM2501]

Point of Discussion

There has been some discussion about what we can do to make raylib's 3d animation system more robust. The current API and model structure are heavily focused around CPU based animation and there being one animation instance per model. With the addition of GPU animation support and advanced needs of raylib users, this API is proving to be weak.

Several people have different ideas on how to enhance the API, but in different ways, and this leads to confusion and large PR "code drops" that people don't understand.
This discussion is to talk about the needs of raylib and come up with a design before talk of any code.

Design of raylib

Raylib is designed first and foremost as a library to help people get into gamedev and learn how lower level systems are made. As has been said many times before, raylib is not an engine, and along that line, we should not entertain complex systems that are not easy for new users to pickup and user. Raylib APIs are meant to be simple and meant to be extended. We should focus on the lowest level APIs possible to do what is desired, and leave higher level systems up to external/drop in code (such as what is hosted on raylib-extras).

GPU vs CPU animation

Raylib has traditionally only supported CPU based animation, with GPU support just added in 5.5, because of this much of the API and structure is based around the model and mesh structure storing the current state of the transformed vertices. This is not always the best case for GPU based animation.

Splitting the current pose out from the model

Currently the animation data and animation pose cache is stored spread over several areas, some in the model, some in the animation, and some in the mesh.
I don't feel that storing any animation data in the mesh is correct, as there are no raylib APIs that use the data directly on the mesh, it's always used in the context of a model. Thus I propose that we remove the animation data from the mesh itself, DrawMesh doesn't need it, so why should it be there?

We should make a new structure called ModelAnimationPose, and it should contain a list of MeshAnimationPoses, in the same order as the meshes in a model. This structure would contain the animation related data that is currently in the mesh, most importantly the boneMatrices list. The boneMatrices is a cache of the current bone transforms for a pose. It is what the base vertices are transformed by for CPU animation, and are what is uploaded to the shader for GPU animation.

The model structure itself should hold one of these pose structures, giving the model the same storage for a 'current pose' that it has now, allowing all the existing animation functions to continue to work.

Any new APIs that change the pose on a model would take a ModelAnimationPose as input and modify it to the new post. Again, this lets all existing animation functions work as expected.

We should make low level functions that apply a type of animation to a pose, such as
SetModelAnimationPoseToFrame(Model*, ModelAnimation*, ModelAnimationPose*, int frame);
InterpolateModelAnimationPoseToFrame(Model*, ModelAnimation*, ModelAnimationPose*, int frame1, int frame2, float paramater);
These functions would simply update the pose passed in.

GPU based animations would then just upload the matrix cache for a mesh from the post as they draw the model.

For CPU animation we should have a function that applies a pose to the vertices of a model, by transforming the verts and uploading them to the GPU
ApplyPoseToModel(Model*, ModelAnimationPose*);

Pose Per Instance

Now that we have the animation system working on a self contained pose structure, we can add functions to extend this to add more features.

We can add a function to clone a pose.
ModelAnimationPose CopyAnimationPose(ModelAnimationPose*);
This will allow games that reuse the same model with different animation states to cache a pose per instance and update them as necessary. Right now if I have 10 copies of a mesh in different parts of the animation, I have to constantly rewrite the boneMatrices of the meshes before I draw anything. I would much rather maintain my own cache of poses and update them as needed (possibly in a thread), but the current system prevents that.

We can then also look into adding additional functions, internal, or external to update a pose in different ways such as doing animation blending, or programmatic modification of bones for features such as mouse control or inverse kinematics.

The separation of pose and model gives us a large range of possibilities for the future, with no real downside on the raylib side.

Scope

I think we should start with this low level of changes so that we can have a good foundation for storing and applying animation states as poses. This will then people expand as needed.
These low level functions will be used to reimplement the current high level animation functions so they work the same as they always have.

Then we can prototype/discuss further high level functions after this low level framework is in place.

[Kirandeep-Singh-Khehra]

IMO Using ModelAnimationPose as name might not be a good idea. Because across raylib the term "pose" is generally represented by Transform *. Like bindPose in Model is Transform * and framePoses in ModelAnimation is of Transform **(Array of poses). So, it might create two definitions of word Pose. I would suggest words similar to Rig(not good name in animation domain), Skeleton(way too broad) or Armature(good but not used a lot in game development but in sounds good in modeling) (<-- Please don't consider these as suggestions for functionality but name and a rough base for new Name).

1. Model.bindPose reference
   

   raylib/src/raylib.h

   Line 417 in 79facde

   Transform *bindPose; // Bones base transformation (pose)

2. ModelAnimation.framePoses reference

   raylib/src/raylib.h

   Line 425 in 79facde

   Transform **framePoses; // Poses array by frame

[Kirandeep-Singh-Khehra]

As per the requirements discussed i am trying to implement a drop in system here https://github.com/Kirandeep-Singh-Khehra/raylib-3d-anim-system. It is using layer based animation approach with Skeleton and Pose and needs no change in raylib code. Here are types defined:

typedef Transform* Pose;

typedef struct Skeleton {
  /* Extracted from Model */
  int boneCount;         // Number of bones
  BoneInfo *bones;       // Bones information (skeleton)
  Pose bindPose;   // Bones base transformation (pose)

  /* Extracted from Mesh */
  Matrix *boneMatrices;  // Bones animated transformation matrices (not used yet)

  /* Custom defined */
  Pose pose;      // Current pose
} Skeleton;

And here is a basic usage example.
https://github.com/Kirandeep-Singh-Khehra/raylib-3d-anim-system/blob/main/examples/skeleton/skeleton_animation_blending.c

Primary intent for this repo is to act as a playground or external base and then add some contributions (step by step) of basic commonly used and simple functions to raylib when it will be good enough to call it stable. Currently naming convention is poor and some redundant functions are present.

Please do give it a look(and little review if possible). And if anything in it looks good and can be added safely to raylib now or anytime in future then do let me know.

[Kirandeep-Singh-Khehra]

We might also need a pose with local transformations(relative to parent bone). It can add possibility of Forward Kinematics, better blending and additive blending. and it will make it little easy to have IK(Inverse Kinematics) too.

[JeffM2501]

I highly recommend we get the foundational API in place before we work on drop in libs for higher level functions.

[Kirandeep-Singh-Khehra]

Yes, i totally agree, and i am on same path but different approach. I m trying to implement high level stuff and then breaking down those to build a low level functions and then iterating back up. It also helps me to get an idea of what all is needed. And as per my current progress only Transform* pose is required in struct Model to do all things related to pose. And some bunch of functions around typedef Transform *Pose. This makes it easier to work with anim.framePoses[frame] too. Like some implemented here in pose.c. But this implementation is not good in terms of memory management and there are some useless functions too. I need to rewrite this whole part(I know i'm again jumping to code. But that's my habit to continuously iterate up and down and i'm not forcing any high level functions but just trying to get the minimal set and structure of low level systems for easy implementation most of high level things).

[JeffM2501]

In looking at the current API, one thing that I don't like is that DrawMesh sets the uniforms in the shader, I think that should be done at model/material level.
Right now we are re-sending all the bones for all meshes for every mesh.
I think DrawModel should walk the material list and upload all the matrices once, from the pose.
this way we also don't need to deep copy the pose to every mesh.

[JeffM2501]

I have made a prototype branch with some low level functions to support the ideas I proposed above.
https://github.com/JeffM2501/raylib/tree/external_animation_pose

The main change is moving the animation support out of the mesh and into the model. The bones are already all the bones for all meshes in a model, and an animation is model based.

All the existing APIs work exactly as they did before, the animation data is just now stored in a "ModelBonePose" structure on the model instead of duplicated for each mesh, thus this change uses less memory and does fewer deep copies than the previous one, but the model structure is now 1 size_t larger, while the models are 1 int and 1 size_t smaller.

I have also added a few extended APIs that work directly on ModelBonePose structures to allow callers to keep separate states for multiple draws of the same model during the same frame. The GPU mesh animation example has been updated to run 2 animations to show this.

The only breaking change is that this API no longer supports animation during DrawMeshInstanced using a GPU animation shader, because the animation data is not part of the mesh anymore. If we want to support animation in instanced models, then we'd need to add DrawModelInstanced that did the matrix upload. Note that I don't think anyone did this before now (all instances would need to use the same pose), so I don't think anyone is majorly broken.

If this looks good, it can be extended to add interpolation support when updating a pose. I am not married to any of the specific names, so if there are better suggestions, please let me know.

[Kirandeep-Singh-Khehra]

This is really good and easy to understand. Working with matrices for pose is a great idea. And a lot less intermediary steps between loading, updating and draw calls.

[Kirandeep-Singh-Khehra]

I don't know most of maths but will matrices(as pose) cause some issues like buggy interpolation(considering matrix decomposition as overhead to fix it) and other things like position of bone attachments(Only idea in my mind is to first move the object/attachment(like sword) at place with respect to bindpose using transform matrices and then multiply it to its parent bone matrices to keep it in place). And advanced foot placement and collisions in ragdolls(and other examples which need position of bone or only one component of transform like position, rotation or scale) can become complex. Are their any ways or plans to make this easy and simple.

[JettMonstersGoBoom]

This is a good proposal, as folks try and create more advanced animation systems for their games, By allowing the developer to update the matrices as they see fit is a really positive step.
This change doesn't need to be the most complex system, and is actually simpler than the current system to implement ( once untangled )
the previous animation system was too heavily tied to the model structures and as has been pointed out, led to confusion.
this change opens the door to ragdolls, IK, FK, and also more mesh and animation formats.
Is there any thought to supplying an API that matches the current one, using the new system with appropriate setup ?

[JeffM2501]

I have found some more duplicated data that I am in the process of consolidating,
1)each animation has a full copy of the bone info for all bones, this seems superfluous. Every time we use the animation we have a model since we need the bind pose, so it makes sense to store the skeleton entirely in the model.

I am going to make some incremental changes

1. move all bone info into the model and just keep one copy.
2. see about moving the entire skeleton into it's own structure just to keep it logically grouped.

[JeffM2501]

I guess the only real reason the animation has any info about the bones is that it's used to validate that the animation goes with the model.

This seems like a lot of data just to validate something the user has to manage anyway.

Should it just not be on the user to ensure that the animations go with the skeleton? Raylib doesn't do this kind of compatibility check for anything else.

[JettMonstersGoBoom]

Minimum expectation is throw a log error with mismatched number of bones and refuse to update at all. The model only needs it's bind pose but keeping bonenames could be useful AND help detect mismatched data ?
I agree it's on the user to use the correct skeleton though.

[JeffM2501]

I don't want to remove data from the model, I want to remove the duplicate bones from the each animation sequence is storing a complete copy of the boneinfos, 36 bytes per bone, per sequence, all duplicated.

I'm trying to think where else raylib logs an error if the user screws up like this. If you pass in the wrong image buffer size to a raylib function, it does not log an error, it just crashes.

[JeffM2501]

Optimally we could compute a hash for each unique skeleton and just store that for error checking, but that may be overkill.
Right now any validation of model->animation binding involves checking every bone to ensure they have the same parent linkage, names are not used for validation.

[JeffM2501]

To continue, I have also implemented a C++ system for my personal project that basically does what my proposed change to raylib does (just with some C++ features for my specific needs).

it's all part of this little 'model_lib'.

https://github.com/JeffM2501/ModernBoomerShooter/blob/main/model_lib/include/model.h