How to enable a mask for raycasts

[benjaminleezhiyuan]

Hi I was wondering how I would use a mask for raycasting to check against certain layers only.

Currently I have the default layers moving and non moving.

So if I set the layerMask to 0011 it should check for collisions on both layers.

bool PhysicsSystem::RayCast(glm::vec3 origin, glm::vec3 direction, float maxDistance, Bitset32 LayerMask)
	{
		direction = glm::normalize(direction);
		direction *= maxDistance; //set distance of ray
		RRayCast ray { Vec3(origin.x, origin.y, origin.z), Vec3(direction.x, direction.y, direction.z) };
		auto& narrowPhaseQuery = sData.mSystem->GetNarrowPhaseQuery();
		RayCastResult result;
		return narrowPhaseQuery.CastRay(ray, result);
	}

I've tried doing it this way

Bitset32 layerMask;
layerMask.set(1);

if (PhysicsSystem::RayCast(glm::vec3(0, 0, 0), glm::vec3(0, -1, 0), 5.f, layerMask))
{
    BOREALIS_CORE_INFO("Ray hit");
}

bool PhysicsSystem::RayCast(glm::vec3 origin, glm::vec3 direction, float maxDistance, Bitset32 LayerMask)
{
    direction = glm::normalize(direction);
    direction *= maxDistance; // set distance of ray
    RRayCast ray { Vec3(origin.x, origin.y, origin.z), Vec3(direction.x, direction.y, direction.z) };
    auto& narrowPhaseQuery = sData.mSystem->GetNarrowPhaseQuery();
    RayCastResult result;

    if (narrowPhaseQuery.CastRay(ray, result))
    {
        JPH::BodyLockRead lock(sData.mSystem->GetBodyLockInterface(), result.mBodyID);
        if (lock.Succeeded())
        {
            const JPH::Body& body = lock.GetBody();
            ObjectLayer objectLayer = body.GetObjectLayer();

            if (LayerMask.to_ulong() & (1u << objectLayer))
            {
                return true;
            }
            else
            {
                return false;
            }
        }
    }

    return false;
}

But the issue is that if the object im looking for is behind an object on the wrong layer it will return false.

[jrouwe]

I would suggest reading the collision filtering section: https://jrouwe.github.io/JoltPhysics/#collision-filtering

Basically, you should be using the BroadPhaseLayerFilter/ObjectLayerFilter interfaces to filter out the hits. The version of CastRay you're using will return the first hit that wasn't filtered, so if the object is 'behind an object' then you will only see the object that comes first unless you filtered the hit out using a filter function. Given that you want to use bitmasks for collision filtering you should probably take a look at ObjectLayerPairFilterMask, ObjectVsBroadPhaseLayerFilterMask and BroadPhaseLayerInterfaceMask.

[benjaminleezhiyuan]

Yeah we were able to implement something like this that works for our use case

class ObjectLayerFilterImpl : public ObjectLayerFilter
{
public:
	ObjectLayerFilterImpl(Bitset32 l) : bitset(l)
	{

	}
	bool ShouldCollide(ObjectLayer inLayer) const override
	{
		// some function to extract the bitset to index numbers;
		// for example: bitset contains index 3 and 6 (layer 3 & 6)

		for (int index : bitset.ToBitsList())
		{
			if (index == inLayer)
			{
				return true;
			}
		}
		return false;
	}
private:
	Bitset32 bitset;
};