Multi-User Labeling with MONAILabel-Backed CVAT

[bilalcodehub]

Dear MONAILabel Team,

I’m exploring the use of MONAILabel-backed CVAT for multi-user labeling of 2D images. CVAT supports collaboration via organizations and users, with personal workspaces provided by default in offline installations. Currently, MONAILabel creates projects and tasks seamlessly in personal workspaces, which works well.

I have a few questions:

1. Can MONAILabel create CVAT organizations, similar to how it creates projects?
2. Does MONAILabel support creating projects/tasks within a CVAT organization?
3. Is it possible to create multiple jobs within a MONAILabel-generated task? For instance, instead of one job based on segment size, could we generate five jobs that can be assigned to different users for completion?

Thank you for your guidance and support!

Best regards,
Muhammad

[diazandr3s]

Hi @bilalcodehub,

Thanks for those great questions about MONAI Label and CVAT

First off, I should mention that the CVAT implementation in MONAI Label is a reference implementation. There's definitely room for improvement, so keep that in mind. The open-source nature of MONAI Label means that developers could potentially modify the CVAT datastore and upload logic to implement these features.

Can MONAILabel create CVAT organizations, similar to how it creates projects?

Does MONAILabel support creating projects/tasks within a CVAT organization?

While MONAI Label currently doesn't have built-in functionality to create CVAT organizations or multiple tasks within a CVAT organization, there are no significant technical blockers preventing these features from being implemented.

MONAI Label integration with CVAT focuses primarily on creating projects and tasks within CVAT. Unfortunately it does not have the functionality of creating CVAT organizations.

Is it possible to create multiple jobs within a MONAILabel-generated task? For instance, instead of one job based on segment size, could we generate five jobs that can be assigned to different users for completion?

Yes, it is possible. One MONAI Label could serve multiple models - one model per task?

Hope this helps,

[bilalcodehub]

@diazandr3s: Thank you for your response—it’s much appreciated! While waiting, I took the initiative to explore and implement the features I inquired about. I spent time understanding how CVAT works, particularly the implementation of cvat.py.

I’m happy to share that I successfully implemented the requested features. Although I’m new to this, I’d love to contribute the improvements we’ve made to the MONAILabel project, as these additions are crucial for maximizing the platform’s utility.

Can MONAILabel create CVAT organizations, similar to how it creates projects?
Does MONAILabel support creating projects/tasks within a CVAT organization?

To address these, I introduced two new parameters to the MONAILabel start_server command: cvat_org_name and cvat_org_slug. These enable the creation of organizations in the CVAT database. I also added a method for organization creation and made some adjustments to the project and task functionality. Everything is working as expected—organizations are created when these values are provided, and projects, tasks, and jobs are generated seamlessly.

Is it possible to create multiple jobs within a MONAILabel-generated task? For instance, instead of one job based on segment size, could we generate five jobs that can be assigned to different users for completion?

This has already been implemented using cvat_segment_size and cvat_epistemic_top_k. I had to experiment with several configurations to achieve the desired behavior.

The final piece I’m working on is automating annotator assignment for the created jobs, making the process fully end-to-end by eliminating manual steps in CVAT. One potential solution involves reading a file with a list of users, creating CVAT accounts with default passwords, and assigning jobs based on a predefined strategy. I’ll keep you updated on this.

Overall, thanks a lot for your guidance and support. I’m looking forward to using MONAILabel as the de facto platform for our clinical partners.

[diazandr3s]

Many thanks for the update, @bilalcodehub!! This is great to hear :)