Domain adaptation

[Thiziri-Hafir]

Hello, I am working on an unsupervised classification project of a very specific corpus of textual data. It contains some common multilingual vocabulary but also a big specific vocabulary. Pre-trained models don't work very well, and I already tried to train a model on a masking task from scratch but the vocabulary size is huge. A colleague told me about your library so I would like to understand how to use it to train a masking model on my data and then fine tune the model on down stream tasks or on an embedding transformation task.?

Thank you for your help!