Synthetic data generation using existing data

[jvhgit]

Hi,

I was wondering about the following.
Is it possible to use existing data such a private code repositories or private documentation and use that as input for the synthetic data generation?
I am looking for a scalable way to leverage unstructered information in my field/company into structered chat format without the need for actual direct user feedback (as you would get from thumbs up/down kind of feedback in an application). I think this could give a model a better starting point for company specific use cases.

Kind regards,

[davidberenstein1957]

@jvhgit, This is possible for sure, but that beats the purpose of the tool in the current state it is in. In the short term, we do plan on adding support for ingesting custom and private data from the Hub for RAG (#10) and evaluation(#9) purposes. Feel free add some more thoughts and user scenarios that might help us shape the tool to make it as useful as possible to you and the community.

[Israel-Laguan]

@davidberenstein1957 what about ingesting a json or csv (maybe you can define the schema or we can select in a dropdown, etc.), but the idea of using existing data is sound in these scenarios

• this very same tool failed mid generation and we want to continue
• we have some old dataset we just need to expand
• he would need the upcoming JUDGE (add support for dataset evaluation with LLMs as a Judge #9) with some existing dataset

[davidberenstein1957]

Hi @Israel-Laguan,

• Feel free to try again. We might have been changing/updating the tool at the time due to some scaling issues.
• W.r.t. expanding, for which scenarios would you like to do that? SFT/Textcat or others too?
• Yes we agree, there is already some logic for this in te codebase already. https://github.com/argilla-io/synthetic-data-generator/blob/main/src/synthetic_dataset_generator/apps/eval.py . you can test it out by installing it, importing it and adding it like shown here https://github.com/argilla-io/synthetic-data-generator/blob/main/src/synthetic_dataset_generator/app.py