diff --git a/.github/workflows/documentation.yml b/.github/workflows/documentation.yml
new file mode 100644
index 0000000..1d7241b
--- /dev/null
+++ b/.github/workflows/documentation.yml
@@ -0,0 +1,34 @@
+# creates the documentation on pushes it to the gh-pages branch
+name: Documentation
+
+on:
+ pull_request:
+ branches: [main]
+ push:
+ branches: [main]
+
+
+permissions:
+ contents: write
+
+jobs:
+ deploy:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v3
+ - uses: actions/setup-python@v4
+ with:
+ python-version: '3.10'
+
+ - name: Dependencies
+ run: |
+ python -m pip install --upgrade pip
+ pip install turftopic[pyro-ppl,docs]
+
+ - name: Build and Deploy
+ if: github.event_name == 'push'
+ run: mkdocs gh-deploy --force
+
+ - name: Build
+ if: github.event_name == 'pull_request'
+ run: mkdocs build
\ No newline at end of file
diff --git a/docs/clustering.md b/docs/clustering.md
index a7c2a55..58df63f 100644
--- a/docs/clustering.md
+++ b/docs/clustering.md
@@ -188,6 +188,11 @@ top2vec = ClusteringTopicModel(
Theoretically the model descriptions above should result in the same behaviour as the other two packages, but there might be minor changes in implementation.
We do not intend to keep up with changes in Top2Vec's and BERTopic's internal implementation details indefinitely.
+### _(Optional)_ 5. Dynamic Modeling
+
+Clustering models are also capable of dynamic topic modeling. This happens by fitting a clustering model over the entire corpus, as we expect that there is only one semantic model generating the documents.
+To gain temporal representations for topics, the corpus is divided into equal, or arbitrarily chosen time slices, and then term importances are estimated using Soft-c-TF-IDF, c-TF-IDF, or distances from cluster centroid for each of the time slices separately. When distance from cluster centroids is used to estimate topic importances in dynamic modeling, cluster centroids are computed based on documents and terms present within a given time slice.
+
## Considerations
### Strengths
diff --git a/docs/dynamic.md b/docs/dynamic.md
index 9e90628..3120d95 100644
--- a/docs/dynamic.md
+++ b/docs/dynamic.md
@@ -28,7 +28,7 @@ Dynamic topic models in Turftopic have a unified interface.
To fit a dynamic topic model you will need a corpus, that has been annotated with timestamps.
The timestamps need to be Python `datetime` objects, but pandas `Timestamp` object are also supported.
-Models that have dynamic modeling capabilities have a `fit_transform_dynamic()` method, that fits the model on the corpus over time.
+Models that have dynamic modeling capabilities (currently, `GMM` and `ClusteringTopicModel`) have a `fit_transform_dynamic()` method, that fits the model on the corpus over time.
```python
from datetime import datetime
diff --git a/pyproject.toml b/pyproject.toml
index ef717d4..be2c2ed 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -18,17 +18,13 @@ torch = "^2.1.0"
scipy = "^1.10.0"
rich = "^13.6.0"
pyro-ppl = { version = "^1.8.0", optional = true }
+mkdocs = { version = "^1.5.2", optional = true }
+mkdocs-material = { version = "^9.5.12", optional = true }
+mkdocstrings = { version = "^0.24.0", extras = ["python"], optional = true }
[tool.poetry.extras]
pyro-ppl = ["pyro-ppl"]
-
-[tool.poetry.group.docs]
-optional = true
-
-[tool.poetry.group.docs.dependencies]
-mkdocs = "^1.5.2"
-mkdocs-material = "^9.5.12"
-mkdocstrings = { version = "^0.24.0", extras = ["python"] }
+docs = ["mkdocs", "mkdocs-material", "mkdocstrings"]
[build-system]
requires = ["poetry-core"]
diff --git a/site/404.html b/site/404.html
deleted file mode 100644
index 2ce9a28..0000000
--- a/site/404.html
+++ /dev/null
@@ -1,625 +0,0 @@
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Turftopic
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
GMM is a generative probabilistic model over the contextual embeddings.
-The model assumes that contextual embeddings are generated from a mixture of underlying Gaussian components.
-These Gaussian components are assumed to be the topics.
-
-
- Components of a Gaussian Mixture Model (figure from scikit-learn documentation)
-
-
-
The Model
-
1. Generative Modeling
-
GMM assumes that the embeddings are generated according to the following stochastic process:
-
-
Select global topic weights: \(\Theta\)
-
For each component select mean \(\mu_z\) and covariance matrix \(\Sigma_z\) .
Priors are optionally imposed on the model parameters.
-The model is fitted either using expectation maximization or variational inference.
-
2. Topic Inference over Documents
-
After the model is fitted, soft topic labels are inferred for each document.
-A document-topic-matrix (\(T\)) is built from the likelihoods of each component given the document encodings.
-
Or in other words for document \(i\) and topic \(z\) the matrix entry will be: \(T_{iz} = p(\rho_i|\mu_z, \Sigma_z)\)
-
3. Soft c-TF-IDF
-
Term importances for the discovered Gaussian components are estimated post-hoc using a technique called Soft c-TF-IDF,
-an extension of c-TF-IDF, that can be used with continuous labels.
-
Let \(X\) be the document term matrix where each element (\(X_{ij}\)) corresponds with the number of times word \(j\) occurs in a document \(i\).
-Soft Class-based tf-idf scores for terms in a topic are then calculated in the following manner:
-
-
Estimate weight of term \(j\) for topic \(z\):
-\(tf_{zj} = \frac{t_{zj}}{w_z}\), where
-\(t_{zj} = \sum_i T_{iz} \cdot X_{ij}\) and
-\(w_{z}= \sum_i(|T_{iz}| \cdot \sum_j X_{ij})\)
-
Estimate inverse document/topic frequency for term \(j\):
-\(idf_j = log(\frac{N}{\sum_z |t_{zj}|})\), where
-\(N\) is the total number of documents.
-
Calculate importance of term \(j\) for topic \(z\):
-\(Soft-c-TF-IDF{zj} = tf_{zj} \cdot idf_j\)
-
-
(Optional) 4. Dynamic Modeling
-
GMM is also capable of dynamic topic modeling. This happens by fitting one underlying mixture model over the entire corpus, as we expect that there is only one semantic model generating the documents.
-To gain temporal representations for topics, the corpus is divided into equal, or arbitrarily chosen time slices, and then term importances are estimated using Soft-c-TF-IDF for each of the time slices separately.
-
Similarities with Clustering Models
-
Gaussian Mixtures can in some sense be considered a fuzzy clustering model.
-
Since we assume the existence of a ground truth label for each document, the model technically cannot capture multiple topics in a document,
-only uncertainty around the topic label.
-
This makes GMM better at accounting for documents which are the intersection of two or more semantically close topics.
-
Another important distinction is that clustering topic models are typically transductive, while GMM is inductive.
-This means that in the case of GMM we are inferring some underlying semantic structure, from which the different documents are generated,
-instead of just describing the corpus at hand.
-In practical terms this means that GMM can, by default infer topic labels for documents, while (some) clustering models cannot.
-
Performance Tips
-
GMM can be a bit tedious to run at scale. This is due to the fact, that the dimensionality of parameter space increases drastically with the number of mixture components, and with embedding dimensionality.
-To counteract this issue, you can use dimensionality reduction. We recommend that you use PCA, as it is a linear and interpretable method, and it can function efficiently at scale.
-
-
Through experimentation on the 20Newsgroups dataset I found that with 20 mixture components and embeddings from the all-MiniLM-L6-v2 embedding model
- reducing the dimensionality of the embeddings to 20 with PCA resulted in no performance decrease, but ran multiple times faster.
- Needless to say this difference increases with the number of topics, embedding and corpus size.
-
-
fromturftopicimportGMM
-fromsklearn.decompositionimportPCA
-
-model=GMM(20,dimensionality_reduction=PCA(20))
-
-# for very large corpora you can also use Incremental PCA with minibatches
-
-fromsklearn.decompositionimportIncrementalPCA
-
-model=GMM(20,dimensionality_reduction=IncrementalPCA(20))
-
-
Considerations
-
Strengths
-
-
Efficiency, Stability: GMM relies on a rock solid implementation in scikit-learn, you can rest assured that the model will be fast and reliable.
-
Coverage of Ingroup Variance: The model is very efficient at describing the extracted topics in all their detail.
- This means that the topic descriptions will typically cover most of the documents generated from the topic fairly well.
-
Uncertainty: GMM is capable of expressing and modeling uncertainty around topic labels for documents.
-
Dynamic Modeling: You can model changes in topics over time using GMM.
-
-
Weaknesses
-
-
Curse of Dimensionality: The dimensionality of embeddings can vary wildly from model to model. High-dimensional embeddings might decrease the efficiency and performance of GMM, as it is sensitive to the curse of dimensionality. Dimensionality reduction can help mitigate these issues.
-
Assumption of Gaussianity: The model assumes that topics are Gaussian components, it might very well be that this is not the case.
- Fortunately enough this rarely effects real-world perceived performance of models, and typically does not present an issue in practical settings.
-
Moderate Scalability: While the model is scalable to a certain extent, it is not nearly as scalable as some of the other options. If you experience issues with computational efficiency or convergence, try another model.
-
Moderate Robustness to Noise: GMM is similarly sensitive to noise and stop words as BERTopic, and can sometimes find noise components. Our experience indicates that GMM is way less volatile, and the quality of the results is more reliable than with clustering models using C-TF-IDF.
Prior to impose on component weights, if None,
-maximum likelihood is optimized with expectation maximization,
-otherwise variational inference is used.
-
-
-
- None
-
-
-
-
gamma
-
- Optional[float]
-
-
-
-
Concentration parameter of the symmetric prior.
-By default 1/n_components is used.
-Ignored when weight_prior is None.
-
-
-
- None
-
-
-
-
dimensionality_reduction
-
- Optional[TransformerMixin]
-
-
-
-
Optional dimensionality reduction step before GMM is run.
-This is recommended for very large datasets with high dimensionality,
-as the number of parameters grows vast in the model otherwise.
-We recommend using PCA, as it is a linear solution, and will likely
-result in Gaussian components.
-For even larger datasets you can use IncrementalPCA to reduce
-memory load.
KeyNMF is a topic model that relies on contextually sensitive embeddings for keyword retrieval and term importance estimation,
-while taking inspiration from classical matrix-decomposition approaches for extracting topics.
-
The Model
-
-
- Schematic overview of KeyNMF
-
-
-
1. Keyword Extraction
-
The first step of the process is gaining enhanced representations of documents by using contextual embeddings.
-Both the documents and the vocabulary get encoded with the same sentence encoder.
-
Keywords are assigned to each document based on the cosine similarity of the document embedding to the embedded words in the document.
-Only the top K words with positive cosine similarity to the document are kept.
-
These keywords are then arranged into a document-term importance matrix where each column represents a keyword that was encountered in at least one document,
-and each row is a document.
-The entries in the matrix are the cosine similarities of the given keyword to the document in semantic space.
-
2. Topic Discovery
-
Topics in this matrix are then discovered using Non-negative Matrix Factorization.
-Essentially the model tries to discover underlying dimensions/factors along which most of the variance in term importance
-can be explained.
-
Considerations
-
Strengths
-
-
Stability, Robustness and Quality: KeyNMF extracts very clean topics even when a lot of noise is present in the corpus, and the model's performance remains relatively stable across domains.
-
Scalability: The model can be fitted in an online fashion, and we recommend that you choose KeyNMF when the number of documents is large (over 100 000).
-
Fail Safe and Adjustable: Since the modelling process consists of multiple easily separable steps it is easy to repeat one if something goes wrong. This also makes it an ideal choice for production usage.
-
Can capture multiple topics in a document.
-
-
Weaknesses
-
-
Lack of Multilingual Capabilities: KeyNMF as it is currently implemented cannot be used in a multilingual context. Changes to the model that allow this are possible, and will likely be ijmplemented in the future.
-
Lack of Nuance: Since only the top K keywords are considered and used for topic extraction some of the nuances, especially in long texts might get lost. We therefore recommend that you scale K with the average length of the texts you're working with. For tweets it might be worth it to scale it down to 5, while with longer documents, a larger number (let's say 50) might be advisable.
-
Practitioners have to choose the number of topics a priori.
Extracts keywords from documents based on semantic similarity of
-term encodings to document encodings.
-Topics are then extracted with non-negative matrix factorization from
-keywords' proximity to documents.
Fit KeyNMF on very large datasets, that cannot fit in memory.
-Internally uses minibatch NMF.
-The stream of documents has to be a reusable iterable,
-as multiple passes are needed over the corpus to
-learn the vocabulary and then fit the model.
defbig_fit(
- self,
- document_stream:Iterable[str],
- keyword_file:str="./__keywords.jsonl",
- batch_size:int=500,
- max_epochs:int=100,
-):
-"""Fit KeyNMF on very large datasets, that cannot fit in memory.
- Internally uses minibatch NMF.
- The stream of documents has to be a reusable iterable,
- as multiple passes are needed over the corpus to
- learn the vocabulary and then fit the model.
- """
- console=Console()
- withconsole.status("Fitting model")asstatus:
- status.update("Learning vocabulary.")
- self.learn_vocabulary(document_stream)
- console.log("Vocabulary learnt.")
- status.update("Extracting keywords.")
- self.cache_keywords(document_stream,keyword_file,batch_size)
- console.log("Keywords extracted.")
- keywords=KeywordIterator(keyword_file)
- status.update("Fitting NMF.")
- self.minibatch_train(keywords,max_epochs,batch_size,console=console)# type: ignore
- console.log("NMF fitted.")
- returnself
-
-
-
-
-
-
-
-
-
-
-
-
-transform(raw_documents,embeddings=None)
-
-
-
-
-
-
-
Infers topic importances for new documents based on a fitted model.
- 
-