Development

.github/workflows/docs.yml

@@ -0,0 +1,45 @@
+name: documentation
+
+on: 
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+    paths: 
+      - 'src/**.py'
+      - 'docs/**'
+      - '.github/workflows/docs.yml'
+
+permissions:
+  contents: write
+
+jobs:
+  docs:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Install poetry
+        uses: abatilo/actions-poetry@v3
+
+      - name: Install dependencies
+        run: |
+          poetry install --without test,dev
+
+      - name: Sphinx build
+        run: |
+          source $(poetry env info --path)/bin/activate
+          cd docs && make html
+
+      - name: Deploy to GitHub Pages
+        uses: peaceiris/actions-gh-pages@v3
+        if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
+        with:
+          publish_branch: gh-pages
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_dir: docs/_build/html
+          force_orphan: true

CONTRIBUTING.md

@@ -1,23 +1,23 @@
-Install the repository after cloning with [poetry](https://python-poetry.org/), and setup [pre-commit](https://pre-commit.com/) such that code is linted and formatted with [Ruff](https://docs.astral.sh/ruff/).
+Install the repository after cloning with [poetry](https://python-poetry.org/), and setup [pre-commit](https://pre-commit.com/) such that code is linted and formatted with [Ruff](https://docs.astral.sh/ruff/) and checked with [mypy](https://mypy-lang.org/).
 
 ```bash
-> pip install poetry
-> cd textmate-grammar-python
-> poetry install
-> pre-commit install
+pip install poetry
+cd textmate-grammar-python
+poetry install
+pre-commit install
 ```
 
 Run unit tests
 ```bash
-> tox run
+tox run
 ```
 
 Run static type checker
 ```bash
-> tox run -e mypy
+tox run -e mypy
 ```
 
-Run regression testing against vscode-textmate (will install npm and required packages)
+Run regression testing against vscode-textmate (will install npm and required packages). 
 ```bash
-> tox run -e regression
+tox run -e regression
 ```

README.md

@@ -5,15 +5,35 @@
 [![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)
 [![Python versions](https://img.shields.io/pypi/pyversions/textmate-grammar-python.svg)](https://pypi.python.org/pypi/textmate-grammar-python)
 [![CI/CD](https://github.com/watermarkhu/textmate-grammar-python/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/watermarkhu/textmate-grammar-python/blob/main/.github/workflows/ci.yml)
-
+[![readthedocs](https://readthedocs.org/projects/textmate-grammar-python/badge/?version=latest)](https://textmate-grammar-python.readthedocs.io)
 
 # textmate-grammar-python
 
-A lexer and tokenizer for grammar files as defined by TextMate and used in VSCode, implemented in Python. TextMate grammars use the oniguruma dialect (https://github.com/kkos/oniguruma). Supports loading grammar files from JSON, PLIST, or YAML format. 
+A lexer and tokenizer for grammar files as defined by TextMate and used in VSCode, implemented in Python. 
+
+Textmate grammars are made for [vscode-texmate](https://github.com/microsoft/vscode-textmate), allowing for syntax highlighting in VSCode after tokenization. This presents textmate-grammar-python with a large list of potentially supported languages. 
+
+```mermaid
+flowchart TD
+    A[grammar file] 
+    Z[code]
+    B("`vscode-textmate **js**`")
+    C("`textmate-grammar-**python**`")
+    D[tokens]
 
+    click C "https://github.com/microsoft/vscode-textmate"
+
+    Z --> B
+    Z --> C
+    A -.-> B --> D
+    A -.-> C --> D
+```
 
 ## Usage
-Install the module using `pip install textmate-grammar-python`.
+Install the module with:
+```bash
+pip install textmate-grammar-python
+```
 
 Before tokenization is possible, a `LanguageParser` needs to be initialized using a loaded grammar. 
 
@@ -23,9 +43,9 @@ from textmate_grammar.grammars import matlab
 parser = LanguageParser(matlab.GRAMMAR)
 ```
 
-After this, one can either choose to call `parser.parsing_string` to parse a input string directly, or call `parser.parse_file` with the path to the appropiate source file as the first argument, such as the the example `example.py`. 
+After this, one can either choose to call [`parser.parsing_string`](https://textmate-grammar-python.readthedocs.io/en/latest/apidocs/textmate_grammar/textmate_grammar.language.html#textmate_grammar.language.LanguageParser.parse_string) to parse a input string directly, or call [`parser.parse_file`](https://textmate-grammar-python.readthedocs.io/en/latest/apidocs/textmate_grammar/textmate_grammar.language.html#textmate_grammar.language.LanguageParser.parse_file) with the path to the appropiate source file as the first argument, such as in the example [`example.py`](https://github.com/watermarkhu/textmate-grammar-python/blob/main/example.py). 
 
-The parsed `element` object can be displayed directly by calling the `print` method. By default the element is printed as an element tree in a dictionary format. 
+The parsed `element` object can be displayed directly by calling the [`print`](https://textmate-grammar-python.readthedocs.io/en/latest/apidocs/textmate_grammar/textmate_grammar.elements.html#textmate_grammar.elements.ContentElement.print) method. By default the element is printed as an element tree in a dictionary format. 
 
 ```python
 >>> element = parser.parse_string("value = num2str(10);")
@@ -59,9 +79,7 @@ Alternatively, with the keyword argument `flatten` the element is displayed as a
  [(0, 19), ';', ['source.matlab', 'punctuation.terminator.semicolon.matlab']]]
 ```
 
-## Supported Languages
-- [MATLAB](https://github.com/mathworks/MATLAB-Language-grammar)
-
-## TODO
-- Implement Begin/While pattern, required for other grammars.
+## Information
 
+- For further information, please checkout the [documentation](https://textmate-grammar-python.readthedocs.io/en/latest/). 
+- To setup an environment for development, see [CONTRIBUTING.md](https://github.com/watermarkhu/textmate-grammar-python/blob/main/CONTRIBUTING.md)