diff --git a/docs/source/cli.md b/docs/source/cli.md new file mode 100644 index 00000000..0aed12f4 --- /dev/null +++ b/docs/source/cli.md @@ -0,0 +1,71 @@ +# Datapipe CLI + +Datapipe provides `datapipe` CLI tool which can be useful for inspecting +pipeline, tables, and running steps. + +`datapipe` CLI is build using `click` and provides several levels of commands +and subcommands each of which can have parameters. `click` parameters are +level-specific, i.e. global-level arguments should be specified at global level +only: + +`datapipe --debug run`, but NOT `datapipe run --debug` + +## Global arguments + +### `--pipeline` + +By default `datapipe` looks for a file `app.py` in working directory and looks +for `app` object of type `DatapipeApp` inside this file. `--pipeline` argument +allows user to provide location for `DatapipeApp` object. + +Format: `:` + +Format is similar to other systems, like uvicorn. + +Example: `datapipe --pipeline my_project.pipeline:app` will try to import module +`my_project.pipeline` and will look for object `app`, it will expect this object +to be of type `DatapipeApp`. + +### `--executor` + +Possible values: + +* `SingleThreadExecutor` +* `RayExecutor` + +TODO add separate section which describes Executor + +### `--debug`, `--debug-sql` + +`--debug` turns on debug logging in most places and shows internals of +datapipe processing. + +`--debug-sql` additionally turns on logging for all SQL queries which might be +quite verbose, but provides insight on how datapipe interacts with database. + +### `--trace-*` + +* `--trace-stdout` +* `--trace-jaeger` +* `--trace-jaeger-host HOST` +* `--trace-jaeger-port PORT` +* `--trace-gcp` + +This set of flags turns on different exporters for OpenTelemetry + +## `db` + +### `create-all` + +`datapipe db create-all` is a handy shortcut for local development. It makes +datapipe to create all known SQL tables in a configured database. + +## `lint` + +Runs checks on current state of database. Can detect and fix commong issues. + +## `run` + +## `step` + +## `table` diff --git a/docs/source/extending-cli.md b/docs/source/extending-cli.md index 2a79321b..89c60544 100644 --- a/docs/source/extending-cli.md +++ b/docs/source/extending-cli.md @@ -27,5 +27,5 @@ Plugin can expect some information in `click.Context`: ## Example -To see example of extending `datapipe` cli see -[`datapipe_app.cli`](https://github.com/epoch8/datapipe-app/blob/master/datapipe_app/cli.py) +To see example of extending `datapipe` cli see `datapipe_app.cli`: +[https://github.com/epoch8/datapipe-app/blob/master/datapipe_app/cli.py](https://github.com/epoch8/datapipe-app/blob/master/datapipe_app/cli.py) diff --git a/docs/source/index.rst b/docs/source/index.rst index 3537305a..5b9ba34c 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -3,14 +3,18 @@ .. toctree:: :caption: Introduction :maxdepth: 2 - :hidden: introduction +.. toctree:: + :caption: Command Line Interface + :maxdepth: 2 + + cli + .. toctree:: :caption: Advanced Topics :maxdepth: 2 - :hidden: transformation-lifecycle sqlite @@ -19,7 +23,6 @@ .. toctree:: :caption: Migration :maxdepth: 2 - :hidden: migration-v013-to-v014