Workflow submission scenarios

[jluethi]

While working out the web GUI interactions, I came up with a more detailed version of workflow submissions. I'm summarizing here such that we have a centralized overview of the scenarios (see also draft of new functional specs, submission section).

1. A user submits a full workflow for the first time (already supported)
2. A user reruns a workflow from the beginning (should be supported in the 1.x release, right)
3. A user runs a workflow until a given step (e.g. 4 task workflow, the user runs the first 2 => running the preprocessing steps, but not the image analysis steps yet)
4. A user continues the run of a workflow (e.g. after 3, now runs the second part of the workflow, tasks 3 & 4)
5. A user tests parameters of the workflow on a subset of the data. E.g. user ran scenario 3 (tasks 1 & 2), then runs multiple options of tasks 3 on a subset of the data. These subsets are saved to temporary files, not overwriting the main OME-Zarr file => see Running a workflow on a subset of data #109
6. A user reruns part of a workflow (e.g. reruns tasks 3 & 4). There need to be some restrictions to what can be rerun, will cover this in a separate issue.

Here is a sketch of these scenarios:

There is a main workflow (see discussion here: #236) that the user sees & where the user edits parameters. They first submit tasks 1 & 2 (workflow submission ID1). Then, the user experiments with options for task 3 (workflow submission IDs 2-4). Finally, they submit the fitting parameter on the whole dataset & run tasks 3 & 4 (workflow submission ID5)

[tcompa]

I'm dumping here some meetings note concerning the "running a workflow from task A to task B" feature.

---

Running workflow from A to B: to be more clearly defined => what do we need to clarify? User stories? Implementation approach?

User stories:

1. I’m not ready for the whole workflow yet. I just want to look at the image first (=> run 0 to n)
2. Continue a workflow: n to m
3. Something failed at step 3 (because input parameters were wrong). Let me correct parameters and rerun from there
4. Tricky rerun part: clean up old outputs? Partial failure? => not initial scope [reserved keyword argument of the task]
5. I’m just testing things (trying some parameters)

Goal: Tackle user stories 1-3

Assumptions:

Current status
Datasets are little metadata layers (collection of resources => paths + metadata)
Running a workflow has an input & output. Those are for the whole workflow => no clear access to intermediate states
Use case 3 is tricky => no info on what the last valid dataset is [some history is stored, but only at the end of the workflow]
Database is never touched from beginning to end of the workflow (separation of concern between runner & db)
Get job status is retrieved from disk (from the metadata.json, which is updated after every task [at least the history is updated, metadata dict updates are optional]) => use of this file is only for job monitoring endpoint
At the end of the workflow, db is updated

New
If a workflow fails, should the metadata be updated?
What we can do now: Update to the last valid state, i.e. to the state at the end of step 2.
Writing the last valid state

Requirement: We need to select to correct task to restart from
User needs to start from the correct dataset (the output dataset)
This will require an additional check: IO compatibility of tasks: We only check for input of workflow and output of workflow, not for each task
Tasks can modify dataset types

Q: Does the user need to define the output dataset? If the task defines (& changes) the output type, not necessarily
=> Tasks define their input/output

Runner has no access to the db => monitoring & updates goes via the metadata file

[tcompa]

Use case 6:
Switch back and forth between 3D and 2D OME-Zarrs, e.g. go back to 3D dataset after performing MIP.

Likely this could go through a more structured Dataset.meta attribute. Instead of just a list parallelization_level -> list_of_components, we could have something a bit more structured (possibly mimicking some part of the OME-Zarr structure).
This work would happen mostly on the task side, and possibly lead to some limited fractal-server updates.

The guiding principle in this should be that "Each attribute in metadata needs to exist somewhere else in the OME-Zarr file", and we would also need to rely on "Bootstrap metadata from an existing OME-Zarr file".

Other related fractal-tasks-core issues:

• Enable 2D to 3D workflows fractal-tasks-core#342
• Extend/improve copy-zarr task fractal-tasks-core#279

[jluethi]

Reviewed. Functionality of 1-4 is already implemented, 5 is indeed covered with:
fractal-analytics-platform/fractal-tasks-core#342
fractal-analytics-platform/fractal-tasks-core#279