Merge remote-tracking branch 'origin/master' into benc-python3.13

docs/devguide/roadmap.rst

@@ -15,7 +15,7 @@ Code Maintenance
 * **Type Annotations and Static Type Checking**: Add static type annotations throughout the codebase and add typeguard checks.
 * **Release Process**: `Improve the overall release process <https://github.com/Parsl/parsl/issues?q=is%3Aopen+is%3Aissue+label%3Arelease_process>`_ to synchronize docs and code releases, automatically produce changelog documentation.
 * **Components Maturity Model**: Defines the `component maturity model <https://github.com/Parsl/parsl/issues/2554>`_ and tags components with their appropriate maturity level.
-* **Define and Document Interfaces**: Identify and document interfaces via which `external components <https://parsl.readthedocs.io/en/stable/userguide/plugins.html>`_ can augment the Parsl ecosystem.
+* **Define and Document Interfaces**: Identify and document interfaces via which `external components <https://parsl.readthedocs.io/en/stable/userguide/advanced/plugins.html>`_ can augment the Parsl ecosystem.
 * **Distributed Testing Process**: All tests should be run against all possible schedulers, using different executors, on a variety of remote systems. Explore the use of containerized schedulers and remote testing on real systems.
 
 New Features and Integrations

docs/index.rst

@@ -4,7 +4,7 @@ Parsl - Parallel Scripting Library
 Parsl extends parallelism in Python beyond a single computer.
 
 You can use Parsl
-`just like Python's parallel executors <userguide/workflow.html#parallel-workflows-with-loops>`_
+`just like Python's parallel executors <userguide/workflows/workflow.html#parallel-workflows-with-loops>`_
 but across *multiple cores and nodes*.
 However, the real power of Parsl is in expressing multi-step workflows of functions.
 Parsl lets you chain functions together and will launch each function as inputs and computing resources are available.
@@ -37,8 +37,8 @@ Parsl lets you chain functions together and will launch each function as inputs
 
 
 Start with the `configuration quickstart <quickstart.html#getting-started>`_ to learn how to tell Parsl how to use your computing resource,
-see if `a template configuration for your supercomputer <userguide/configuring.html>`_ is already available,
-then explore the `parallel computing patterns <userguide/workflow.html>`_ to determine how to use parallelism best in your application.
+see if `a template configuration for your supercomputer <userguide/configuration/examples.html>`_ is already available,
+then explore the `parallel computing patterns <userguide/workflows/workflows.html>`_ to determine how to use parallelism best in your application.
 
 Parsl is an open-source code, and available on GitHub: https://github.com/parsl/parsl/
 
@@ -57,7 +57,7 @@ Parsl works everywhere
 
 *Parsl can run parallel functions on a laptop and the world's fastest supercomputers.*
 Scaling from laptop to supercomputer is often as simple as changing the resource configuration.
-Parsl is tested `on many of the top supercomputers <userguide/configuring.html>`_.
+Parsl is tested `on many of the top supercomputers <userguide/configuration/examples.html>`_.
 
 Parsl is flexible
 -----------------

docs/quickstart.rst

@@ -70,7 +70,7 @@ We describe these components briefly here, and link to more details in the `User
 
 .. note::
 
-    Parsl's documentation includes `templates for many supercomputers <userguide/configuring.html>`_.
+    Parsl's documentation includes `templates for many supercomputers <userguide/configuring/examples.html>`_.
     Even though you may not need to write a configuration from a blank slate,
     understanding the basic terminology below will be very useful.
 
@@ -112,7 +112,7 @@ with hello world Python and Bash apps.
     with open('hello-stdout', 'r') as f:
         print(f.read())
 
-Learn more about the types of Apps and their options `here <userguide/apps.html>`__.
+Learn more about the types of Apps and their options `here <userguide/apps/index.html>`__.
 
 Executors
 ^^^^^^^^^
@@ -127,7 +127,7 @@ You can dynamically set the number of workers based on available memory and
 pin each worker to specific GPUs or CPU cores
 among other powerful features.
 
-Learn more about Executors `here <userguide/execution.html#executors>`__.
+Learn more about Executors `here <userguide/configuration/execution.html#executors>`__.
 
 Execution Providers
 ^^^^^^^^^^^^^^^^^^^
@@ -141,7 +141,7 @@ Another key role of Providers is defining how to start an Executor on a remote c
 Often, this simply involves specifying the correct Python environment and
 (described below) how to launch the Executor on each acquired computers.
 
-Learn more about Providers `here <userguide/execution.html#execution-providers>`__.
+Learn more about Providers `here <userguide/configuration/execution.html#execution-providers>`__.
 
 Launchers
 ^^^^^^^^^
@@ -151,7 +151,7 @@ A common example is an :class:`~parsl.launchers.launchers.MPILauncher`, which us
 for starting a single program on multiple computing nodes.
 Like Providers, Parsl comes packaged with Launchers for most supercomputers and clouds.
 
-Learn more about Launchers `here <userguide/execution.html#launchers>`__.
+Learn more about Launchers `here <userguide/configuration/execution.html#launchers>`__.
 
 
 Benefits of a Data-Flow Kernel
@@ -164,7 +164,7 @@ and performs the many other functions needed to execute complex workflows.
 The flexibility and performance of the DFK enables applications with
 intricate dependencies between tasks to execute on thousands of parallel workers.
 
-Start with the Tutorial or the `parallel patterns <userguide/workflow.html>`_
+Start with the Tutorial or the `parallel patterns <userguide/workflows/workflow.html>`_
 to see the complex types of workflows you can make with Parsl.
 
 Starting Parsl
@@ -210,7 +210,7 @@ An example which launches 4 workers on 1 node of the Polaris supercomputer looks
     )
 
 
-The documentation has examples for other supercomputers `here <userguide/configuring.html>`__.
+The documentation has examples for other supercomputers `here <userguide/configuration/examples.html>`_.
 
 The next step is to load the configuration
 

docs/userguide/advanced/index.rst

@@ -0,0 +1,13 @@
+Advanced Topics
+===============
+
+More to learn about Parsl after starting a project.
+
+.. toctree::
+   :maxdepth: 2
+
+   modularizing
+   usage_tracking
+   monitoring
+   parsl_perf
+   plugins

docs/userguide/advanced/modularizing.rst

@@ -0,0 +1,109 @@
+.. _codebases:
+
+Structuring Parsl programs
+--------------------------
+
+While convenient to build simple Parsl programs as a single Python file,
+splitting a Parsl programs into multiple files and a Python module
+has significant benefits, including:
+
+   1. Better readability
+   2. Logical separation of components (e.g., apps, config, and control logic)
+   3. Ease of reuse of components
+
+Large applications that use Parsl often divide into several core components:
+
+.. contents::
+   :local:
+   :depth: 2
+
+The following sections use an example where each component is in a separate file:
+
+.. code-block::
+
+    examples/logic.py
+    examples/app.py
+    examples/config.py
+    examples/__init__.py
+    run.py
+    pyproject.toml
+
+Run the application by first installing the Python library and then executing the "run.py" script.
+
+.. code-block:: bash
+
+    pip install .  # Install module so it can be imported by workers
+    python run.py
+
+
+Core application logic
+======================
+
+The core application logic should be developed without any deference to Parsl.
+Implement capabilities, write unit tests, and prepare documentation
+in which ever way works best for the problem at hand.
+
+Parallelization with Parsl will be easy if the software already follows best practices.
+
+The example defines a function to convert a single integer into binary.
+
+.. literalinclude:: examples/library/logic.py
+   :caption: library/logic.py
+
+Workflow functions
+==================
+
+Tasks within a workflow may require unique combinations of core functions.
+Functions to be run in parallel must also meet :ref:`specific requirements <function-rules>`
+that may complicate writing the core logic effectively.
+As such, separating functions to be used as Apps is often beneficial.
+
+The example includes a function to convert many integers into binary.
+
+Key points to note:
+
+- It is not necessary to have import statements inside the function.
+  Parsl will serialize this function by reference, as described in :ref:`functions-from-modules`.
+
+- The function is not yet marked as a Parsl PythonApp.
+  Keeping Parsl out of the function definitions simplifies testing
+  because you will not need to run Parsl when testing the code.
+
+- *Advanced*: Consider including Parsl decorators in the library if using complex workflow patterns,
+  such as :ref:`join apps <label-joinapp>` or functions which take :ref:`special arguments <special-kwargs>`.
+
+.. literalinclude:: examples/library/app.py
+   :caption: library/app.py
+
+
+Parsl configuration functions
+=============================
+
+Create Parsl configurations specific to your application needs as functions.
+While not necessary, including the Parsl configuration functions inside the module
+ensures they can be imported into other scripts easily.
+
+Generating Parsl :class:`~parsl.config.Config` objects from a function
+makes it possible to change the configuration without editing the module.
+
+The example function provides a configuration suited for a single node.
+
+.. literalinclude:: examples/library/config.py
+   :caption: library/config.py
+
+Orchestration Scripts
+=====================
+
+The last file defines the workflow itself.
+
+Such orchestration scripts, at minimum, perform at least four tasks:
+
+1. *Load execution options* using a tool like :mod:`argparse`.
+2. *Prepare workflow functions for execution* by creating :class:`~parsl.app.python.PythonApp` wrappers over each function.
+3. *Create configuration then start Parsl* with the :meth:`parsl.load` function.
+4. *Launch tasks and retrieve results* depending on the needs of the application.
+
+An example run script is as follows
+
+.. literalinclude:: examples/run.py
+   :caption: run.py

docs/userguide/advanced/monitoring.rst

@@ -0,0 +1,121 @@
+Monitoring
+==========
+
+Parsl includes a monitoring system to capture task state as well as resource
+usage over time. The Parsl monitoring system aims to provide detailed
+information and diagnostic capabilities to help track the state of your
+programs, down to the individual apps that are executed on remote machines.
+
+The monitoring system records information to an SQLite database while a
+workflow runs. This information can then be visualised in a web dashboard
+using the ``parsl-visualize`` tool, or queried using SQL using regular
+SQLite tools.
+
+
+Monitoring configuration
+------------------------
+
+Parsl monitoring is only supported with the `parsl.executors.HighThroughputExecutor`. 
+
+The following example shows how to enable monitoring in the Parsl
+configuration. Here the `parsl.monitoring.MonitoringHub` is specified to use port
+55055 to receive monitoring messages from workers every 10 seconds.
+
+.. code-block:: python
+
+   import parsl
+   from parsl.monitoring.monitoring import MonitoringHub
+   from parsl.config import Config
+   from parsl.executors import HighThroughputExecutor
+   from parsl.addresses import address_by_hostname
+
+   import logging
+
+   config = Config(
+      executors=[
+          HighThroughputExecutor(
+              label="local_htex",
+              cores_per_worker=1,
+              max_workers_per_node=4,
+              address=address_by_hostname(),
+          )
+      ],
+      monitoring=MonitoringHub(
+          hub_address=address_by_hostname(),
+          monitoring_debug=False,
+          resource_monitoring_interval=10,
+      ),
+      strategy='none'
+   )
+
+
+Visualization
+-------------
+
+To run the web dashboard utility ``parsl-visualize`` you first need to install
+its dependencies:
+
+   $ pip install 'parsl[monitoring,visualization]'
+
+To view the web dashboard while or after a Parsl program has executed, run
+the ``parsl-visualize`` utility::
+
+   $ parsl-visualize
+
+By default, this command expects that the default ``monitoring.db`` database is used
+in the runinfo directory. Other databases can be loaded by passing
+the database URI on the command line.  For example, if the full path
+to the database is ``/tmp/my_monitoring.db``, run::
+
+   $ parsl-visualize sqlite:////tmp/my_monitoring.db
+
+By default, the visualization web server listens on ``127.0.0.1:8080``. If the web server is deployed on a machine with a web browser, the dashboard can be accessed in the browser at ``127.0.0.1:8080``. If the web server is deployed on a remote machine, such as the login node of a cluster, you will need to use an ssh tunnel from your local machine to the cluster::
+
+   $ ssh -L 50000:127.0.0.1:8080 username@cluster_address
+
+This command will bind your local machine's port 50000 to the remote cluster's port 8080.
+The dashboard can then be accessed via the local machine's browser at ``127.0.0.1:50000``. 
+
+.. warning:: Alternatively you can deploy the visualization server on a public interface. However, first check that this is allowed by the cluster's security policy. The following example shows how to deploy the web server on a public port (i.e., open to Internet via ``public_IP:55555``)::
+
+   $ parsl-visualize --listen 0.0.0.0 --port 55555
+
+
+Workflows Page
+^^^^^^^^^^^^^^
+
+The workflows page lists all Parsl workflows that have been executed with monitoring enabled
+with the selected database.
+It provides a high level summary of workflow state as shown below:
+
+.. image:: ../../images/mon_workflows_page.png
+
+Throughout the dashboard, all blue elements are clickable. For example, clicking a specific worklow
+name from the table takes you to the Workflow Summary page described in the next section.
+
+Workflow Summary
+^^^^^^^^^^^^^^^^
+
+The workflow summary page captures the run level details of a workflow, including start and end times
+as well as task summary statistics. The workflow summary section is followed by the *App Summary* that lists
+the various apps and invocation count for each. 
+
+.. image:: ../../images/mon_workflow_summary.png
+
+
+The workflow summary also presents three different views of the workflow:
+
+* Workflow DAG - with apps differentiated by colors: This visualization is useful to visually inspect the dependency
+  structure of the workflow. Hovering over the nodes in the DAG shows a tooltip for the app represented by the node and it's task ID.
+
+.. image:: ../../images/mon_task_app_grouping.png
+
+* Workflow DAG - with task states differentiated by colors: This visualization is useful to identify what tasks have been completed, failed, or are currently pending.
+
+.. image:: ../../images/mon_task_state_grouping.png
+
+* Workflow resource usage: This visualization provides resource usage information at the workflow level.
+  For example, cumulative CPU/Memory utilization across workers over time.
+
+.. image:: ../../images/mon_resource_summary.png
+