Tracking file provenance

.gitignore

@@ -121,3 +121,13 @@ ENV/
 
 # emacs buffers
 \#*
+
+runinfo*
+parsl/tests/.pytest*
+
+# documentation generation
+docs/stubs/*
+docs/1-parsl-introduction.ipynb
+
+/tmp
+parsl/data_provider/dyn.new.py

docs/userguide/data.rst

@@ -3,7 +3,7 @@
 Passing Python objects
 ======================
 
-Parsl apps can communicate via standard Python function parameter passing 
+Parsl apps can communicate via standard Python function parameter passing
 and return statements. The following example shows how a Python string
 can be passed to, and returned from, a Parsl app.
 
@@ -12,31 +12,31 @@ can be passed to, and returned from, a Parsl app.
     @python_app
     def example(name):
         return 'hello {0}'.format(name)
-	
+
     r = example('bob')
     print(r.result())
 
-Parsl uses the dill and pickle libraries to serialize Python objects 
+Parsl uses the dill and pickle libraries to serialize Python objects
 into a sequence of bytes that can be passed over a network from the submitting
 machine to executing workers.
 
-Thus, Parsl apps can receive and return standard Python data types 
+Thus, Parsl apps can receive and return standard Python data types
 such as booleans, integers, tuples, lists, and dictionaries. However, not
-all objects can be serialized with these methods (e.g., closures, generators, 
+all objects can be serialized with these methods (e.g., closures, generators,
 and system objects), and so those objects cannot be used with all executors.
 
-Parsl will raise a `SerializationError` if it encounters an object that it cannot 
-serialize. This applies to objects passed as arguments to an app, as well as objects 
+Parsl will raise a `SerializationError` if it encounters an object that it cannot
+serialize. This applies to objects passed as arguments to an app, as well as objects
 returned from an app. See :ref:`label_serialization_error`.
 
 
 Staging data files
 ==================
 
 Parsl apps can take and return data files. A file may be passed as an input
-argument to an app, or returned from an app after execution. Parsl 
-provides support to automatically transfer (stage) files between 
-the main Parsl program, worker nodes, and external data storage systems. 
+argument to an app, or returned from an app after execution. Parsl
+provides support to automatically transfer (stage) files between
+the main Parsl program, worker nodes, and external data storage systems.
 
 Input files can be passed as regular arguments, or a list of them may be
 specified in the special ``inputs`` keyword argument to an app invocation.
@@ -69,13 +69,13 @@ interface.
 Parsl files
 -----------
 
-Parsl uses a custom :py:class:`~parsl.data_provider.files.File` to provide a 
-location-independent way of referencing and accessing files.  
-Parsl files are defined by specifying the URL *scheme* and a path to the file. 
+Parsl uses a custom :py:class:`~parsl.data_provider.files.File` to provide a
+location-independent way of referencing and accessing files.
+Parsl files are defined by specifying the URL *scheme* and a path to the file.
 Thus a file may represent an absolute path on the submit-side file system
 or a URL to an external file.
 
-The scheme defines the protocol via which the file may be accessed. 
+The scheme defines the protocol via which the file may be accessed.
 Parsl supports the following schemes: file, ftp, http, https, and globus.
 If no scheme is specified Parsl will default to the file scheme.
 
@@ -89,8 +89,8 @@ README file.
     File('https://github.com/Parsl/parsl/blob/master/README.rst')
 
 
-Parsl automatically translates the file's location relative to the 
-environment in which it is accessed (e.g., the Parsl program or an app). 
+Parsl automatically translates the file's location relative to the
+environment in which it is accessed (e.g., the Parsl program or an app).
 The following example shows how a file can be accessed in the app
 irrespective of where that app executes.
 
@@ -113,22 +113,23 @@ As described below, the method by which this files are transferred
 depends on the scheme and the staging providers specified in the Parsl
 configuration.
 
+
 Staging providers
 -----------------
 
-Parsl is able to transparently stage files between at-rest locations and 
+Parsl is able to transparently stage files between at-rest locations and
 execution locations by specifying a list of
-:py:class:`~parsl.data_provider.staging.Staging` instances for an executor. 
+:py:class:`~parsl.data_provider.staging.Staging` instances for an executor.
 These staging instances define how to transfer files in and out of an execution
 location. This list should be supplied as the ``storage_access``
-parameter to an executor when it is constructed. 
+parameter to an executor when it is constructed.
 
-Parsl includes several staging providers for moving files using the 
+Parsl includes several staging providers for moving files using the
 schemes defined above. By default, Parsl executors are created with
-three common staging providers: 
+three common staging providers:
 the NoOpFileStaging provider for local and shared file systems
 and the HTTP(S) and FTP staging providers for transferring
-files to and from remote storage locations. The following 
+files to and from remote storage locations. The following
 example shows how to explicitly set the default staging providers.
 
 .. code-block:: python
@@ -146,12 +147,12 @@ example shows how to explicitly set the default staging providers.
             )
         ]
     )
-				
-		
-Parsl further differentiates when staging occurs relative to 
-the app invocation that requires or produces files. 
+
+
+Parsl further differentiates when staging occurs relative to
+the app invocation that requires or produces files.
 Staging either occurs with the executing task (*in-task staging*)
-or as a separate task (*separate task staging*) before app execution.  
+or as a separate task (*separate task staging*) before app execution.
 In-task staging
 uses a wrapper that is executed around the Parsl task and thus
 occurs on the resource on which the task is executed. Separate
@@ -167,9 +168,9 @@ NoOpFileStaging for Local/Shared File Systems
 The NoOpFileStaging provider assumes that files specified either
 with a path or with the ``file`` URL scheme are available both
 on the submit and execution side. This occurs, for example, when there is a
-shared file system. In this case, files will not moved, and the 
+shared file system. In this case, files will not moved, and the
 File object simply presents the same file path to the Parsl program
-and any executing tasks. 
+and any executing tasks.
 
 Files defined as follows will be handled by the NoOpFileStaging provider.
 
@@ -207,14 +208,14 @@ will be executed as a separate
 Parsl task that will complete before the corresponding app
 executes. These providers cannot be used to stage out output files.
 
-The following example defines a file accessible on a remote FTP server. 
+The following example defines a file accessible on a remote FTP server.
 
 .. code-block:: python
 
     File('ftp://www.iana.org/pub/mirror/rirstats/arin/ARIN-STATS-FORMAT-CHANGE.txt')
 
 When such a file object is passed as an input to an app, Parsl will download the file to whatever location is selected for the app to execute.
-The following example illustrates how the remote file is implicitly downloaded from an FTP server and then converted. Note that the app does not need to know the location of the downloaded file on the remote computer, as Parsl abstracts this translation. 
+The following example illustrates how the remote file is implicitly downloaded from an FTP server and then converted. Note that the app does not need to know the location of the downloaded file on the remote computer, as Parsl abstracts this translation.
 
 .. code-block:: python
 
@@ -234,17 +235,17 @@ The following example illustrates how the remote file is implicitly downloaded f
     # call the convert app with the Parsl file
     f = convert(inputs=[inp], outputs=[out])
     f.result()
-		
-HTTP and FTP separate task staging providers can be configured as follows. 
+
+HTTP and FTP separate task staging providers can be configured as follows.
 
 .. code-block:: python
 
     from parsl.config import Config
     from parsl.executors import HighThroughputExecutor
     from parsl.data_provider.http import HTTPSeparateTaskStaging
     from parsl.data_provider.ftp import FTPSeparateTaskStaging
-    
-		config = Config(
+
+    config = Config(
         executors=[
             HighThroughputExecutor(
                 storage_access=[HTTPSeparateTaskStaging(), FTPSeparateTaskStaging()]
@@ -263,10 +264,10 @@ task staging providers described above, but will do so in a wrapper around
 individual app invocations, which guarantees that they will stage files to
 a file system visible to the app.
 
-A downside of this staging approach is that the staging tasks are less visible 
+A downside of this staging approach is that the staging tasks are less visible
 to Parsl, as they are not performed as separate Parsl tasks.
 
-In-task staging providers can be configured as follows. 
+In-task staging providers can be configured as follows.
 
 .. code-block:: python
 
@@ -345,16 +346,16 @@ In some cases, for example when using a Globus `shared endpoint <https://www.glo
                 )
             ]
         )
-        
+
 
 Globus Authorization
 """"""""""""""""""""
 
-In order to transfer files with Globus, the user must first authenticate. 
-The first time that Globus is used with Parsl on a computer, the program 
+In order to transfer files with Globus, the user must first authenticate.
+The first time that Globus is used with Parsl on a computer, the program
 will prompt the user to follow an authentication and authorization
 procedure involving a web browser. Users can authorize out of band by
-running the parsl-globus-auth utility. This is useful, for example, 
+running the parsl-globus-auth utility. This is useful, for example,
 when running a Parsl program in a batch system where it will be unattended.
 
 .. code-block:: bash
@@ -370,7 +371,7 @@ rsync
 
 The ``rsync`` utility can be used to transfer files in the ``file`` scheme in configurations where
 workers cannot access the submit-side file system directly, such as when executing
-on an AWS EC2 instance or on a cluster without a shared file system. 
+on an AWS EC2 instance or on a cluster without a shared file system.
 However, the submit-side file system must be exposed using rsync.
 
 rsync Configuration
@@ -399,13 +400,13 @@ and public IP address of the submitting system.
 rsync Authorization
 """""""""""""""""""
 
-The rsync staging provider delegates all authentication and authorization to the 
-underlying ``rsync`` command. This command must be correctly authorized to connect back to 
-the submit-side system. The form of this authorization will depend on the systems in 
+The rsync staging provider delegates all authentication and authorization to the
+underlying ``rsync`` command. This command must be correctly authorized to connect back to
+the submit-side system. The form of this authorization will depend on the systems in
 question.
 
-The following example installs an ssh key from the submit-side file system and turns off host key 
-checking, in the ``worker_init`` initialization of an EC2 instance. The ssh key must have 
+The following example installs an ssh key from the submit-side file system and turns off host key
+checking, in the ``worker_init`` initialization of an EC2 instance. The ssh key must have
 sufficient privileges to run ``rsync`` over ssh on the submit-side system.
 
 .. code-block:: python

docs/userguide/monitoring.rst

@@ -15,7 +15,7 @@ SQLite tools.
 Monitoring configuration
 ------------------------
 
-Parsl monitoring is only supported with the `parsl.executors.HighThroughputExecutor`. 
+Parsl monitoring is only supported with the `parsl.executors.HighThroughputExecutor`.
 
 The following example shows how to enable monitoring in the Parsl
 configuration. Here the `parsl.monitoring.MonitoringHub` is specified to use port
@@ -50,6 +50,58 @@ configuration. Here the `parsl.monitoring.MonitoringHub` is specified to use por
    )
 
 
+File Provenance
+---------------
+
+The monitoring system can also be used to track file provenance. File provenance is defined as the history of a file including:
+
+* When the files was created
+* File size in bytes
+* File md5sum
+* What task created the file
+* What task(s) used the file
+* What inputs were given to the task that created the file
+* What environment was used (e.g. the 'worker_init' entry from a :py:class:`~parsl.providers.ExecutionProvider`), not available with every provider.
+
+The purpose of the file provenance tracking is to provide a mechanism where the user can see exactly how a file was created and used in a workflow. This can be useful for debugging, understanding the workflow, for ensuring that the workflow is reproducible, and reviewing past work. The file provenance information is stored in the monitoring database and can be accessed using the ``parsl-visualize`` tool. To enable file provenance tracking, set the ``capture_file_provenance`` flag to ``True`` in the `parsl.monitoring.MonitoringHub` configuration.
+
+This functionality also enables you to log informational messages from you scripts, to capture anything not automatically gathered. The main change to your code to use this functionality is to assign the return value of the ``parsl.load`` to a variable. Then use the ``log_info`` function to log the messages in the database. Note that this feature is only available in the main script, not inside apps, unless you pass the variable (``my_cfg`` in the example below), as an argument to the app. The following example shows how to use this feature.
+
+.. code-block:: python
+
+   import parsl
+   from parsl.monitoring.monitoring import MonitoringHub
+   from parsl.config import Config
+   from parsl.executors import HighThroughputExecutor
+   from parsl.addresses import address_by_hostname
+
+   import logging
+
+   config = Config(
+      executors=[
+          HighThroughputExecutor(
+              label="local_htex",
+              cores_per_worker=1,
+              max_workers_per_node=4,
+              address=address_by_hostname(),
+          )
+      ],
+      monitoring=MonitoringHub(
+          hub_address=address_by_hostname(),
+          hub_port=55055,
+          monitoring_debug=False,
+          resource_monitoring_interval=10,
+          capture_file_provenance=True,
+      ),
+      strategy='none'
+   )
+
+   my_cfg = parsl.load(config)
+
+   my_cfg.log_info("This is an informational message")
+
+Known limitations: The file provenance feature will capture the creation of files and the use of files in an app, but currently does not capture the modification of files it already knows about.
+
 Visualization
 -------------
 
@@ -75,7 +127,7 @@ By default, the visualization web server listens on ``127.0.0.1:8080``. If the w
    $ ssh -L 50000:127.0.0.1:8080 username@cluster_address
 
 This command will bind your local machine's port 50000 to the remote cluster's port 8080.
-The dashboard can then be accessed via the local machine's browser at ``127.0.0.1:50000``. 
+The dashboard can then be accessed via the local machine's browser at ``127.0.0.1:50000``.
 
 .. warning:: Alternatively you can deploy the visualization server on a public interface. However, first check that this is allowed by the cluster's security policy. The following example shows how to deploy the web server on a public port (i.e., open to Internet via ``public_IP:55555``)::
 
@@ -99,12 +151,12 @@ Workflow Summary
 
 The workflow summary page captures the run level details of a workflow, including start and end times
 as well as task summary statistics. The workflow summary section is followed by the *App Summary* that lists
-the various apps and invocation count for each. 
+the various apps and invocation count for each.
 
 .. image:: ../images/mon_workflow_summary.png
 
 
-The workflow summary also presents three different views of the workflow:
+The workflow summary also presents three or four different views of the workflow (the number depends on whether file provenance is enabled and files were used in the workflow):
 
 * Workflow DAG - with apps differentiated by colors: This visualization is useful to visually inspect the dependency
   structure of the workflow. Hovering over the nodes in the DAG shows a tooltip for the app represented by the node and it's task ID.
@@ -120,3 +172,35 @@ The workflow summary also presents three different views of the workflow:
 
 .. image:: ../images/mon_resource_summary.png
 
+* Workflow file provenance (only if enabled and files were used in the workflow): This visualization gives a tabular listing of each task that created (output) or used (input) a file. Each listed file has a link to a page detailing the file's information.
+
+.. image:: ../images/mon_workflow_files.png
+
+File Provenance
+^^^^^^^^^^^^^^^
+
+The file provenance page provides an interface for searching for files and viewing their provenance. The % wildcard can be used in the search bar to match any number of characters. Any results are listed in a table below the search bar. Clicking on a file in the table will take you to the file's detail page.
+
+.. image:: ../images/mon_file_provenance.png
+
+File Details
+^^^^^^^^^^^^
+
+The file details page provides information about a specific file, including the file's name, size, md5sum, and the tasks that created and used the file. Clicking on any of the tasks will take you to their respective details page. If the file was created by a task there will be an entry for the Environment used by that task. Clicking that link will take you to the Environment Details page.
+
+.. image:: ../images/mon_file_detail.png
+
+
+Task Details
+^^^^^^^^^^^^
+
+The task details page provides information about a specifiic instantiation of a task. This information includes task dependencies, executor (environment), input and output files, and task arguments.
+
+.. image:: ../images/mon_task_detail.png
+
+Environment Details
+^^^^^^^^^^^^^^^^^^^
+
+The environment details page provides information on the compute environment a task was run including the provider and launcher used and the worker_init that was used.
+
+.. image:: ../images/mon_env_detail.png