Support multiple launchers, change steps, tweak HParams and Log (#19)

* Support multiple launchers, change steps order, tweak HParams and Logging

* Fix documentation

CHANGELOG.md

@@ -5,6 +5,24 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](https://semver.org).
 
 
+## [0.5.0] - 2021-04-30
+
+### Added
+- `NAME` support in extensions for multiple launchers.
+- Better header in `hparams`
+
+### Changed
+- Order of steps in `DefaultLauncher` is now `sweep, log, parse, run`.
+
+### Deprecated
+### Removed
+- `log_config` in logging launcher.
+
+### Fixed
+### Security
+
+
+
 ## [0.4.1] - 2021-04-28
 
 ### Added

Makefile

@@ -32,7 +32,7 @@ examples:  ## [Doc] Run all examples
 	cd docs/examples/change-parser && fromconfig config.yaml launcher.yaml - model - train
 	cd docs/examples/configure-launcher && fromconfig config.yaml --launcher.run=dry - model - train
 	cd docs/examples/configure-launcher && fromconfig config.yaml launcher_dry.yaml - model - train
-	cd docs/examples/configure-launcher && fromconfig config.yaml --logging.level=20 --logging.log_config=False - model - train
+	cd docs/examples/configure-launcher && fromconfig config.yaml --logging.level=20 - model - train
 	cd docs/examples/configure-launcher && fromconfig config.yaml launcher_logging.yaml - model - train
 	cd docs/examples/machine-learning && fromconfig trainer.yaml model.yaml optimizer.yaml params/small.yaml - trainer - run
 	cd docs/examples/machine-learning && fromconfig trainer.yaml model.yaml optimizer.yaml params/big.yaml - trainer - run

docs/development/publish-extensions.md

@@ -1,8 +1,10 @@
 # Publish Extensions <!-- {docsify-ignore} -->
 
+## Discovery
+
 Once you've implemented a custom [`Launcher`](#usage-reference/launcher/), you can share it as a `fromconfig` extension.
 
-To do so, publish a new package on `PyPI` that has a specific entry point that maps to a module defined in your package in which one `Launcher` class is defined.
+To do so, publish a new package on `PyPI` that has a specific entry point that maps to a module defined in your package in which one or more `Launcher` classes is defined.
 
 To add an entry point, update the `setup.py` by adding
 
@@ -13,7 +15,39 @@ setuptools.setup(
 )
 ```
 
-Make sure to look at the available launchers defined directly in `fromconfig`. It is recommended to keep the number of `__init__` arguments as low as possible (if any) and instead retrieve parameters from the `config` itself at run time. A good practice is to use the same name for the config entry that will be used as the shortened name given by the entry-point.
+Each `Launcher` class defined in the entry-point can define a class attribute `NAME` that uniquely identifies the launcher.
+
+For example, if the entry point name (`your_extension_name`) is `debug`, the following launcher will be available under the name `debug.print_command`.
+
+```python
+"""Custom Launcher that prints the command."""
+
+from typing import Any
+
+import fromconfig
+
+
+class PrintCommandLauncher(fromconfig.launcher.Launcher):
+
+    NAME = "print_command"
+
+    def __call__(self, config: Any, command: str = ""):
+        print(command)
+        # Launcher are nested by default
+        self.launcher(config=config, command=command)
+```
+
+
+If you don't specify the `NAME` attribute, the entry point name will be used.
+
+If your extension implements more than one launcher, you need to specify the `NAME` of each `Launcher` class (except one) otherwise there will be a name conflict.
+
+
+## Implementation
+
+Make sure to look at the available launchers defined directly in `fromconfig`.
+
+It is recommended to keep the number of `__init__` arguments as low as possible (if any) and instead retrieve parameters from the `config` itself at run time. A rule of thumb is to use `__init__` arguments only if the launcher is meant to be called multiple times with different options.
 
 If your `Launcher` class is not meant to wrap another `Launcher` class (that's the case of the `LocalLauncher` for example), make sure to override the `__init__` method like so
 
@@ -25,5 +59,3 @@ def __init__(self, launcher: Launcher = None):
 ```
 
 Once your extension is available, update `fromconfig` documentation and add an example in [extensions](extensions/).
-
-Good examples of extensions

docs/examples/configure-launcher/README.md

@@ -45,12 +45,12 @@ fromconfig config.yaml launcher_dry.yaml - model - train
 
 ## Configure Logging
 
-The logging launcher (responsible for basic logging, configured by `log: logging`) can be configured with the `logging.level` and `logging.log_config` parameters.
+The logging launcher (responsible for basic logging, configured by `log: logging`) can be configured with the `logging.level` parameter.
 
 For example,
 
 ```bash
-fromconfig config.yaml --logging.level=20 --logging.log_config=False - model - train
+fromconfig config.yaml --logging.level=20 - model - train
 ```
 
 prints

docs/examples/configure-launcher/launcher_logging.yaml

@@ -1,3 +1,2 @@
 logging:
   level: 20
-  log_config: false

docs/extensions/yarn/launcher.yaml

@@ -10,7 +10,6 @@ yarn:
 
 logging:
   level: 20
-  log_config: false
 
 launcher:
   run: yarn

docs/getting-started/cheat-sheet/launcher.yaml

@@ -6,7 +6,6 @@ hparams:
 # Configure logging level (set to logging.INFO)
 logging:
   level: 20
-  log_config: false
 
 # Configure parser (optional, using this parser is the default behavior)
 parser:

fromconfig/__init__.py

@@ -4,3 +4,4 @@
 from fromconfig import launcher
 from fromconfig import parser
 from fromconfig import utils
+from fromconfig.version import *

fromconfig/launcher/base.py

@@ -10,7 +10,7 @@
 from fromconfig.utils.libimport import from_import_string
 from fromconfig.utils.nest import merge_dict
 from fromconfig.utils.types import is_pure_iterable, is_mapping
-from fromconfig.version import MAJOR
+from fromconfig.version import __major__
 
 
 LOGGER = logging.getLogger(__name__)
@@ -137,12 +137,13 @@ def _load():
     _CLASSES["dry"] = DryLauncher
 
     # Load external classes, use entry point's name for reference
-    for entry_point in pkg_resources.iter_entry_points(f"fromconfig{MAJOR}"):
+    for entry_point in pkg_resources.iter_entry_points(f"fromconfig{__major__}"):
         module = entry_point.load()
         for _, cls in inspect.getmembers(module, lambda m: inspect.isclass(m) and issubclass(m, Launcher)):
-            if entry_point.name in _CLASSES:
-                raise ValueError(f"Duplicate launcher name found {entry_point.name} ({_CLASSES})")
-            _CLASSES[entry_point.name] = cls
+            name = f"{entry_point.name}.{cls.NAME}" if hasattr(cls, "NAME") else entry_point.name
+            if name in _CLASSES:
+                raise ValueError(f"Duplicate launcher name found {name} ({_CLASSES})")
+            _CLASSES[name] = cls
 
     # Log loaded classes
     LOGGER.info(f"Loaded Launcher classes {_CLASSES}")

fromconfig/launcher/default.py

@@ -12,7 +12,14 @@
 
 
 # Special keys for a Launcher config split by steps with defaults
-_STEPS = OrderedDict([("sweep", "hparams"), ("parse", "parser"), ("log", "logging"), ("run", "local")])
+_STEPS = OrderedDict(
+    [
+        ("sweep", "hparams"),  # Hyper Parameter Sweep
+        ("log", "logging"),  # Configure Logging
+        ("parse", "parser"),  # Parse config
+        ("run", "local"),  # Actually run the config
+    ]
+)
 
 
 class DefaultLauncher(base.Launcher):

fromconfig/launcher/hparams.py

@@ -3,6 +3,7 @@
 from typing import Any
 import itertools
 import logging
+import shutil
 
 from fromconfig.core.base import fromconfig
 from fromconfig.launcher import base
@@ -40,7 +41,24 @@ def __call__(self, config: Any, command: str = ""):
                 names = hparams.keys()
                 for values in itertools.product(*[hparams[name] for name in names]):
                     overrides = dict(zip(names, values))
-                    LOGGER.info("Launching with params")
-                    for key, value in overrides.items():
-                        LOGGER.info(f"- {key}: {value}")
+                    print(header(overrides))
                     self.launcher(config=merge_dict(config, {"hparams": overrides}), command=command)
+
+
+def header(overrides) -> str:
+    """Create header for experiment."""
+    # Get terminal number of columns
+    try:
+        columns, _ = shutil.get_terminal_size((80, 20))
+    except Exception:  # pylint: disable=broad-except
+        columns = 80
+
+    # Join key-values and truncate if needed
+    content = ", ".join(f"{key}={value}" for key, value in overrides.items())
+    if len(content) >= columns - 2:
+        content = content[: columns - 2 - 3] + "." * 3
+    content = f"[{content}]"
+
+    # Add padding
+    padding = "=" * max((columns - len(content)) // 2, 0)
+    return f"{padding}{content}{padding}"

fromconfig/launcher/logger.py

@@ -4,7 +4,6 @@
 import logging
 
 from fromconfig.launcher import base
-from fromconfig.utils.nest import flatten
 from fromconfig.utils.types import is_mapping
 
 
@@ -22,18 +21,10 @@ def __call__(self, config: Any, command: str = ""):
         # Extract params
         params = (config.get("logging") or {}) if is_mapping(config) else {}  # type: ignore
         level = params.get("level", None)
-        log_config = params.get("log_config", False)
 
         # Change verbosity level (applies to all loggers)
         if level is not None:
             logging.basicConfig(level=level)
 
-        # Log flattened config
-        if log_config:
-            for key, value in flatten(config):
-                if key in ("logging.level", "logging.log_config"):
-                    continue
-                LOGGER.info(f"- {key}: {value}")
-
         # Execute sub-launcher with no parser (already parsed)
         self.launcher(config=config, command=command)  # type: ignore

fromconfig/launcher/parser.py

@@ -22,7 +22,7 @@ def __call__(self, config: Any, command: str = ""):
             parser = fromconfig(config.pop("parser")) if "parser" in config else DefaultParser()
         else:
             parser = DefaultParser()
-        LOGGER.info(f"Resolved parser {parser}")
+        LOGGER.debug(f"Resolved parser {parser}")
 
         # Launch
         self.launcher(config=parser(config) if callable(parser) else config, command=command)  # type: ignore

fromconfig/version.py

@@ -1,9 +1,8 @@
 # pylint: disable=all
 
-__version__ = "0.4.1"
+__version__ = "0.5.0"
 __author__ = "Criteo"
 
-
-MAJOR = __version__.split(".")[0]
-MINOR = __version__.split(".")[1]
-PATCH = __version__.split(".")[2]
+__major__ = __version__.split(".")[0]
+__minor__ = __version__.split(".")[1]
+__patch__ = __version__.split(".")[2]

tests/unit/launcher/test_launcher_logger.py

@@ -13,7 +13,6 @@
         pytest.param({"foo": "bar"}, ({}, []), id="default"),
         pytest.param(None, ({}, []), id="none"),
         pytest.param({"foo": "bar", "logging": {"level": 20}}, ({"level": 20}, []), id="set-level"),
-        pytest.param({"foo": "bar", "logging": {"log_config": True}}, ({}, ["- foo: bar"]), id="log-config"),
     ],
 )
 def test_launcher_logger(config, expected, monkeypatch):