The config file is the experiment

By November 2019 my masters evaluation platform had grown to four algorithms (3DMV, MinkowskiEngine at two voxel sizes, SemanticFusion), two datasets (ScanNet, NYUv2), four kinds of assessment, and enough hyperparameters per algorithm that a single SemanticFusion run needed 25 of them. Some runs took hours on a GPU I was sharing with three other people. Argparse does not scale to that, and re-typing forty flags by hand every time I wanted to swap one voxel size is how you lose an evening to a typo you don’t notice until the run is half finished. The fix, one YAML file per experiment, is the whole subject of this post, and the repo’s own README.md says why better than I can retell it:

Three sentences, and they commit to three separate engineering decisions: the config is the experiment record, parsing happens in one pass before anything runs, and runtime failures are contained to the thing that failed. The rest of this post is how segtester/configs/base.py actually implements the first two, and how the platform’s call sites implement the third.

The workflow

assets/eval_platform_design.pdf is a five-box draw.io flowchart. Redrawn:

Everything left of that last accent-coloured box is RunnableConfig.parse_from_yaml_file running once, all the way through, before a single algorithm executes. Everything inside it is what “skip any scenes that cause errors” turns into in real code, which I’ll get to after the parser.

The parser primitives

segtester/configs/base.py builds one small vocabulary of “members” that every config class is written in terms of. A config class is a plain Python object whose __init__ assigns each field to one of these, instead of reading it straight from a dict:

# segtester/configs/runnablebase.py:7-15
class RunnableConfig(BCNF.ConfigParser):

    def __init__(self):
        super().__init__()
        self.name: str = BCNF.OptionalMember()
        self.description: str = BCNF.OptionalMember()
        self.base_result_path: str = BCNF.RequiredMember()
        self.tests: List[Callable] = BCNF.RequiredMember(BCNF.IterableMember(BCNF.MappableMember(RUNNABLE_MAP)))

That’s the entire “Base Config” box from the schema diagram, and it already shows the two axes the members vary on: required or optional, and plain scalar or nested parser. OptionalMember and RequiredMember are the same class with one difference: RequiredMember just checks the key exists before delegating to its parent’s logic:

# segtester/configs/base.py:5-27
class OptionalMember:
    def __init__(self, parsable_object=None, default_ret=None):
        parse_op = getattr(parsable_object, "_parse_args", None)
        assert (parsable_object is None or callable(parse_op)), f"provided object: {parsable_object} is not parsable"
        self.parsable_object = parsable_object
        self.default_ret = default_ret

    def _parse_args(self, key: str, trace: list, parent=None, **kwargs):
        try:
            logger.debug(f"Parsing member {' -> '.join(trace)}: {key}...")
            if key not in kwargs:
                return self.default_ret, True
            val = kwargs[key]
            if self.parsable_object is not None:
                return self.parsable_object()._parse_args(key, trace, parent, **kwargs)
            return val, True
        except Exception as e:
            logger.error(f"Unknown error occurred while parsing trace:\n{' -> '.join(trace)}")
            logger.error(str(e))
            return None, False


class RequiredMember(OptionalMember):
    def __init__(self, parsable_object=None):
        super().__init__(parsable_object)

    def _parse_args(self, key: str, trace: list, parent=None, **kwargs):
        try:
            if key not in kwargs:
                logger.error(f"Could not find required parameter: {key} with trace:\n{' -> '.join(trace)}")
                return None, False
            return super()._parse_args(key, trace, parent, **kwargs)
        except Exception as e:
            logger.error(f"Unknown error occurred while parsing trace:\n{' -> '.join(trace)}")
            logger.error(str(e))
            return None, False

The load-bearing detail is the return type: (value, success), always, never an exception escaping a normal validation failure. OptionalMember with no key present just returns its default and True, which is how name and description are allowed to be absent from every config in configs/*.yaml without a warning. RequiredMember with no key present logs the specific field name and returns False. Neither raises.

IterableMember applies another member to every item of a list, ANDing every item’s success into one flag for the list:

# segtester/configs/base.py:58-90 (trimmed)
class IterableMember:
    def _parse_args(self, key: str, trace: list, parent=None, **kwargs):
        conf_list = kwargs.get(key, [])
        ret = []
        success = True
        try:
            for i, item in enumerate(conf_list):
                key_it = str(i)
                if self.parsable_object is not None:
                    item, succ_it = self.parsable_object()._parse_args(key_it, trace + [key_it], parent, **{key_it: item})
                    success = success and succ_it
                ret.append(item)
            return ret, success
        except TypeError:
            logger.error(f"Expected iterable object:\n{' -> '.join(trace)}")
            return None, False

And MappableMember is the one that makes a single tests: list able to hold five completely different shapes of object (an ExecuteAlg, a Segmentation3d, an OdometryAssessment and so on, each with its own required fields), by branching on an id: key:

# segtester/configs/base.py:93-119 (trimmed)
class MappableMember:
    def __init__(self, type_map: dict):
        self.type_map = type_map

    def _parse_args(self, key: str, trace: list, parent=None, **kwargs):
        val = kwargs[key]
        if not isinstance(val, dict):
            logger.error(f"Expected a dictionary for {key}, trace: {' -> '.join(trace)}")
            return None, False
        if val.get('id') not in self.type_map:
            logger.error(f"Expected id to be one of: {set(self.type_map.keys())}\n -> trace: {' -> '.join(trace)}")
            return None, False
        return self.type_map[val.get('id')]()._parse_args(key, trace, parent, **kwargs)

Stack those three together: RequiredMember(IterableMember(MappableMember(RUNNABLE_MAP))), and you get “tests: must exist, must be a list, and every item in it must be a dict whose id: names one of seven registered runnable types, each validated against its own schema.” One line encodes all of that. configs/runnable/__init__.py builds the map the same way the other two dispatch tables (algorithms, datasets) are built: a plain dict literal, populated by each config class registering itself:

# segtester/configs/runnable/__init__.py:10-18
RUNNABLE_MAP = {}
ExecuteAlgConfig.register_type(RUNNABLE_MAP)
ViewDatasetConfig.register_type(RUNNABLE_MAP)
VisualisePredictionsConfig.register_type(RUNNABLE_MAP)

OdometryConfig.register_type(RUNNABLE_MAP)
Segmentation3d.register_type(RUNNABLE_MAP)
Segmentation2dReproj.register_type(RUNNABLE_MAP)
SummarizeRes.register_type(RUNNABLE_MAP)

There’s a fifth member class, ConstantMember (base.py:47-55): it always returns its constructor value and success=False, which looks deliberate (maybe meant to force a warning wherever it’s used?), but I never actually used it anywhere in the codebase. It’s dead code, and a fun one to notice while transcribing this: a class that exists in the schema diagram and is never instantiated anywhere in the repository.

Fail loudly at parse time, quietly at run time

Every member _parse_args returns (value, success). ConfigParser._parse_args (the method every config class shares by inheriting from ConfigParser) walks its own __dict__, calls _parse_args on every member, and ANDs every one of those flags together:

# segtester/configs/base.py:136-165
def _parse_args(self, key, trace: list, parent=None, **kwargs):
    try:
        curr_config = kwargs[key]
        if not isinstance(curr_config, dict):
            logger.error(f"Expected a dictionary for {key}, trace: {' -> '.join(trace)}")
            return None, False

        kwarg_keys = set(curr_config.keys())
        parse_success = True
        for key, value in self.__dict__.items():
            parse_op = getattr(value, "_parse_args", None)
            if not callable(parse_op):
                logger.warn(f"Parameter: {key} with trace:\n{' -> '.join(trace)} is not parsable... Skipping!")
                continue
            kwarg_keys.discard(key)
            res_val, success = value._parse_args(key, trace + [key], self, **curr_config)
            parse_success = parse_success and success
            setattr(self, key, res_val)

        if len(kwarg_keys) > 0:
            logger.warn(f"Skipping unknown keys: {kwarg_keys}\n -> in config file: {' -> '.join(trace)}")
        self.parent_config = parent
        return self, parse_success
    except Exception as e:
        logger.error(f"Unknown error occurred while parsing trace:\n{' -> '.join(trace)}")
        logger.error(str(e))
        return None, False

Nothing in that loop returns early on a failure: a missing base_result_path doesn’t stop the loop from also checking tests, and a bad id: three levels deep in the third test doesn’t stop the fourth test from being checked too. The flag just keeps getting ANDed down. The only place this ever turns into an exception is right at the top level, once, after everything has been walked:

# segtester/configs/base.py:167-173
def parse_from_config(self, init_trace: str, **kwargs):
    trace = []
    if init_trace is not None:
        trace += [init_trace]
    _, success = self._parse_args(init_trace, trace, **{init_trace: kwargs})
    assert success, "Invalid configurations detected please look at the logs."
    return self

One assert, at the very end, after the logger has already printed every error found anywhere in the tree. That’s “fail loudly at parse time,” loudly meaning all at once, not meaning immediately. Compare that to what happens once parsing has actually succeeded and the platform starts running: RunnableConfig.__call__ wraps each runnable’s call in its own try/except, and a broken one just gets logged and skipped so the rest of the list still runs:

# segtester/configs/runnablebase.py:17-26
def __call__(self, *args, **kwargs):
    logger.info("Starting Evaluation")
    for i, test in enumerate(self.tests):
        logger.info(f"Running test {i}/{len(self.tests)}")
        logger.info(SEP)
        try:
            test(base_result_path=self.format_string_with_meta(self.base_result_path))
        except Exception as e:
            logger.error(f"An unknown error has occured. Skipping test {i}")
            logger.error(f"{e}")

That’s “quiet at run time”: an unhandled exception here doesn’t crash the process, it just costs you that one runnable. The same pattern repeats one level down, per scene, inside every assessment and algorithm driver: segtester/assessments/segmentation3d.py wraps its whole per-scene body in a try, and on any exception, logs it and moves to the next scene instead of aborting the run:

# segtester/assessments/segmentation3d.py:134-139
except Exception as e:
    curr_gt_scene_id = None  # reload gt scene incase there is an error
    logger.error(f"Exception when performing 3d seg assessment on {est_dataset_conf.id}:{scene.id}. "
                 f"Skipping scene and moving on...")
    logger.exception(e)

Put together: config errors are checked exhaustively and stop the run before it starts; runtime errors are caught narrowly and only cost you the one runnable or the one scene that hit them. Two failure modes, two deliberately different philosophies, and the code draws the line between them at exactly the moment a multi-hour GPU job would otherwise be one bad scene away from losing everything after it.

The schema, and what each runnable actually needs

assets/eval_platform_design_part_2.pdf draws the dispatch: one Base Config, five runnable shapes hanging off tests:. Redrawn as a diagram of the relationship: the full field lists are transcribed as tables underneath, not squeezed into SVG text:

gt_dataset/dataset dispatch again, into DATASET_MAP; est_dataset/result_dataset are always a fixed ResultsConfig (file_map, dataset_id, load_path, save_path), not dispatched, because a result is always read the same shape regardless of which algorithm produced it. assets/eval_platform_design_part_3.pdf draws that dataset side:

The algorithm side (assets/eval_platform_design_part_4.pdf) is the one place the PDF’s field lists are genuinely long: SemanticFusion alone carries 17 fields for the CNN and CRF plus 16 more for the nested ElasticFusion SLAM backend it wraps. A diagram with that much text in it is the report’s text, cropped, so here it’s a table instead:

Every default above is quoted from segtester/configs/alg/{MV3DConfig,MinkowskiEngineConfig, SemanticFusionConfig}.py, and matches eval_platform_design_part_4.pdf field-for-field: that PDF has no defaults, only names and types, since defaults live only in code.

SafeDict: a path template that formats itself in stages

Every save_path default above has unresolved {placeholders} in it: {dataset_id}, {scene_id}, {alg_name}. None of those are known when the config is parsed: they only exist once a dataloader has listed real scenes and an algorithm has actually run on one. So the platform needs to format the same template string more than once, with more information available each time, and not crash on the placeholders it can’t fill yet. That’s SafeDict: three lines wrapping dict.__missing__:

# segtester/configs/base.py:122-124
class SafeDict(dict):
    def __missing__(self, key):
        return '{' + key + '}'

str.format_map calls __missing__ for any key it doesn’t find, instead of raising KeyError. Make the fallback dict return the literal '{key}' for a key it doesn’t have, and "{a}/{b}".format_map(SafeDict(a="x")) becomes "x/{b}", partially formatted and still a valid template for the next format call. ConfigParser wraps this once for config-level metadata:

# segtester/configs/base.py:203-212
@staticmethod
def safe_format_string(string_template: str, **kwargs):
    return string_template.format_map(SafeDict(**kwargs))

def format_string_with_meta(self, string_template: str, **kwargs):
    return self.safe_format_string(string_template, **{
        **ConfigParser.GLOBAL_META,
        **self.meta,
        **kwargs
    })

GLOBAL_META is just {"timestamp": datetime.now().strftime(...)} (base.py:128-130), computed once at import time. self.meta is whatever the meta: dict in the YAML file holds, empty in every committed config, but the mechanism is there for a reader to stash their own values and reference them from base_result_path. That first stage runs exactly once, in RunnableConfig.__call__, before any runnable executes (runnablebase.py:23, quoted above): base_result_path gets {timestamp} and meta filled in, nothing else. That’s why configs/test2.yaml can write base_result_path: "/mnt/.../RESULTS/{timestamp}" and have every run land in its own directory without any extra code.

The second stage happens inside each assessment, once per scene, with the keys that are only known there:

# segtester/assessments/segmentation3d.py:62-65
save_path = self.conf.format_string_with_meta(f"{base_result_path}/{self.conf.save_path}", **{
    "dataset_id": gt_dataset_conf.id, "scene_id": scene.id,
    "alg_name": scene.alg_name,
})

Two calls, two different sets of kwargs, the same template mechanism, and a placeholder that neither call supplies just survives unresolved. The playground’s “resolved path templates” panel reproduces exactly this: it fills in {timestamp} and meta: the way RunnableConfig.__call__ does, then {dataset_id} and {alg_name} where the config already names them, and leaves {scene_id} (and {voxel_size}, {alignment_opt}, {pose_relation}) literal, because (this is the platform’s real hard blocker, not this widget’s) no dataset CSV, .ply, poses.npz or probs.npz is committed to this repository, so there is no real scene list anywhere to resolve them against.

skip_existing: the output path is the completion marker

The third README sentence, “the platform will skip any test that causes an error during runtime,” is the try/except per scene, already quoted above. But there’s a second, quieter form of “skip” that isn’t about errors at all:

# segtester/assessments/segmentation3d.py:67-71
if self.conf.skip_existing and os.path.exists(f"{save_path}"):
    logger.warn(f"When getting results for 3d segmentation of {est_dataset_conf.id}->"
                f"{scene.id}->{scene.alg_name}, "
                f"found existing path {save_path}.\n Skipping this scene...")
    continue

If the scene’s already-formatted output directory exists, and skip_existing is true, skip it: no flag file, no database row, no separate manifest of “what’s done.” The resolved save_path is both where the result gets written and the question the platform asks itself on the next run to decide whether to redo the work. Kill a multi-hour job at scene 40 of 200, fix whatever crashed it, and re-run the exact same config: scenes 1–39 already have a directory on disk, so they’re skipped in milliseconds, and the job picks up close to where it stopped. No separate checkpointing system, because the filesystem the results already live on is the checkpoint.

The same six lines, with the same field name, are duplicated seven times across the codebase: segmentation3d.py, segmentation2d_reproj.py, segmentation2d_reproj_copy.py (and a third near-identical _no_infil.py variant), mv3d.py, minkowski.py, semanticfusion.py and visualise_labels.py, each with its own self.conf.skip_existing and its own os.path.exists(...) check, none of them sharing a helper. It works, and it’s not DRY, the kind of duplication that’s obvious in hindsight and easy to have not noticed while adding the seventh call site under deadline. Segmentation2dReproj’s default is False rather than the True every algorithm and Segmentation3d use, presumably because reprojection is fast enough that re-running it isn’t the same cost as redoing a Minkowski forward pass, but that’s my guess reading it four years later, not something the code or a comment says.

Two worked examples

configs/run_algs_scannet_val.yaml: three ExecuteAlg runnables sharing one tests: list, one per algorithm, each dispatching to a different ALG_MAP member and the same SCANNET dataset:

# configs/run_algs_scannet_val.yaml
---
name: Run scannet validation dataset
description: Run all the algorithms on the validation part of scannet
base_result_path: "/mnt/1C562D12562CEDE8/RESULTS/ActualResults/CheckingMEOutputs"
tests:
  - id: ExecuteAlg
    alg:
      id: MinkowskiEngine
      weights_path: "/data/algs/minkowski_runner/weights.pth"
    dataset:
      id: "SCANNET"
      file_map: "/data/datasets/scenenn_val_dataset.csv"
  - id: ExecuteAlg
    alg:
      id: 3DMV
      model_path: "/mnt/1C562D12562CEDE8/MODELS/3DMV/scannetv2/scannet5_model.pth"
      model2d_orig_path: "/mnt/1C562D12562CEDE8/MODELS/3DMV/scannetv2/2d_scannet.pth"
      process_nth_frame: 5
      num_nearest_images: 4
      voxel_size: 0.048
    dataset:
      id: "SCANNET"
      file_map: "/data/datasets/scenenn_val_dataset.csv"
  - id: ExecuteAlg
    alg:
      id: SemanticFusion
      prototext_model_path: "/mnt/1C562D12562CEDE8/MODELS/semanticFusion/nyu_rgbd/inference.prototxt"
      caffemodel_path: "/mnt/1C562D12562CEDE8/MODELS/semanticFusion/nyu_rgbd/inference.caffemodel"
      class_colour_lookup_path: "/mnt/1C562D12562CEDE8/COMMONDEPS/semanticfusion2/class_colour_scheme.data"
      caffe_use_cpu: true
      cnn_skip_frames: 10
    dataset:
      id: "SCANNET"
      file_map: "/data/datasets/scenenn_val_dataset.csv"
meta: {}

configs/run_3d_seg_assessment.yaml: a single Segmentation3d runnable, with skip_existing: True written out explicitly even though it’s already the default, and a save_path override that starts with ../ (it writes one directory above base_result_path, which the template mechanism happily allows: SafeDict doesn’t validate paths, only fills placeholders):

# configs/run_3d_seg_assessment.yaml
---
name: Base Test
description: This test serves as an example to how a config file may look
base_result_path: "/data/results/ActualResults/20191008"
tests:
- id: Segmentation3d
  save_path: "../eigen13id/{dataset_id}/{scene_id}/{alg_name}/seg/seg3d"
  est_dataset:
    file_map: "/data/results/ActualResults/scannet_results_conf.csv"
    dataset_id: "SCANNET"
  gt_dataset:
    id: "SCANNET"
    file_map: "/data/datasets/scenenn_val_dataset.csv"
  label_map:
    csv_path: "/data/results/label_map.csv"
  label_map_dest_col: "eigen13id"
  skip_existing: True
meta: {}

Both are quoted verbatim, personal filesystem paths included: that really is what’s committed. Nothing in this post runs against them; they’re static YAML.

main.py’s get_config_file() is how these actually got picked at the terminal: glob configs/*.yaml and user_configs/*.yaml, list ten at a time, type a number:

Config playground

A YAML textarea, a validation panel that runs the same JS port of base.py as the tables above, a run graph that redraws from the same result, and a panel that partially formats save_path the way SafeDict would. Five presets are the real files from configs/*.yaml: pick one, edit it, or reach for Break it to inject a missing required field, a typo’d extra key, or a bad id:, and watch every resulting error and warning get listed in one pass rather than stopping at the first.

tests:
  - id: ExecuteAlg
    alg:
      id: MinkowskiEngine
      weights_path: "...weights.pth"
    dataset:
      id: "SCANNET"
      file_map: "...scenenn_val_dataset.csv"

Everything the widget validates against is either quoted from configs/*.yaml verbatim (the five presets) or transcribed field-for-field from the Python source cited throughout this post (the schema). Nothing it shows is a simulated dataset, algorithm or result: there’s no .ply, poses.npz or probs.npz committed anywhere in the repository for it to load even if it wanted to.

What this pattern is actually for

None of this is specific to 3D vision. RequiredMember/OptionalMember/IterableMember/ MappableMember is a general recipe for “validate a whole tree, collect every problem, fail once at the end”: the kind of thing that’s worth reaching for whenever a config error costs more than a few seconds to discover, which for me meant a benchmark that took long enough to run that finding error #2 by re-running after fixing error #1 wasn’t a five-second loop, it was losing another slot in a shared GPU queue. SafeDict solves a narrower but equally general problem (formatting the same template at different points with different information available at each), and skip_existing solves the oldest problem in long-running jobs by refusing to invent a new mechanism for it: the answer to “is this done” was already sitting on disk.

I didn’t write a report to go with this platform, and by December the repository had grown around a “run it, look at the numbers, adjust, run again” workflow rather than a “write up what happened” one, which is its own kind of honesty about what a masters project’s code actually optimises for under a deadline. The parser held up regardless; every config file this post quotes still parses today, four years after I stopped touching this codebase, because the schema was never implicit in a script: it was always written down, in three small classes, right here.