Configuration & Tracking

Pydantic settings, environment overrides, and experiment tracking with WandB and MLflow.

Introduction

Feature Forge uses pydantic-settings with a layered configuration system. This notebook shows how settings are loaded, overridden, and validated — plus how to use the built-in experiment trackers.

Setup

import os
import warnings
warnings.filterwarnings("ignore")

import pandas as pd

print("Configuration & Tracking Demo")

Configuration & Tracking Demo

Default Settings

from feature_forge.config import Settings

settings = Settings()
print(f"Task: {settings.task}")
print(f"Metric: {settings.metric}")
print(f"Rounds: {settings.n_rounds}")
print(f"Model: {settings.llm.model}")
print(f"Router strategy: {settings.router.strategy}")
print(f"CV folds: {settings.evaluation.cv_folds}")

Task: classification
Metric: auc
Rounds: 4
Model: deepseek-chat
Router strategy: hybrid
CV folds: 5

Constructor Overrides

from feature_forge.config import LLMConfig, RouterConfig

custom = Settings(
    task="regression",
    metric="rmse",
    n_rounds=2,
    llm=LLMConfig(model="gpt-4", temperature=0.5),
    router=RouterConfig(strategy="llm"),
)
print(f"Custom task: {custom.task}")
print(f"Custom metric: {custom.metric}")
print(f"Custom temp: {custom.llm.temperature}")
print(f"Custom router: {custom.router.strategy}")

Custom task: regression
Custom metric: rmse
Custom temp: 0.5
Custom router: llm

Environment Variable Overrides

# Simulate setting env vars
os.environ["FF_TASK"] = "classification"
os.environ["FF_N_ROUNDS"] = "3"
os.environ["FF_LLM__MODEL"] = "deepseek-chat"

env_settings = Settings()
print(f"Task from env: {env_settings.task}")
print(f"Rounds from env: {env_settings.n_rounds}")
print(f"Model from env: {env_settings.llm.model}")

Task from env: classification
Rounds from env: 3
Model from env: deepseek-chat

Validation

# Invalid temperature
try:
    bad = Settings(llm=LLMConfig(temperature=3.0))
except Exception as exc:
    print(f"Temp validation caught: {exc}")

try:
    bad = Settings(router=RouterConfig(min_agents=0))
except Exception as exc:
    print(f"Min agents validation caught: {exc}")

Temp validation caught: 1 validation error for LLMConfig
temperature
  Value error, temperature must be in [0, 2], got 3.0 [type=value_error, input_value=3.0, input_type=float]
    For further information visit https://errors.pydantic.dev/2.12/v/value_error
Min agents validation caught: 1 validation error for RouterConfig
min_agents
  Value error, min_agents must be >= 1, got 0 [type=value_error, input_value=0, input_type=int]
    For further information visit https://errors.pydantic.dev/2.12/v/value_error

Settings from YAML

from pathlib import Path

yaml_path = Path("config/settings.yaml")
if yaml_path.exists():
    content = yaml_path.read_text()
    print(f"settings.yaml exists ({len(content)} chars)")
    print("Preview:")
    print("\n".join(content.splitlines()[:15]))
else:
    print("settings.yaml not found")

settings.yaml not found

Dataset Registry

from feature_forge.data import DatasetRegistry

registry = DatasetRegistry()
print(f"Available datasets: {registry.list()}")

for name in registry.list():
    info = registry.info(name)
    print(f"  {name}: {info.get('task', 'unknown')} — {info.get('source', 'unknown')}")

Available datasets: ['titanic', 'house_prices']
  titanic: classification — kaggle
  house_prices: regression — kaggle

Experiment Tracker

from feature_forge.experiment import ExperimentTracker, NoOpTracker

# NoOp tracker (default when no backend configured)
noop = NoOpTracker(project="test")
noop.init_run(run_name="test", config={"x": 1})
noop.log_metrics({"auc": 0.85})
noop.finish()
print("NoOp tracker: OK (no external calls)")

NoOp tracker: OK (no external calls)

WandB Tracker

from feature_forge.experiment import WandBTracker

try:
    wandb_tracker = WandBTracker(
        project="feature-forge-notebooks",
    )
    wandb_tracker.init_run(run_name="config_demo", config={
        "task": "classification",
        "model": "deepseek-chat",
    })
    wandb_tracker.log_metrics({"baseline_auc": 0.74, "enhanced_auc": 0.79})
    wandb_tracker.finish()
    print("WandB tracking: OK")
except Exception as exc:
    print(f"WandB skipped: {exc}")

wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /Users/minghao/.netrc.

wandb: W&B API key is configured. Use `wandb login --relogin` to force relogin

wandb: WARNING Using a boolean value for 'reinit' is deprecated. Use 'return_previous' or 'finish_previous' instead.

wandb: setting up run b2n6w997

WandB skipped: user is not logged in

MLflow Tracker

from feature_forge.experiment import MLflowTracker

try:
    mlflow_tracker = MLflowTracker(project="feature_forge_config")
    mlflow_tracker.init_run(run_name="config_demo", config={
        "task": "classification",
        "model": "deepseek-chat",
    })
    mlflow_tracker.log_metrics({"baseline_auc": 0.74, "enhanced_auc": 0.79})
    mlflow_tracker.finish()
    print("MLflow tracking: OK")
except Exception as exc:
    print(f"MLflow skipped: {exc}")

MLflow skipped: mlflow not installed. Run: uv pip install mlflow

Plot: Config Hierarchy

import matplotlib.pyplot as plt

hierarchy = {
    "Constructor args": 100,
    "Environment vars": 80,
    ".env file": 60,
    "YAML files": 40,
    "Defaults": 20,
}

fig, ax = plt.subplots(figsize=(7, 4))
ax.barh(list(hierarchy.keys()), list(hierarchy.values()), color="darkslateblue")
ax.set_title("Configuration Priority (Highest → Lowest)")
ax.set_xlabel("Priority")
ax.invert_yaxis()
plt.tight_layout()
plt.show()

Summary

• Configuration priority: constructor args > env vars > .env > YAML > defaults
• Settings uses pydantic-settings with FF_* prefixed env var mapping
• Validation happens at instantiation (temperature range, min_agents ≥ 1, etc.)
• DatasetRegistry manages Kaggle and local sample datasets
• ExperimentTracker abstracts WandB and MLflow with a shared interface
• NoOpTracker lets you run experiments without any external dependencies