05_17_05: Dependency Management - AudioLab Lock Files & Resolution¶

Executive Summary¶

Deterministic dependency management for AudioLab audio components. Ensures reproducible builds across all environments through lock files, version constraints, and automated resolution.

Key Features: - Lock file for reproducible builds - Dependency resolution with SAT solver - Internal dependency management (L0-L3 layers) - External dependency tracking (JUCE, Eigen, FFTW3, etc.) - Security auditing and license compliance - Dependency graph visualization - Cache management for faster builds

Problem Statement¶

Without dependency management: - ❌ Non-reproducible builds ("works on my machine") - ❌ Version conflicts between packages - ❌ Security vulnerabilities go undetected - ❌ License compliance issues - ❌ Manual dependency tracking (error-prone)

With dependency management: - ✅ Deterministic builds (same result everywhere) - ✅ Automatic conflict resolution - ✅ Vulnerability scanning - ✅ License compliance tracking - ✅ Automated lock file generation

Architecture¶

Dependency Types¶

Internal Dependencies (AudioLab components)
Kernels (L0) → Atoms (L1) → Cells (L2) → Engines (L3)
Coordinated versioning (all L0-L3 packages have same version)
Exact version constraints (=2.1.0)
External Dependencies (Third-party libraries)
JUCE: Audio plugin framework
Eigen: Linear algebra
FFTW3: Fast Fourier Transform
Catch2: Testing framework
nlohmann-json: JSON serialization
Build Tools
CMake: Build system
vcpkg: Package manager
Python: Automation scripts
Runtime Dependencies
Platform SDKs
System libraries

Lock File Format¶

audiolab-lock.yaml¶

version: "1.0"
generated: "2024-01-15T10:30:00Z"
generator: "dependency_resolver.py v2.1.0"

# Internal packages (coordinated versioning)
internal:
  kernels:
    version: "2.1.0"
    layer: "L0"
    hash: "sha256:abc123..."
    dependencies: []

  atoms:
    version: "2.1.0"
    layer: "L1"
    hash: "sha256:def456..."
    dependencies:
      - name: "kernels"
        version: "2.1.0"
        constraint: "exact"

# External packages (locked versions)
external:
  juce:
    version: "7.0.5"
    source: "github"
    url: "https://github.com/juce-framework/JUCE"
    hash: "sha256:1a2b3c4d..."
    license: "GPL-3.0"
    installed: "2024-01-15T10:25:00Z"

Key Properties: - Deterministic: Exact versions, no ranges - Verifiable: SHA256 hashes for integrity - Timestamped: Installation and generation times - Complete: All transitive dependencies included

Usage¶

Install Dependencies¶

# Install from lock file (deterministic)
python dependency_resolver.py install --locked

# Install and resolve versions (updates lock file)
python dependency_resolver.py install

Update Dependencies¶

# Update specific package
python dependency_resolver.py update juce

# Update all packages
python dependency_resolver.py update

# Updates lock file automatically after resolution

Audit Dependencies¶

# Check for vulnerabilities, license issues, outdated packages
python dependency_resolver.py audit

# Output:
# ============================================================
# Checking for known vulnerabilities...
# ✓ No known vulnerabilities
#
# Checking license compatibility...
# ✓ All licenses compatible
#
# Checking for outdated dependencies...
# ℹ️  Outdated dependencies:
#   - juce: 7.0.5 → 7.0.6
# ============================================================

Generate Lock File¶

# Generate from dependency_management.yaml
python dependency_resolver.py lock

# Output: audiolab-lock.yaml

Validate Lock File¶

# Verify lock file integrity
python dependency_resolver.py validate

# Checks:
# - Valid YAML format
# - Required keys present
# - Version format correct
# - Hash integrity

Generate Dependency Graph¶

# Create DOT file for visualization
python dependency_resolver.py graph

# Output: dependency-graph.dot

# Convert to SVG:
dot -Tsvg dependency-graph.dot -o dependency-graph.svg

Version Constraints¶

Internal Dependencies (Exact)¶

atoms:
  dependencies:
    - name: "kernels"
      version: "2.1.0"
      constraint: "exact"  # Must be =2.1.0

External Dependencies (Flexible)¶

# Exact version
juce: "=7.0.5"

# Caret (compatible updates)
eigen: "^3.4.0"  # >=3.4.0 <4.0.0

# Tilde (patch updates only)
fftw3: "~3.3.10"  # >=3.3.10 <3.4.0

# Range
catch2: ">=3.4.0 <4.0.0"

Dependency Resolution¶

Algorithm¶

Parse dependency manifest (dependency_management.yaml)
Collect all direct + transitive dependencies
Resolve version constraints using SAT solver
Verify no conflicts
Lock exact versions in lock file

Conflict Resolution¶

Strategy: Highest compatible version

Example: - Package A requires JUCE ^7.0.0 - Package B requires JUCE >=7.0.5 <8.0.0 - Resolution: JUCE 7.0.5 (satisfies both)

Fallback: Manual intervention if no compatible version exists

Security¶

Vulnerability Scanning¶

security:
  vulnerability_scanning:
    enabled: true
    sources: ["NVD", "GitHub Advisory"]
    schedule: "daily"

  audit:
    frequency: "on_install"
    severity_threshold: "medium"

Checks: - CVE database (NVD) - GitHub Security Advisory - Package-specific security issues

Response Times: - Critical: Hotfix within 24h - High: Update within 24h - Medium: Update within 1 week - Low: Update in next release

License Compliance¶

licensing:
  allowed_licenses:
    - "MIT"
    - "Apache-2.0"
    - "BSD-3-Clause"
    - "MPL-2.0"
    - "GPL-3.0"  # With commercial license option

  forbidden_licenses:
    - "AGPL"
    - "SSPL"

Audit Output:

python dependency_resolver.py audit

# Checking license compatibility...
# ⚠️  License issues:
#   - some-lib: AGPL-3.0 (incompatible)

Caching¶

Strategy¶

Content-addressed storage: {package}-{version}-{platform}-{hash}

Locations: - Global cache: ~/.audiolab/cache - Local cache: .audiolab/cache

Benefits: - Faster installs (no re-download) - Offline builds - Bandwidth savings

Retention: - Duration: 90 days - Max size: 10 GB - Auto-cleanup of oldest entries

CI/CD Integration¶

Install from Lock File¶

# .github/workflows/build.yml
- name: Install dependencies
  run: python dependency_resolver.py install --locked

- name: Cache dependencies
  uses: actions/cache@v3
  with:
    path: ~/.audiolab/cache
    key: ${{ hashFiles('audiolab-lock.yaml') }}

Verify Lock File¶

# Ensure lock file is up to date
- name: Validate lock file
  run: |
    python dependency_resolver.py validate
    git diff --exit-code audiolab-lock.yaml

Daily Security Audit¶

# .github/workflows/security-audit.yml
on:
  schedule:
    - cron: '0 9 * * *'  # Daily at 9 AM

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - run: python dependency_resolver.py audit

Workflow¶

Adding a Dependency¶

Add to manifest

# dependency_management.yaml
external_dependencies:
  new-library:
    version: "1.2.3"
    source: "vcpkg"
    license: "MIT"

Update lock file
```
python dependency_resolver.py lock
```

Validate changes

python dependency_resolver.py validate
python dependency_resolver.py audit

Commit both files

git add dependency_management.yaml audiolab-lock.yaml
git commit -m "feat(deps): add new-library 1.2.3"

Updating a Dependency¶

Update version constraint
```
juce:
  version: "7.0.6"  # Was 7.0.5
```

Resolve and test

python dependency_resolver.py update juce
# Run tests to verify compatibility

Commit changes

git add dependency_management.yaml audiolab-lock.yaml
git commit -m "chore(deps): update JUCE to 7.0.6"

Removing a Dependency¶

Remove from manifest

# Delete entry from dependency_management.yaml

Update lock file
```
python dependency_resolver.py lock
```

Verify no breakage

# Build and test to ensure nothing broke

Commit changes

git add dependency_management.yaml audiolab-lock.yaml
git commit -m "chore(deps): remove unused library"

Tools¶

dependency_resolver.py¶

Commands: - install: Install dependencies from lock file or manifest - update [package]: Update specific or all dependencies - lock: Generate lock file from manifest - validate: Verify lock file integrity - audit: Check vulnerabilities, licenses, outdated packages - graph: Generate dependency graph visualization

Flags: - --locked: Install from lock file only (fail if missing) - --no-cache: Skip cache, download fresh - --verify-hashes: Verify SHA256 checksums

Examples:

# Deterministic install (CI/CD)
python dependency_resolver.py install --locked

# Update with hash verification
python dependency_resolver.py update --verify-hashes

# Audit with detailed output
python dependency_resolver.py audit --verbose

Configuration Files¶

dependency_management.yaml (9 KB)
Dependency manifest
Version constraints
Security policies
License rules
audiolab-lock.yaml (Auto-generated)
Locked versions
SHA256 hashes
Installation metadata
DO NOT EDIT MANUALLY

Integration¶

With Version Manager¶

# Version bump updates internal dependency versions
python version_manager.py bump minor
# → Updates all L0-L3 packages to 2.2.0 in lock file

With Monorepo Manager¶

# Build order respects dependencies
python monorepo_manager.py build --all
# → Builds: kernels → atoms → cells → engines

With Release Pipeline¶

# Release validates dependencies
python release_manager.py release
# → Runs dependency audit before release

Metrics¶

Tracked automatically: - Dependency count (internal vs external) - Outdated dependencies - Vulnerabilities found - License compliance rate - Update frequency - Cache hit rate - Average resolution time

Dashboard:

Dependency Health Score: 95/100

Internal Dependencies: 4 (all coordinated)
External Dependencies: 5
Outdated: 1 (minor update available)
Vulnerabilities: 0
License Issues: 0
Cache Hit Rate: 87%

Best Practices¶

Always commit lock file with manifest changes
Use --locked in CI/CD for reproducible builds
Run audit regularly (daily in CI/CD)
Update conservatively (PATCH auto, MINOR review, MAJOR test)
Verify hashes for security-critical builds
Document breaking changes when updating major versions
Test after updates before committing lock file

Troubleshooting¶

Lock file out of date¶

# Regenerate from manifest
python dependency_resolver.py lock

Version conflict¶

# Error: Cannot resolve conflicting versions
# Solution: Relax version constraints or update packages

# View dependency graph to identify conflict
python dependency_resolver.py graph

Cache corruption¶

# Clear cache and reinstall
rm -rf ~/.audiolab/cache .audiolab/cache
python dependency_resolver.py install

Vulnerability detected¶

# Update affected package
python dependency_resolver.py update <package>

# Or temporarily accept risk (document reason)
# Add to dependency_management.yaml:
security:
  exceptions:
    - package: "some-lib"
      vulnerability: "CVE-2024-1234"
      reason: "No fix available, mitigated by sandboxing"
      expires: "2024-06-01"

Platform-Specific Notes¶

Windows¶

Uses vcpkg with x64-windows triplet
Requires Visual Studio 2022

macOS¶

Universal binaries (arm64 + x64)
Uses vcpkg with arm64-osx / x64-osx triplets
Deployment target: macOS 10.13+

Linux¶

Uses vcpkg with x64-linux triplet
Tested on Ubuntu 20.04+

Part of AudioLab Version Control System (05_17_VERSION_CONTROL)