🧪 AudioLab Testing Framework (03_04)¶

Comprehensive testing infrastructure for real-time audio processing

📋 Table of Contents¶

Overview
Quick Start
Learning Paths
Module Structure
Running Tests
Writing Tests
Resources

🎯 Overview¶

The AudioLab Testing Framework provides complete testing infrastructure for audio DSP:

Unit Testing: Catch2-based tests with audio-specific assertions
Integration Testing: Full plugin/processor validation
Performance Testing: Benchmarks with regression detection
Quality Testing: Golden file comparison, listening tests
Real-Time Safety: Allocation/lock detection in audio callbacks

Key Features: - ⚡ Real-time aware: Detects allocations, locks, syscalls in audio thread - 🎯 Audio-specific assertions: Epsilon comparisons, dB tolerance, denormal detection - 📊 Performance regression: Automatic detection of slowdowns - 🎧 Perceptual testing: Golden file comparison for audio quality - 🔬 Test utilities: Signals, fixtures, mocks for audio testing

🚀 Quick Start¶

1. Prerequisites¶

# Ensure Catch2 is installed via vcpkg
vcpkg install catch2:x64-windows

# Verify CMake is available
cmake --version  # Need 3.20+

2. Build Tests¶

# Windows
cd "2 - FOUNDATION\03_INFRA\03_04_testing_framework"
cmake -S . -B build
cmake --build build --config Debug

# macOS/Linux
cd "2 - FOUNDATION/03_INFRA/03_04_testing_framework"
cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug
cmake --build build

3. Run All Tests¶

# Windows
.\scripts\run_tests.ps1

# macOS/Linux
./scripts/run_tests.sh

4. Run Specific Test¶

# Windows
.\build\tests\Debug\test_example_processor.exe

# macOS/Linux
./build/tests/test_example_processor

Expected output:

✅ All tests passed (42 assertions in 8 test cases)

Total time: 5-10 minutes

For detailed walkthrough, see QUICK_START.md

📚 Learning Paths¶

🌱 Path 1: Test Runner (Beginner)¶

Goal: Run existing tests and understand output

Time: 1-2 hours

QUICK_START.md (30 min)
Build and run tests
Understand test output
Use test filters
TEST_CHEAT_SHEET.md (15 min)
Common test commands
CTest reference
Debugging failed tests
03_04_05_troubleshooting/FAILURE_DIAGNOSIS.md (30 min)
Diagnose test failures
Handle flaky tests
Debug techniques

🌿 Path 2: Test Writer (Intermediate)¶

Goal: Write your own audio tests

Time: 4-6 hours

Complete Path 1 first
03_04_00_test_frameworks/TEST_FRAMEWORK_PHILOSOPHY.md (2 hours)
Why audio testing is different
Test pyramid strategy
Best practices and anti-patterns
03_04_01_test_organization/TEST_CATEGORIES.md (30 min)
Unit vs integration vs performance tests
Test organization patterns
03_04_02_test_utilities/TEST_UTILITIES_GUIDE.md (1 hour)
Use audio test helpers
Fixtures and mocks
Golden file comparison
examples/ (1 hour)
Study example tests
Adapt patterns to your code
Write first test

🌲 Path 3: Testing Architect (Advanced)¶

Goal: Design comprehensive test strategies

Time: 8-10 hours

Complete Paths 1 & 2 first
03_04_03_performance_testing/PERFORMANCE_CRITERIA.md (2 hours)
Performance targets
Benchmark design
Regression detection
03_04_03_performance_testing/PROFILING_SETUP.md (2 hours)
Profiling tools (VTune, Instruments, perf)
Hotspot identification
Optimization workflow
03_04_04_quality_testing/AUDIO_QUALITY_GUIDE.md (2 hours)
Golden file strategies
Listening test protocols
Plugin validation
03_04_04_quality_testing/GOLDEN_FILES_MANAGEMENT.md (1 hour)
Managing reference audio
Update workflows
Git LFS integration
03_04_05_troubleshooting/FLAKY_TEST_PATTERNS.md (1 hour)
Identify flaky tests
Fix non-determinism
Race condition debugging

🗂️ Module Structure¶

03_04_testing_framework/
│
├── 📄 README.md                    ← You are here
├── 📄 QUICK_START.md               ← 5-minute tutorial
├── 📄 TEST_CHEAT_SHEET.md          ← Command reference
├── 📄 CMakeLists.txt               ← Build configuration
│
├── 📁 03_04_00_test_frameworks/    ← Philosophy & frameworks
│   └── TEST_FRAMEWORK_PHILOSOPHY.md (1,156 lines - ESSENTIAL READ!)
│
├── 📁 03_04_01_test_organization/  ← Test structure
│   ├── COVERAGE_REQUIREMENTS.md
│   ├── DIRECTORY_STRUCTURE.md
│   ├── TEST_CATEGORIES.md
│   ├── TEST_NAMING_CONVENTION.md
│   └── test_structure.cmake
│
├── 📁 03_04_02_test_utilities/     ← Test helpers (9 headers + 1 cpp)
│   ├── TEST_UTILITIES_GUIDE.md
│   ├── audio_test_helpers.hpp      ← Audio-specific assertions
│   ├── audiolab_test_utils.hpp     ← Main utilities header
│   ├── complete_test_helpers.hpp
│   ├── fixture_base.hpp            ← Base test fixtures
│   ├── golden_file_manager.hpp     ← Golden file comparison
│   ├── mock_factories.hpp          ← Mock objects
│   ├── test_data_generator.hpp     ← Test signal generation
│   └── test_signals.cpp
│
├── 📁 03_04_03_performance_testing/ ← Benchmarks & profiling
│   ├── PERFORMANCE_CRITERIA.md
│   ├── PROFILING_SETUP.md
│   ├── benchmark_config.hpp
│   ├── regression_detection.ps1    ← Auto-detect slowdowns
│   └── regression_detection.sh
│
├── 📁 03_04_04_quality_testing/     ← Audio quality validation
│   ├── AUDIO_QUALITY_GUIDE.md
│   ├── GOLDEN_FILES_MANAGEMENT.md
│   ├── LISTENING_TEST_PROTOCOL.md
│   └── PLUGIN_VALIDATION.md
│
├── 📁 03_04_05_troubleshooting/     ← Debug failed tests
│   ├── FAILURE_DIAGNOSIS.md
│   ├── FLAKY_TEST_PATTERNS.md
│   └── run_test_repeatedly.ps1     ← Find flaky tests
│
├── 📁 scripts/                      ← Test runners
│   ├── README.md                    ← Script documentation
│   ├── run_tests.ps1                ← Windows test runner
│   └── run_tests.sh                 ← Unix/macOS test runner
│
├── 📁 examples/                     ← Example tests
│   ├── README.md
│   ├── example_unit_test.cpp
│   ├── example_integration_test.cpp
│   ├── example_benchmark.cpp
│   └── example_golden_file_test.cpp
│
├── 📁 tests/                        ← Actual test suite
│   ├── CMakeLists.txt
│   ├── test_example_processor.cpp   (16KB - comprehensive example)
│   ├── test_infrastructure.cpp
│   ├── test_l4_framework.cpp
│   ├── test_l5_framework.cpp
│   └── test_plugin_interfaces.cpp
│
└── 📁 golden_files/                 ← Reference audio files
    └── (Various .wav files for comparison)

🏃 Running Tests¶

Quick Commands¶

# Run all tests
./scripts/run_tests.ps1                    # Windows
./scripts/run_tests.sh                     # macOS/Linux

# Run specific test
./build/tests/Debug/test_example_processor.exe
./build/tests/test_example_processor

# Run tests matching pattern
ctest -R "unit_*"                          # Only unit tests
ctest -R "benchmark_*"                     # Only benchmarks

# Run with verbose output
ctest --verbose

# Run with failure details
ctest --output-on-failure

# Repeat flaky test 100 times
./03_04_05_troubleshooting/run_test_repeatedly.ps1 -TestName "test_example_processor" -Iterations 100

Using CTest Directly¶

# From build directory
cd build

# List all tests
ctest -N

# Run with specific config
ctest -C Debug
ctest -C Release

# Run in parallel (4 jobs)
ctest -j 4

# Run only failed tests
ctest --rerun-failed

# Generate XML report (for CI)
ctest --output-junit test_results.xml

Using Test Filters (Catch2)¶

# Run only unit tests
./test_example_processor "[unit]"

# Run only integration tests
./test_example_processor "[integration]"

# Run only benchmarks
./test_example_processor "[benchmark]"

# Run specific test case
./test_example_processor "SimpleProcessor processes audio correctly"

# List all tests
./test_example_processor --list-tests

# List all tags
./test_example_processor --list-tags

For complete reference, see TEST_CHEAT_SHEET.md

✍️ Writing Tests¶

Anatomy of an Audio Test¶

#include <catch2/catch_test_macros.hpp>
#include "audio_test_helpers.hpp"

TEST_CASE("Gain applies correctly", "[unit][gain]") {
    // Arrange: Set up test data
    std::vector<float> input = generate_sine_wave(440.0f, 48000.0f, 1024);
    std::vector<float> output(1024);
    float gain = 0.5f;

    // Act: Run the code under test
    apply_gain(input.data(), output.data(), gain, 1024);

    // Assert: Verify results
    for (size_t i = 0; i < 1024; ++i) {
        REQUIRE_THAT(output[i],
            WithinRel(input[i] * gain, 0.0001f));  // Audio epsilon
    }
}

Using Test Utilities¶

#include "audiolab_test_utils.hpp"

TEST_CASE("Filter frequency response", "[unit][filter]") {
    // Use fixture for common setup
    AudioTestFixture fixture(48000.0f);

    // Generate test signal
    auto sine = fixture.generate_sine(1000.0f, 1024);

    // Create filter
    LowPassFilter filter(48000.0f, 500.0f);

    // Process
    auto output = filter.process(sine);

    // Verify with audio-specific tolerance
    REQUIRE(fixture.measure_rms(output) < 0.1f);  // Attenuated
}

Performance Testing¶

#include <catch2/benchmark/catch_benchmark.hpp>

TEST_CASE("FFT performance", "[benchmark]") {
    std::vector<float> input(4096);
    std::vector<std::complex<float>> output(2049);

    BENCHMARK("FFT 4096 samples") {
        return fft_forward(input.data(), output.data(), 4096);
    };

    // Verify meets real-time constraint
    // At 48kHz, 512 samples = 10.67ms budget
}

Golden File Testing¶

#include "golden_file_manager.hpp"

TEST_CASE("Reverb output matches golden", "[quality]") {
    GoldenFileManager golden("../golden_files");

    // Load input
    auto input = golden.load_wav("reverb_input.wav");

    // Process
    Reverb reverb;
    auto output = reverb.process(input);

    // Compare with reference (0.01 = 1% tolerance)
    REQUIRE(golden.compare_wav("reverb_output.wav", output, 0.01));
}

For complete guide, see 03_04_02_test_utilities/TEST_UTILITIES_GUIDE.md

📊 Test Categories¶

AudioLab uses 5 test categories (marked with Catch2 tags):

Category	Tag	Purpose	Frequency
Unit	`[unit]`	Test single functions/classes	Every commit
Integration	`[integration]`	Test component interactions	Every commit
Performance	`[benchmark]`	Measure execution time	Daily (CI)
Quality	`[quality]`	Audio quality validation	Pre-release
RT-Safety	`[rt-safety]`	Real-time constraint checks	Every commit

Running by Category¶

# Fast feedback (< 1 minute)
ctest -R "unit_*"

# Comprehensive (5-10 minutes)
ctest -R "unit_*|integration_*"

# Full suite (15-30 minutes)
ctest

# Performance only (for optimization work)
./test_example_processor "[benchmark]"

# Real-time safety only (critical for audio thread)
./test_example_processor "[rt-safety]"

🔬 Test Utilities Reference¶

Key Headers¶

Header	Purpose	Key Functions
audio_test_helpers.hpp	Audio-specific assertions	`REQUIRE_AUDIO_EQUAL()`, `CHECK_NO_DENORMALS()`
fixture_base.hpp	Base test fixtures	`AudioTestFixture`, setup/teardown
test_data_generator.hpp	Signal generation	`generate_sine()`, `generate_noise()`, `generate_sweep()`
golden_file_manager.hpp	Golden file I/O	`load_wav()`, `save_wav()`, `compare_wav()`
mock_factories.hpp	Mock objects	`MockAudioBuffer`, `MockParameter`

Example: Using Fixtures¶

class MyProcessorTest : public AudioTestFixture {
protected:
    void SetUp() override {
        AudioTestFixture::SetUp();
        processor = std::make_unique<MyProcessor>(sample_rate);
    }

    std::unique_ptr<MyProcessor> processor;
};

TEST_CASE_METHOD(MyProcessorTest, "Processor initializes", "[unit]") {
    REQUIRE(processor != nullptr);
    REQUIRE(processor->get_sample_rate() == sample_rate);
}

📈 Performance Testing¶

Setting Performance Targets¶

See 03_04_03_performance_testing/PERFORMANCE_CRITERIA.md

Key metrics: - Real-time factor: Must be < 0.1 (10% of available time) - Max block time: < 1ms for 512 samples @ 48kHz - Memory allocations: ZERO in audio thread - CPU usage: < 50% average

Regression Detection¶

# Windows
.\03_04_03_performance_testing\regression_detection.ps1

# macOS/Linux
./03_04_03_performance_testing/regression_detection.sh

This compares current benchmark results with previous runs and alerts on slowdowns > 10%.

🎧 Audio Quality Testing¶

Golden File Workflow¶

Generate reference (manual verification):

GoldenFileManager golden("../golden_files");
auto output = my_processor.process(input);
golden.save_wav("my_effect_output.wav", output);

Listen and approve the generated file
Commit to git (use Git LFS for .wav files)

Automate comparison:

REQUIRE(golden.compare_wav("my_effect_output.wav", output, 0.01));

For complete workflow, see 03_04_04_quality_testing/GOLDEN_FILES_MANAGEMENT.md

🐛 Troubleshooting¶

Test Fails Randomly (Flaky)¶

Diagnose:

# Run test 100 times to reproduce
.\03_04_05_troubleshooting\run_test_repeatedly.ps1 -TestName "test_example_processor" -Iterations 100

Common causes: - Uninitialized variables - Race conditions - Floating-point non-determinism - Time-dependent tests

Fix guide: 03_04_05_troubleshooting/FLAKY_TEST_PATTERNS.md

Test Fails Consistently¶

Diagnose:

# Run with debugger
gdb ./build/tests/test_example_processor
# or
lldb ./build/tests/test_example_processor

Debug guide: 03_04_05_troubleshooting/FAILURE_DIAGNOSIS.md

Performance Test Regression¶

Check:

# Compare with baseline
.\03_04_03_performance_testing\regression_detection.ps1

Profile: - Windows: VTune, Tracy Profiler - macOS: Instruments - Linux: perf, Valgrind

Guide: 03_04_03_performance_testing/PROFILING_SETUP.md

📚 Resources¶

Internal Documentation¶

TEST_FRAMEWORK_PHILOSOPHY.md - Essential read (1,156 lines)
TEST_CHEAT_SHEET.md - Quick command reference
QUICK_START.md - 5-minute tutorial

External Resources¶

Catch2: https://github.com/catchorg/Catch2/tree/devel/docs
CTest: https://cmake.org/cmake/help/latest/manual/ctest.1.html
Google Benchmark: https://github.com/google/benchmark
Audio Testing Best Practices: Will Pirkle - Designing Audio Effect Plugins in C++

Key Concepts¶

Test Pyramid: More unit tests, fewer integration tests, even fewer E2E
TDD: Red (fail) → Green (pass) → Refactor (improve)
Real-Time Safety: No allocations, locks, syscalls in audio callback
Floating-Point: Use epsilon comparisons, detect denormals
Golden Files: Reference audio for regression testing

🤝 Contributing¶

To improve the testing framework:

Write tests for new features (TDD approach)
Add test utilities if you find patterns repeating
Update golden files when intentionally changing output
Report flaky tests via issue tracker
Share testing patterns in documentation

🚀 Next Steps¶

Choose your path:

🌱 New to testing? → Read QUICK_START.md
🌿 Writing tests? → Study TEST_FRAMEWORK_PHILOSOPHY.md
🌲 Optimizing code? → Check PERFORMANCE_CRITERIA.md
🔧 Debugging tests? → See FAILURE_DIAGNOSIS.md

Happy Testing! 🎧