UnifyWeaver

Case Study: Production Pipeline Integration

Educational Topic: System architecture and integration patterns
Real-World Application: Complete compilation flow from Prolog to executable bash
Case Study: How UnifyWeaver’s components work together in production

---

Introduction

This case study traces a complete journey through UnifyWeaver’s production pipeline, showing how a simple Prolog predicate transforms into optimized, executable bash code. We’ll follow the ancestor/2 predicate from definition to deployment, demonstrating how pattern detection, constraint application, template rendering, and code generation integrate seamlessly.

Understanding this pipeline is crucial for appreciating UnifyWeaver’s architecture and the engineering decisions that make it robust and extensible.

---

The Journey: From Prolog to Bash

Stage 0: Input - Prolog Predicate Definition

User Input:

% Define base relationship
parent(alice, bob).
parent(alice, charlie).
parent(bob, diana).
parent(charlie, eve).

% Define recursive relationship
ancestor(X, Y) :- parent(X, Y).
ancestor(X, Z) :- parent(X, Y), ancestor(Y, Z).

Goal: Transform this into efficient, executable bash code.

---

Stage 1: Compilation Request

Entry Point: compile_recursive/3

?- compile_recursive(ancestor/2, [output_dir('output')], BashCode).

What Happens:

1. Constraint Analysis: Extract declared constraints and runtime options
2. Firewall Check: Validate compilation request against security policies
3. Pattern Classification: Determine the type of recursion pattern
4. Compilation Dispatch: Route to appropriate specialized compiler

Code Flow:

compile_recursive(ancestor/2, [output_dir('output')], BashCode) :-
    % 1. Merge constraints and options
    get_constraints(ancestor/2, DeclaredConstraints),
    preferences:get_final_options(ancestor/2, [output_dir('output')], FinalOptions),
    merge_options(FinalOptions, DeclaredConstraints, MergedOptions),
    
    % 2. Check firewall policy
    firewall:validate_against_firewall(bash, MergedOptions, FirewallRules),
    
    % 3. Dispatch to classifier
    compile_dispatch(ancestor/2, MergedOptions, BashCode).

---

Stage 2: Pattern Classification

Classifier: classify_predicate/2

Analysis Process:

classify_predicate(ancestor/2, Classification) :-
    % Extract all clauses for ancestor/2
    findall(Body, clause(ancestor(X,Y), Body), Bodies),
    % Bodies = [parent(X,Y), (parent(X,Y), ancestor(Y,Z))]
    
    % Check if recursive
    contains_recursive_call(ancestor, Bodies),  % true
    
    % Analyze pattern type
    analyze_recursion_pattern(ancestor, 2, Bodies, Pattern).

Pattern Analysis:

1. Base Cases: [parent(X,Y)] - calls to parent/2
2. Recursive Cases: [(parent(X,Y), ancestor(Y,Z))] - contains ancestor/2 call
3. Structure Check: parent(X,Y), ancestor(Y,Z) matches transitive closure pattern
4. Classification: transitive_closure(parent)

Result: Classification = transitive_closure(parent)

---

Stage 3: Transitive Closure Compilation

Dispatcher: Routes to compile_transitive_closure/5

compile_transitive_closure(ancestor, 2, parent, MergedOptions, BashCode) :-
    template_system:generate_transitive_closure(ancestor, parent, MergedOptions, BashCode).

Template Selection: generate_transitive_closure/4 in template_system.pl

---

Stage 4: Template System Processing

Template Resolution:

generate_transitive_closure(ancestor, parent, Options, Code) :-
    % Get deduplication strategy from constraints
    constraint_analyzer:get_dedup_strategy(Options, Strategy),  % Strategy = "sort_u"
    
    % Render main template with variables
    render_template(TransitiveClosureTemplate, [
        pred = "ancestor",
        base = "parent", 
        strategy = "sort_u"
    ], Code).

Template Variables:

• `` → ancestor
• `` → parent
• `` → sort_u

---

Stage 5: Code Generation

Template Expansion:

The template system expands this core template:

#!/bin/bash
#  - transitive closure of 

# Check for base stream function
_get_stream() {
    if declare -f _stream >/dev/null 2>&1; then
        _stream
    elif declare -f  >/dev/null 2>&1; then
        
    else
        echo "Error:  not found" >&2
        return 1
    fi
}

# Main logic to find all descendants  
_all() {
    local start="$1"
    declare -A visited
    local queue_file="/tmp/_queue_$$"
    local next_queue="/tmp/_next_$$"
    
    trap "rm -f $queue_file $next_queue" EXIT PIPE
    
    echo "$start" > "$queue_file"
    visited["$start"]=1
    
    while [[ -s "$queue_file" ]]; do
        > "$next_queue"
        
        while IFS= read -r current; do
            while IFS=":" read -r from to; do
                if [[ "$from" == "$current" && -z "${visited[$to]}" ]]; then
                    visited["$to"]=1
                    echo "$to" >> "$next_queue"
                    echo "$start:$to"
                fi
            done < <(_get_stream | grep "^$current:")
        done < "$queue_file"
        
        mv "$next_queue" "$queue_file"
    done
    
    rm -f "$queue_file" "$next_queue"
}

# Check specific relationship (SIGPIPE-safe)
_check() {
    local start="$1"
    local target="$2"
    local tmpflag="/tmp/_found_$$"
    local timeout_duration="5s"
    
    # Timeout prevents infinite execution, tee prevents SIGPIPE
    timeout "$timeout_duration" _all "$start" | 
    tee >(grep -q "^$start:$target$" && touch "$tmpflag") >/dev/null
    
    if [[ -f "$tmpflag" ]]; then
        echo "$start:$target"
        rm -f "$tmpflag"
        return 0
    else
        rm -f "$tmpflag"
        return 1
    fi
}

# Main entry point
() {
    local start="$1"
    local target="$2"

    if [[ -z "$target" ]]; then
        # One-argument call: find all descendants
        if [[ "" == "sort_u" ]]; then
            _all "$start" | sort -u
        elif [[ "" == "hash_dedup" ]]; then
            declare -A seen
            _all "$start" | while IFS= read -r line; do
                if [[ -z "${seen[$line]}" ]]; then
                    seen[$line]=1
                    echo "$line"
                fi
            done
        else
            _all "$start"
        fi
    else
        # Two-argument call: check relationship
        if _check "$start" "$target"; then
            echo "$start:$target"
            return 0
        else
            return 1
        fi
    fi
}

After Variable Substitution:

#!/bin/bash
# ancestor - transitive closure of parent

parent_get_stream() {
    if declare -f parent_stream >/dev/null 2>&1; then
        parent_stream
    elif declare -f parent >/dev/null 2>&1; then
        parent
    else
        echo "Error: parent not found" >&2
        return 1
    fi
}

ancestor_all() {
    local start="$1"
    # ... BFS implementation with ancestor/parent variables
}

ancestor_check() {
    local start="$1"
    local target="$2"
    local tmpflag="/tmp/ancestor_found_$$"
    local timeout_duration="5s"
    
    timeout "$timeout_duration" ancestor_all "$start" | 
    tee >(grep -q "^$start:$target$" && touch "$tmpflag") >/dev/null
    
    # ... SIGPIPE-safe result handling
}

ancestor() {
    local start="$1"
    local target="$2"
    
    if [[ -z "$target" ]]; then
        ancestor_all "$start" | sort -u  # sort_u strategy
    else
        if ancestor_check "$start" "$target"; then
            echo "$start:$target"
            return 0
        else
            return 1
        fi
    fi
}

---

Stage 6: Integration Points

Throughout this process, multiple systems integrate seamlessly:

A. Constraint System Integration

Input Constraints:

:- constraint(unique(true)).      % Default - enable deduplication
:- constraint(unordered(true)).   % Default - order doesn't matter

Effect on Generation:

• unique(true) → generates sort -u deduplication
• unordered(true) → allows hash-based deduplication
• Combined → selects sort_u strategy

B. Firewall System Integration

Security Check:

firewall:validate_against_firewall(bash, [output_dir('output')], Policy) :-
    % Check if bash generation is allowed
    member(bash_generation(allowed), Policy),
    
    % Check if output directory is permitted
    member(output_path_pattern('output/*'), Policy).

Result: ✅ Compilation allowed

C. Template System Integration

Multi-source Template Loading:

load_template_with_strategy(transitive_closure, Config, Template) :-
    % Try sources in order: file → cached → generated
    get_option(source_order, Config, [generated], Order),
    try_sources(transitive_closure, Order, Config, Template).

Template Caching:

• Generated templates cached in memory
• File-based templates cached with timestamps
• Cache invalidation on template changes

D. Pattern Priority Integration

If multiple patterns could apply, priority order determines selection:

compile_advanced_recursive(ancestor/2, Options, BashCode) :-
    (   try_tail_recursion(ancestor/2, Options, BashCode) -> true
    ;   try_linear_recursion(ancestor/2, Options, BashCode) -> true
    ;   try_fold_pattern(ancestor/2, Options, BashCode) -> true
    ;   try_tree_recursion(ancestor/2, Options, BashCode) -> true
    ;   try_mutual_recursion(ancestor/2, Options, BashCode) -> true
    ;   compile_fallback(ancestor/2, Options, BashCode)
    ).

For ancestor/2: Matches transitive closure (specialized pattern), so advanced patterns aren’t tried.

---

Stage 7: Output and Deployment

Final Result:

$ swipl -g "compile_recursive(ancestor/2, [output_dir('output')], Code), halt."
# Code variable now contains complete bash implementation

$ echo "$Code" > output/ancestor.sh
$ chmod +x output/ancestor.sh
$ source output/ancestor.sh

$ ancestor alice
alice:bob
alice:charlie  
alice:diana
alice:eve

$ ancestor alice eve
alice:eve

Performance Characteristics:

• Time Complexity: O(V + E) BFS traversal
• Space Complexity: O(V) for visited tracking
• SIGPIPE Safety: ✅ Uses timeout + tee pattern
• Deduplication: ✅ Uses sort -u strategy

---

Advanced Integration Scenarios

Scenario 1: Constraint-Modified Compilation

Input with Constraints:

:- constraint(unique(false)).     % Disable deduplication
:- constraint(unordered(false)).  % Preserve order

ancestor_no_dedup(X, Y) :- parent(X, Y).
ancestor_no_dedup(X, Z) :- parent(X, Y), ancestor_no_dedup(Y, Z).

Pipeline Changes:

1. Constraint Analysis: Detects unique(false), unordered(false)
2. Strategy Selection: get_dedup_strategy returns no_dedup
3. Template Modification: Uses regular arrays instead of associative arrays
4. Code Generation: Preserves duplicates, maintains order

Generated Difference:

# Instead of: sort -u
ancestor_no_dedup_all "$start"  # No deduplication

Scenario 2: Fold Pattern Integration

Input with Fold Directive:

:- assertz(forbid_linear_recursion(fib/2)).

fib(0, 0).
fib(1, 1). 
fib(N, F) :- N > 1, N1 is N-1, N2 is N-2, fib(N1, F1), fib(N2, F2), F is F1 + F2.

Pipeline Changes:

1. Pattern Classification: Detects tree recursion + forbid_linear_recursion
2. Advanced Compilation: Routes to try_fold_pattern/3
3. Fold Generation: Creates graph builder, fold computer, wrapper
4. Bash Generation: 1903 characters of fold pattern code

Integration Point:

compile_advanced_recursive(fib/2, Options, BashCode) :-
    forbid_linear_recursion(fib/2),           % Directive present
    is_tree_fold_pattern(fib/2),              % Pattern matches
    !,                                        % Commit to fold pattern  
    compile_fold_pattern(fib/2, Options, BashCode).

Scenario 3: Firewall Policy Enforcement

Input with Restrictive Policy:

:- firewall([bash_generation(forbidden), output_dir('restricted/*')]).

secure_pred(X, Y) :- base_relation(X, Y).

Pipeline Changes:

1. Firewall Check: bash_generation(forbidden) blocks compilation
2. Early Exit: Compilation fails with policy violation error
3. User Feedback: Clear error message about firewall restriction

Error Handling:

validate_against_firewall(bash, Options, Firewall) :-
    member(bash_generation(forbidden), Firewall),
    !,
    format(user_error, 'Firewall policy forbids bash generation~n', []),
    fail.

---

System Architecture Insights

Component Interaction Diagram

Prolog Input
     ↓
[compile_recursive/3] ← Runtime Options
     ↓
[Constraint Analysis] ← Declared Constraints  
     ↓
[Firewall Validation] ← Security Policies
     ↓
[Pattern Classification] ← Clause Analysis
     ↓  
[Compilation Dispatch]
     ↓
[Template System] ← Pattern-Specific Templates
     ↓
[Code Generation] ← Variable Substitution
     ↓
[Output Integration] ← Final Assembly
     ↓
Executable Bash Code

Design Patterns Demonstrated

1. Strategy Pattern - Multiple Compilation Strategies

compile_dispatch(Pred/Arity, Options, Code) :-
    classify_predicate(Pred/Arity, Pattern),
    compile_strategy(Pattern, Pred/Arity, Options, Code).

compile_strategy(transitive_closure(Base), Pred, Options, Code) :-
    compile_transitive_closure(Pred, Base, Options, Code).
compile_strategy(tail_recursion, Pred, Options, Code) :-
    compile_tail_recursion(Pred, Options, Code).
% ... other strategies

2. Template Method Pattern - Consistent Compilation Flow

compile_pattern(Pred/Arity, Options, Code) :-
    validate_input(Pred/Arity, Options),
    analyze_constraints(Pred/Arity, Constraints),
    generate_code_template(Pred/Arity, Constraints, Template),
    render_template(Template, Variables, Code),
    post_process_code(Code, FinalCode).

3. Chain of Responsibility - Pattern Priority Handling

try_compilation_patterns(Pred/Arity, Options, Code) :-
    (   try_tail_pattern(Pred/Arity, Options, Code) -> true
    ;   try_linear_pattern(Pred/Arity, Options, Code) -> true  
    ;   try_fold_pattern(Pred/Arity, Options, Code) -> true
    ;   try_tree_pattern(Pred/Arity, Options, Code) -> true
    ;   try_mutual_pattern(Pred/Arity, Options, Code) -> true
    ;   compile_fallback(Pred/Arity, Options, Code)
    ).

4. Observer Pattern - Configuration and Constraint Updates

update_constraint(Pred/Arity, NewConstraint) :-
    retract_constraint(Pred/Arity, _),
    assert_constraint(Pred/Arity, NewConstraint),
    notify_constraint_observers(Pred/Arity, NewConstraint).

5. Factory Pattern - Template and Code Generation

create_compiler(Pattern, Compiler) :-
    compiler_factory(Pattern, CompilerModule),
    Compiler = CompilerModule:compile/3.

compiler_factory(transitive_closure, template_system).
compiler_factory(fold_pattern, fold_pattern_compiler).
compiler_factory(tail_recursion, tail_recursion_compiler).

---

Performance and Scalability

Compilation Performance

Metrics for ancestor/2 compilation:

• Analysis time: ~5ms (pattern classification)
• Template rendering: ~10ms (variable substitution)
• Code generation: ~15ms (bash code assembly)
• Total compilation: ~30ms

Scaling characteristics:

• Linear with predicate count - O(n) predicates compiled independently
• Constant per predicate - Template rendering doesn’t depend on clause complexity
• Template caching - Reduces repeated template loading overhead

Runtime Performance

Generated ancestor/2 performance:

• BFS traversal: O(V + E) time complexity
• Memory usage: O(V) for visited tracking
• Deduplication: O(n log n) for sort -u
• SIGPIPE safety: Minimal overhead from timeout + tee

Memory Management

Compilation memory:

• Template caching: ~100KB cached templates
• Constraint storage: ~1KB per predicate
• Code generation: ~50KB temporary structures

Runtime memory:

• Generated scripts: ~2-5KB per compiled predicate
• Execution overhead: Depends on data size, not code complexity

---

Error Handling and Debugging

Compilation Error Propagation

compile_with_error_handling(Pred/Arity, Options, Code) :-
    catch(
        compile_recursive(Pred/Arity, Options, Code),
        Error,
        handle_compilation_error(Pred/Arity, Error, Code)
    ).

handle_compilation_error(Pred/Arity, firewall_violation(Policy), _) :-
    format(user_error, 'Firewall policy ~w blocks compilation of ~w~n', 
           [Policy, Pred/Arity]),
    fail.
handle_compilation_error(Pred/Arity, template_error(TemplateId), _) :-
    format(user_error, 'Template ~w failed for predicate ~w~n',
           [TemplateId, Pred/Arity]),
    fail.

Debug Information Generation

Debug mode compilation:

compile_recursive(Pred/Arity, [debug(true)|Options], Code) :-
    format('=== Debugging compilation of ~w ===~n', [Pred/Arity]),
    
    get_constraints(Pred/Arity, Constraints),
    format('Constraints: ~w~n', [Constraints]),
    
    classify_predicate(Pred/Arity, Pattern),  
    format('Pattern: ~w~n', [Pattern]),
    
    compile_dispatch(Pred/Arity, [debug(true)|Options], Code),
    
    format('Generated ~w characters of code~n', [Code]).

Runtime Debug Features

Generated bash includes debug support:

# Debug mode - set DEBUG=1 for detailed output
if [[ "$DEBUG" == "1" ]]; then
    echo "DEBUG: ancestor_all called with start=$start" >&2
    echo "DEBUG: BFS queue initialized" >&2  
fi

---

Testing Integration

Unit Testing the Pipeline

test_compilation_pipeline :-
    % Test 1: Basic transitive closure
    compile_recursive(ancestor/2, [], Code),
    atom_length(Code, Length),
    Length > 1000,  % Should generate substantial code
    
    % Test 2: Constraint integration  
    compile_recursive(ancestor/2, [unique(false)], CodeNoDedup),
    \+ sub_atom(CodeNoDedup, _, _, _, 'sort -u'),  % No deduplication
    
    % Test 3: Firewall enforcement
    \+ compile_recursive(forbidden_pred/2, [], _),  % Should fail
    
    writeln('✅ All pipeline tests passed').

Integration Testing

Generated code testing:

#!/bin/bash
# Test generated ancestor/2 implementation

source output/ancestor.sh
source output/parent.sh

# Test basic functionality
result=$(ancestor alice)
expected="alice:bob
alice:charlie
alice:diana  
alice:eve"

if [[ "$result" == "$expected" ]]; then
    echo "✅ Basic functionality test passed"
else
    echo "❌ Basic functionality test failed"
    exit 1
fi

# Test specific query
if ancestor alice eve >/dev/null 2>&1; then
    echo "✅ Specific query test passed"
else
    echo "❌ Specific query test failed"
    exit 1
fi

---

Summary and Lessons Learned

Key Architectural Insights

1. Separation of Concerns

• Pattern classification separate from code generation
• Constraint analysis separate from template rendering
• Security policy separate from compilation logic

2. Pluggable Architecture

• New recursion patterns can be added without modifying existing code
• Template system supports multiple sources (generated, file, cached)
• Constraint system is extensible with new constraint types

3. Fail-Fast Principles

• Firewall validation happens early in pipeline
• Pattern classification fails quickly for unsupported patterns
• Template errors are caught and reported immediately

4. Performance Through Caching

• Template caching reduces repeated template loading
• Constraint caching avoids repeated analysis
• Pattern classification caching for frequently compiled predicates

Production Readiness Features

1. Error Recovery

compile_with_fallback(Pred/Arity, Options, Code) :-
    (   compile_recursive(Pred/Arity, Options, Code) -> true
    ;   compile_fallback_strategy(Pred/Arity, Options, Code)
    ).

2. Configuration Management

get_compilation_config(Pred/Arity, Config) :-
    (   user_config(Pred/Arity, UserConfig) -> Config = UserConfig
    ;   default_config(Config)
    ).

3. Monitoring and Metrics

compile_with_metrics(Pred/Arity, Options, Code) :-
    get_time(StartTime),
    compile_recursive(Pred/Arity, Options, Code),
    get_time(EndTime),
    Duration is EndTime - StartTime,
    record_compilation_metric(Pred/Arity, Duration).

Engineering Principles Demonstrated

1. Composition over Inheritance

• Systems compose through well-defined interfaces
• Template system composes templates rather than inheriting behavior
• Constraint system composes constraint effects rather than complex inheritance

2. Single Responsibility Principle

• Each module has one clear purpose
• Pattern classifier only classifies, doesn’t generate code
• Template system only handles template rendering, not pattern logic

3. Open/Closed Principle

• System is open for extension (new patterns, constraints, templates)
• System is closed for modification (core pipeline doesn’t change)

4. Dependency Inversion

• High-level compilation depends on abstractions (pattern interfaces)
• Low-level implementations depend on abstractions (template interfaces)

Key Takeaway

UnifyWeaver’s production pipeline demonstrates that complex systems can be both powerful and maintainable when built with clear architectural patterns, separation of concerns, and pluggable components.

The system successfully transforms high-level declarative specifications (Prolog predicates) into optimized, production-ready executable code (bash scripts) while maintaining flexibility, security, and performance.

---

Further Reading

Related Documentation:

• Chapter 9: Advanced Recursion Patterns
• Appendix A: SIGPIPE and Streaming Safety
• Appendix B: Fold Pattern Deep Dive
• Constraint System Architecture Guide
• Template System Design Documentation

Architecture References:

• Clean Architecture - Robert C. Martin
• Pattern-Oriented Software Architecture - Buschmann et al.
• Compiler Design Patterns - Aho, Sethi, and Ullman

Production Systems:

• Site Reliability Engineering - Google SRE Team
• Building Secure and Reliable Systems - Google
• Software Architecture in Practice - Bass, Clements, and Kazman

---

This case study documents UnifyWeaver’s production pipeline as implemented October 14, 2025, demonstrating integration patterns and system architecture in real production systems.

This site is open source. Improve this page.