ApexBase Storage Architecture¶

Overview¶

ApexBase uses a unified StorageEngine as the single entry point for all storage operations. This document defines the architecture and engineering guidelines for future development.

Architecture Diagram¶

┌─────────────────────────────────────────────────────────────────┐
│                     Python Client (client.py)                   │
│                store() / retrieve() / execute() / ...           │
└─────────────────────────────┬───────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Python Bindings (bindings.rs)                │
│           ApexStorageImpl - PyO3 wrapper for Python             │
│                                                                 │
│  Responsibilities:                                              │
│  - File locking (acquire_write_lock / acquire_read_lock)        │
│  - FTS indexing coordination                                    │
│  - Local backend cache invalidation                             │
│  - Type conversion (Python ↔ Rust)                              │
└─────────────────────────────┬───────────────────────────────────┘
                              │ All storage ops via engine()
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                 StorageEngine (engine.rs) [SINGLETON]           │
│                                                                 │
│  Core Responsibilities:                                         │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │ 1. Smart Write Routing                                  │    │
│  │    - Delta write: existing table + same columns         │    │
│  │    - Full write: new table / schema change / partial    │    │
│  └─────────────────────────────────────────────────────────┘    │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │ 2. Unified Cache Management                             │    │
│  │    - LRU eviction (MAX_CACHE_ENTRIES = 64)              │    │
│  │    - Auto-invalidation on write                         │    │
│  │    - Delta compaction on read                           │    │
│  └─────────────────────────────────────────────────────────┘    │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │ 3. Cache Invalidation                                   │    │
│  │    - invalidate(path) - single table                    │    │
│  │    - invalidate_dir(dir) - all tables in directory      │    │
│  │    - ApexExecutor cache sync                            │    │
│  └─────────────────────────────────────────────────────────┘    │
└─────────────────────────────┬───────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│              TableStorageBackend (backend.rs)                   │
│                                                                 │
│  Low-level operations:                                          │
│  - insert_rows() / insert_rows_to_delta()                       │
│  - delete() / replace()                                         │
│  - add_column() / drop_column() / rename_column()               │
│  - save() / compact()                                           │
└─────────────────────────────┬───────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│               OnDemandStorage (on_demand.rs)                    │
│                                                                 │
│  File format operations:                                        │
│  - Columnar file I/O (.apex format)                             │
│  - Delta file I/O (.apex.delta format)                          │
│  - Memory-mapped reads                                          │
│  - On-demand column loading                                     │
└─────────────────────────────────────────────────────────────────┘

StorageEngine API Reference¶

Write Operations¶

Method	Description	Write Mode
`write(path, rows, durability)`	Smart write routing	Auto (delta/full)
`write_one(path, row, durability)`	Single row write	Auto
`write_typed(path, columns, durability)`	Columnar write	Full

Read Operations¶

Method	Description
`query(sql, base_dir, table_name)`	Execute SQL query
`retrieve(path, base_dir, table_name, id)`	Get single record
`exists(path, id)`	Check if record exists
`row_count(path)`	Total row count
`active_row_count(path)`	Excluding deleted rows

Delete Operations¶

Method	Description
`delete(path, ids, durability)`	Delete multiple by IDs
`delete_one(path, id, durability)`	Delete single record

Schema Operations¶

Method	Description
`create_table(path, durability)`	Create new table
`get_schema(path)`	Get table schema
`add_column(path, name, dtype, durability)`	Add column
`drop_column(path, name, durability)`	Remove column
`rename_column(path, old, new, durability)`	Rename column
`list_columns(path)`	List all columns
`get_column_type(path, name)`	Get column type

Cache Management¶

Method	Description
`invalidate(path)`	Invalidate single table cache
`invalidate_dir(dir)`	Invalidate all tables in directory

Smart Write Routing¶

// Decision logic in StorageEngine::should_use_delta()
fn should_use_delta(table_path, rows) -> bool {
    // 1. Table must exist
    if !table_path.exists() { return false; }

    // 2. Table must have existing data
    if backend.row_count() == 0 { return false; }

    // 3. Columns must match exactly (no new, no missing)
    let schema_cols = backend.get_schema().columns();
    let data_cols = rows.columns();
    schema_cols == data_cols
}

Write Mode Selection¶

Condition	Write Mode	Reason
New table	Full	Need to create file structure
Empty table	Full	Need to establish schema
New columns	Full	Delta doesn't support schema evolution
Missing columns	Full	Preserve NULL semantics (not default 0)
Same columns	Delta	Memory efficient, append-only

Engineering Guidelines¶

1. Always Use StorageEngine¶

DO:

// In bindings.rs
let engine = crate::storage::engine::engine();
engine.write(&table_path, &rows, durability)?;

DON'T:

// Direct backend access - AVOID
let backend = TableStorageBackend::open(&table_path)?;
backend.insert_rows(&rows)?;
backend.save()?;

2. Lock Ordering¶

Always acquire locks in this order to prevent deadlocks: 1. File lock (acquire_write_lock / acquire_read_lock) 2. StorageEngine operation 3. Release file lock 4. Invalidate local caches

// Correct pattern
let lock_file = Self::acquire_write_lock(&table_path)?;
let result = engine.write(&table_path, &rows, durability);
Self::release_lock(lock_file);
self.invalidate_backend(&table_name);

3. Cache Invalidation¶

StorageEngine handles its own cache invalidation internally
bindings.rs should invalidate its local cached_backends after writes
Do NOT call ApexExecutor::invalidate_cache_for_path() directly - StorageEngine does this

4. Error Handling¶

All StorageEngine methods return io::Result<T>. Convert to PyErr at the bindings layer:

engine.write(&table_path, &rows, durability)
    .map_err(|e| PyIOError::new_err(e.to_string()))?;

5. Durability Levels¶

Level	fsync Behavior	Use Case
`Fast`	No fsync	Development, testing
`Safe`	fsync on save	Production default
`Max`	fsync + WAL	Critical data

File Format¶

V4 Row Group Format (.apex) — Default¶

V4 is the default save format since Feb 2026. Data is split into 64K-row Row Groups, each self-contained with IDs, deletion vector, and per-column data.

┌─────────────────────────────────────┐
│ Header (256 bytes)                   │
│ - Magic "APEXV3", version=4         │
│ - row_count, column_count           │
│ - footer_offset, row_group_count    │
├─────────────────────────────────────┤
│ Row Group 0                          │
│ ┌─────────────────────────────────┐ │
│ │ RG Header (32B): magic, counts  │ │
│ │ IDs (u64 array)                 │ │
│ │ Deletion vector (bitmap)        │ │
│ │ Col 0: null bitmap + data       │ │
│ │ Col 1: null bitmap + data       │ │
│ │ ...                             │ │
│ └─────────────────────────────────┘ │
├─────────────────────────────────────┤
│ Row Group 1 ...                      │
├─────────────────────────────────────┤
│ V4 Footer                            │
│ - Schema (with dict-encoded types)   │
│ - Vec<RowGroupMeta> (40B each)       │
│ - footer_size + magic                │
└─────────────────────────────────────┘

Key design decisions: - String columns are dict-encoded on disk for low-cardinality data (transparent to read path) - In-memory state always uses plain String columns (dict encoding is disk-only) - save_v4() pre-filters deleted rows, writes clean data, sets in-memory state directly (no disk reload) - Legacy V3 files are auto-detected and read correctly; first save converts to V4

Delta File (.apex.delta)¶

┌─────────────────────────────────────┐
│ Delta Header                         │
├─────────────────────────────────────┤
│ Appended rows (bincode)              │
└─────────────────────────────────────┘

Compaction¶

Delta files are compacted into the base file when: - Delta size > 10MB (DELTA_COMPACT_SIZE) - Delta rows > 100,000 (DELTA_COMPACT_ROWS) - On read (transparent to caller)

Compaction loads all data in-memory, merges delta, and saves as V4.

Adding New Operations¶

When adding a new storage operation:

Add method to StorageEngine (engine.rs)
Handle cache invalidation
Use appropriate backend method
Return io::Result<T>
Update bindings.rs to use the new engine method
Acquire file lock
Call engine method
Release lock
Invalidate local cache
Write tests covering:
Normal operation
Edge cases (empty data, non-existent table)
Concurrent access
Update this document if the operation changes the data flow

Testing¶

Run all tests to verify storage operations:

conda run -n dev maturin develop --release
conda run -n dev pytest test/ --tb=short -q

Expected: 663 passed