Float16 Vector Storage Guide

ApexBase supports FLOAT16_VECTOR columns for storing embedding vectors in half-precision (float16) format. Compared to the default float32 FixedList column, float16 halves the storage footprint and enables hardware-accelerated SIMD distance kernels on ARM and x86 platforms.

---

Table of Contents

1. When to use float16
2. Creating a float16 table
3. Inserting vectors
4. Querying — TopK and SQL
5. SIMD acceleration details
6. Quantization error and precision
7. Full working example
8. Comparison: float16 vs float32

---

When to use float16

Use float16 (FLOAT16_VECTOR)	Stick with float32 (default FixedList)
Large embedding tables (>1M rows, high dim)	Embeddings with high dynamic range (e.g., raw logits)
Memory-constrained environments	Downstream code that expects exact float32 fidelity
Apple Silicon / AWS Graviton (NEON fp16 hardware)	Dimensions that are odd multiples < 8 (micro-batches)
Models that already quantize (CLIP, text embeddings)	Prototype / exploration phase where precision matters

For most modern embedding models (CLIP, BERT, Sentence-Transformers, OpenAI text-embedding-*), float16 quantization error is well below retrieval noise.

---

Creating a float16 table

from apexbase import ApexClient

client = ApexClient("./mydb")

# SQL CREATE TABLE with FLOAT16_VECTOR column
client.execute("CREATE TABLE embeddings (doc_id TEXT, vec FLOAT16_VECTOR)")
client.use_table("embeddings")

Accepted type name aliases (all equivalent): - FLOAT16_VECTOR (recommended) - FLOAT16VECTOR - F16_VECTOR

---

Inserting vectors

Recommended: batch store with numpy arrays

Use a batch call for best throughput. The fixedlist path converts source vectors to float32 bytes and then the storage layer applies a single f32→f16 conversion, giving correct float16 data.

import numpy as np

n, dim = 50_000, 128
vecs = np.random.rand(n, dim).astype(np.float32)  # source in float32

# Batch list-of-dicts
client.store([{"doc_id": str(i), "vec": vecs[i]} for i in range(n)])

# Columnar dict — fastest for large homogeneous batches
client.store({
    "doc_id": [str(i) for i in range(n)],
    "vec":    [vecs[i] for i in range(n)],
})

numpy float16 and Python-list source data

If your upstream model already produces float16 numpy arrays, they are accepted directly. Numeric Python lists/tuples are also accepted, including single-record writes.

vecs_f16 = np.random.rand(n, dim).astype(np.float16)
client.store([{"doc_id": str(i), "vec": vecs_f16[i]} for i in range(n)])

# Single-record writes are correct too
client.store({"doc_id": "one", "vec": [0.1, 0.2, 0.3, 0.4]})

---

Querying — TopK and SQL

The query API is identical to float32 columns. Query vectors are always provided as float32 — the runtime converts them internally.

topk_distance

query = np.random.rand(128).astype(np.float32)

results = client.topk_distance("vec", query, k=10, metric="l2")
df = results.to_pandas()
# df columns: _id (int64), dist (float64) — sorted nearest first

All six metrics are supported:

metric	Description
'l2'	Euclidean distance √Σ(aᵢ−bᵢ)²
'l2_squared'	Squared L2 Σ(aᵢ−bᵢ)²
'cosine' / 'cosine_distance'	1 − cos(a, b)
'dot' / 'inner_product'	Negative dot product (for min-heap)
'l1' / 'manhattan'	Σ
'linf' / 'chebyshev'	max

batch_topk_distance

N = 100
queries = np.random.rand(N, 128).astype(np.float32)

result = client.batch_topk_distance("vec", queries, k=10, metric="l2")
# result.shape == (N, 10, 2)
# result[i, :, 0] = _id of top-10 for query i
# result[i, :, 1] = distances

SQL distance functions

array_distance, l1_distance, linf_distance all work on FLOAT16_VECTOR columns:

client.execute("""
    SELECT doc_id,
           array_distance(vec, [0.1, 0.2, 0.3, 0.4]) AS l2_dist,
           l1_distance(vec,    [0.1, 0.2, 0.3, 0.4]) AS l1_dist
    FROM embeddings
    ORDER BY l2_dist
    LIMIT 5
""")

SQL TopK via topk_distance + explode_rename:

q_str = ",".join(f"{v:.6f}" for v in query)
client.execute(f"""
    SELECT explode_rename(topk_distance(vec, [{q_str}], 10, 'l2'), '_id', 'dist')
    FROM embeddings
""")

---

SIMD acceleration details

Float16 distance kernels use native hardware half-precision instructions when available, detected at runtime:

aarch64 — NEON fp16 (fp16 feature flag)

Applies to: Apple M1/M2/M3/M4, AWS Graviton 3/4, Ampere Altra.

• Uses FCVTL / FCVTL2 to widen f16→f32 in-register (no memory round-trip)
• 8 elements loaded per NEON vector, widened to two 128-bit f32 registers
• ~2–3× faster than scalar; often ≥2× faster than equivalent float32 kernels

x86_64 — AVX2 + F16C

Applies to: Intel Haswell+ (2013+), AMD Ryzen (2017+).

• Uses _mm256_cvtph_ps to convert 8 f16 → 8 f32 per instruction
• 256-bit AVX2 arithmetic on the widened values
• Falls back to scalar if f16c or avx2 is not detected

Scalar fallback

All platforms have a scalar fallback that converts f16→f32 element-by-element. Performance is comparable to a plain float32 scan of the same data (same number of distance operations, but on narrower storage).

Runtime dispatch (no user action required)

CPU query at startup → select kernel → cache decision
                         ↙            ↘
              NEON fp16              AVX2+F16C
              (aarch64)              (x86_64)
                    ↓ neither ↓
                scalar fallback

---

Quantization error and precision

Float16 (IEEE 754 binary16) has: - 10-bit mantissa → ~3.3 decimal digits of precision - Range: ±65504 - Machine epsilon: ~9.77 × 10⁻⁴

For typical embedding ranges (values in [−1, 1]):

import numpy as np

def f16_quantize(v):
    return v.astype(np.float16).astype(np.float32)

rng = np.random.default_rng(0)
dim = 128
vec   = rng.random(dim).astype(np.float32) * 2 - 1   # uniform [-1, 1]
query = rng.random(dim).astype(np.float32) * 2 - 1

exact    = float(np.sqrt(np.sum((vec - query) ** 2)))
f16_dist = float(np.sqrt(np.sum((f16_quantize(vec) - query) ** 2)))
rel_err  = abs(exact - f16_dist) / exact
print(f"exact: {exact:.4f}  f16: {f16_dist:.4f}  rel_err: {rel_err:.2e}")
# exact: 5.8123  f16: 5.8119  rel_err: 7.23e-05

For top-k retrieval, quantization error is well within retrieval noise for dimensions ≥ 32 and any standard embedding model.

---

Full working example

import numpy as np
from apexbase import ApexClient

# 1. Create database and table
client = ApexClient("./embeddings_db")
client.execute("CREATE TABLE docs (title TEXT, vec FLOAT16_VECTOR)")
client.use_table("docs")

# 2. Generate and store embeddings (float32 source → auto-quantized to f16)
rng = np.random.default_rng(42)
n, dim = 10_000, 384
vecs  = rng.random((n, dim), dtype=np.float32)
titles = [f"document_{i}" for i in range(n)]

client.store({"title": titles, "vec": [vecs[i] for i in range(n)]})

# 3. Nearest-neighbour search
query = rng.random(dim, dtype=np.float32)

# Single query
results = client.topk_distance("vec", query, k=5, metric="l2")
print(results.to_pandas())

# Batch query
queries = rng.random((20, dim), dtype=np.float32)
batch_results = client.batch_topk_distance("vec", queries, k=5, metric="cosine")
print(batch_results.shape)  # (20, 5, 2)

# SQL query
q_str = ",".join(f"{v:.6f}" for v in query)
sql_results = client.execute(f"""
    SELECT title, array_distance(vec, [{q_str}]) AS dist
    FROM docs
    ORDER BY dist
    LIMIT 5
""")
print(sql_results.to_pandas())

client.close()

---

Comparison: float16 vs float32

	FLOAT16_VECTOR	Default FixedList (float32)
Bytes per element	2	4
Storage for 1M × dim=128	~256 MB	~512 MB
Distance precision	~3.3 decimal digits	~7.2 decimal digits
SIMD on Apple Silicon	NEON fp16 (≥2× faster)	NEON f32
SIMD on x86_64	AVX2+F16C	AVX2 f32
Query API	identical	identical
Recommended for	production, large tables	prototyping, high-precision needs

Tip: For the fastest possible inserts, use the columnar dict API:

client.store({"vec": list_of_numpy_arrays, "label": list_of_strings})

This bypasses the row-oriented path entirely and feeds data directly to the Rust columnar encoder.