Vector Indexing in 4D: Faster Similarity Search for 4D.Vector Fields

Vector data is at the heart of modern AI-powered features: semantic search, recommendation engines, RAG (Retrieval-Augmented Generation) pipelines, and more. As 4D.Vector fields become a first-class citizen in 4D databases, the ability to index them efficiently is essential to keep queries fast as datasets grow. This post walks through how vector indexing works in 4D, what algorithms are supported, and how to manage vector indexes programmatically.

HDI_Vector_Indexation

Querying a vector field typically means finding the records whose vectors are “closest” to a reference vector, using a similarity metric. Without an index, 4D has to scan every vector in the table and compute the distance one by one, which is correct but slow as the table grows.

With a vector index, 4D uses an algorithm purpose-built for vector similarity search, dramatically reducing query time while returning the same results as an unindexed query. In other words:

• Correctness is preserved: indexed and non-indexed queries return identical result sets.
• Performance improves: indexed queries are approximately 8 times faster than unindexed ones.

Supported Vector Index Types

4D introduces three dedicated index types for 4D.Vector fields, in addition to the existing “Standard” index:

These are available through CREATE INDEX and SET INDEX, and each index type can be selected independently, enabling a single vector field to have several indexes simultaneously (e.g., both Cosine and Euclidean).

User Interface

Vector fields and their indexes are fully integrated into the Structure Editor and related dialogs:

• Fields typed as 4D.Vector fields display a dedicated vector icon , both in the Structure Editor and in the Inspector.
• The index list window shows vector indexes alongside regular ones.
• From the Inspector, the field’s contextual menu, and the Create/Edit Index window, only Cosine, Dot, Euclidean, and Automatic indexes are offered for vector fields, each selectable independently. Selecting any other index type triggers an error.

Keeping Indexes Consistent

4D takes care of keeping vector indexes consistent with the data:

• Updating a vector via .save() (or any other command) automatically updates the associated index.
• Deleting a vector via .drop() (or any other command) automatically updates the associated index.

Keep in mind that:

• Composite indexes cannot include a vector field.
• Vector index updates are not currently supported within transactions.

Operational Support

Vector indexes are treated as first-class citizens across 4D’s maintenance and administration tools:

• Pause / Resume: PAUSE INDEXES and RESUME INDEXES work on vector indexes just like any other index.
• Storage: 4D creates a folder named <project name>.4DVI in the DATA directory to store vector indexes.
• Reindexing: A reindex triggered on restart also reindexes vector indexes.
• Verification & Repair: The Maintenance and Security Center (MSC) includes vector indexes in both its verification and repair operations.
• Backup: When an index is stored in a separate file, it is included in the default backup configuration.

Vector Precision

4D.Vector.new() now accepts a precision parameter, which can be 32 or 64 bits (mk single precision and mk double precision constants). By default, new vectors are created with 32-bit precision (4 bytes per component). This reduces the storage footprint by half compared to 64-bit vectors while retaining enough precision for similarity and distance calculations, which is why it is the recommended default for most use cases.

$vector32:=4D.Vector.new([1; 2; 3])
// $vector32.precision=32

$vector64:=4D.Vector.new([1; 2; 3]; mk double precision)
// $vector64.precision=64

Compatibility note: This change breaks compatibility with existing code that relied on the previous 64-bit default. To preserve 64-bit precision, pass mk double precision explicitly.

Conclusion

Vector indexing in 4D brings production-grade performance to similarity search directly inside the database engine, without sacrificing consistency or ease of use. Whether you’re building a recommendation system, a semantic search feature, or a RAG pipeline, you can now:

1. Choose the right similarity metric (Cosine, Dot, or Euclidean) for your use case.
2. Create and manage vector indexes through code or the UI.
3. Rely on 4D to keep indexes automatically in sync as data changes.
4. Inspect, pause, verify, and repair vector indexes using the same tools you already use for regular indexes.

As always, indexed and non-indexed queries return identical results, the index is purely a performance optimization, never a behavioral change.