GNU-devel ELPA - vecdb

The vecdb package provides an interface to a vector database, where vectors are vecdbdings representing pieces of text. These databases enable "semantic search", which is a powerful way to search over meaning. This kind of search needs specialized storage and retrieval.

This package doesn't provide end-user functionality on its own; it is designed to be used in other packages that need semantic search.

The package does not provide vecdbdings, that can be done with the llm package, or any source of vecdbdings.

There are two concepts that together define a collection database of vecdbdings: the provider, and the collection. The provider is what kind of backend we are using, right now either chroma, or qdrant. This is a struct defined by the exact provider you want to use.

The collection is, for that provider, what exact database is getting used, with each collection having its own separate data. Collections must be created before being used. The collection is defined by the struct vecdb-collection which has a name (used to identify the collection), vector-size, and payload-fields. The vector-size will be based on the size of the vecdbding vector from your provider. 1536 is what Open AI uses. payload-fields is an alist of fields and their types that defines other data fields that are inserted and retrieved when search happens, and can be queried on as well (eventually, in a future iteration of the package).

An example, putting it all together, is:

(defvar my-vecdb-provider (make-vecdb-provider :api-key my-qdrant-api-key :url my-qdrant-url))
(defvar my-vecdb-collection (make-vecdb-collection :name "my test collection" :vector-size 1536 :payload-fields (('my-id . 'string))))

The provider will be supplied by the end-user, specifying how they want things stored, and any data necessary for that storage and retrieval to function. The collection is typically partially supplied by the application, with the possible exception of vecdbding size, which may be dependent on the exact vecdbding provider they are using.

Collections must be created before they can be used with vecdb-create, and vecdb-exists can return whether the collection exists.

(unless (vecdb-exists my-vecdb-provider my-vecdb-collection)
  (vecdb-create my-vecdb-provider my-vecdb-collection))

They can also be deleted with vecdb-delete.

Before data is queried, it must be added. This is done via a batch operation on a group of data, vecdb-upsert-items. This either creates an item in the collection or replaces it, based on the id of the item. id should be an integer or a string UUID. Seeing as how emacs does not provide a UUID library, probably an integer is the best choice.

(vecdb-upsert-items my-vecdb-provider my-vecdb-collection
		 (list (make-vecdb-item
			:id 91
			:vector [0.1 0.2 0.3 0.4]
			:payload '(:my-id "235913926"))))

These can be deleted with vecdb-delete-item and retrieved by ID with vecdb-get-item.

IDs used in vecdb must be uint64 values. If you have another ID you need to use to tie it together with other storage, that should go into the payload. Also, each item passed in must set the same payload fields.

Querying the database can be done with vecdb-search-by-vector, passing it a vector and optionally a number of results to return (10 is the default).

(vecdb-search-by-vector my-vecdb-provider my-vecdb-collection [0.3 0.1 0.5 -0.9] 20)

This will return the specifies number of vecdb-item structs, with the payloads they were stored with.