sql server – Azure AI Search, Pulling Vectors Embeddings from SQL table
We have an Azure SQL table with a column called Description (varchar(1000)
). While we loaded data into this table via ADF Pipeline, we used the REST API to create vector embeddings on Description. We used text-embedding-ada-002 model which creates 1536 vectors. These embeddings were then saved as a comma delimited values in another column called VectorValues (varchar(max)
).
We are using Azure AI Search Indexer to pull data into an index.
{"name": "Description","type": "Edm.String", "searchable": "true", "retrievable": "true", "sortable": "false", "filterable": "false", "facetable": "false"},
{
"name": "VectorValues",
"type": "Collection(Edm.Single)",
"dimensions": 1536,
"vectorSearchProfile": "myHnswProfile",
"searchable": "true",
"retrievable": "true",
"filterable": "false",
"sortable": "false",
"facetable": "false"
}
How can we convert values of type string/varchar into Collection(Edm.Single), using the Indexer?
Apparently, while creating Indexer we can provide fieldMappings with Mapping Functions. However, we can only convert from string to string.
"fieldMappings" : [
{
"sourceFieldName" : "VectorValues",
"targetFieldName" : "VectorValues",
"mappingFunction" : { "name" : "jsonArrayToStringCollection" }
}]
Read more here: Source link