sql server – Azure AI Search, Pulling Vectors Embeddings from SQL table

We have an Azure SQL table with a column called Description (varchar(1000)). While we loaded data into this table via ADF Pipeline, we used the REST API to create vector embeddings on Description. We used text-embedding-ada-002 model which creates 1536 vectors. These embeddings were then saved as a comma delimited values in another column called VectorValues (varchar(max)).

We are using Azure AI Search Indexer to pull data into an index.

{"name": "Description","type": "Edm.String", "searchable": "true", "retrievable": "true", "sortable": "false", "filterable": "false", "facetable": "false"},
{
  "name": "VectorValues",
  "type": "Collection(Edm.Single)",
  "dimensions": 1536,
  "vectorSearchProfile": "myHnswProfile",
  "searchable": "true",
  "retrievable": "true",
  "filterable": "false",
  "sortable": "false",
  "facetable": "false"
}

How can we convert values of type string/varchar into Collection(Edm.Single), using the Indexer?

Apparently, while creating Indexer we can provide fieldMappings with Mapping Functions. However, we can only convert from string to string.

"fieldMappings" : [
  {
    "sourceFieldName" : "VectorValues", 
    "targetFieldName" : "VectorValues",
    "mappingFunction" : { "name" : "jsonArrayToStringCollection" }
  }]

SQL table showing 2 columns

Read more here: Source link