WOLFRAM

Wolfram Language & System Documentation Center

CreateSemanticSearchIndex[source]

creates a search index from the data in source.

CreateSemanticSearchIndex[{source1,…}]

creates a search index with a collection of sources sourcei.

CreateSemanticSearchIndex[{source1val1,…}]

associates the source sourcei to the value vali.

CreateSemanticSearchIndex[data,"name"]

gives the search index the specified name.

Details and Options
Details and Options Details and Options
Examples  
Basic Examples  
Scope  
Data Sources  
Annotations  
Options  
DistanceFunction  
FeatureExtractor  
GeneratedAssetLocation  
Method  
OverwriteTarget  
WorkingPrecision  
Applications  
Properties & Relations  
Possible Issues  
See Also
Related Guides
History
Cite this Page

CreateSemanticSearchIndex

CreateSemanticSearchIndex[source]

creates a search index from the data in source.

CreateSemanticSearchIndex[{source1,…}]

creates a search index with a collection of sources sourcei.

CreateSemanticSearchIndex[{source1val1,…}]

associates the source sourcei to the value vali.

CreateSemanticSearchIndex[data,"name"]

gives the search index the specified name.

Details and Options

  • CreateSemanticSearchIndex is used to extract features from text that can be used to search the content semantically.
  • Possible values for source are:
  • "string"a plain string
    File["path"]individual file
    URL["url"]the text representation of "url"
    CloudObject[…]a cloud object
    LocalObject[…]a local object
    ContentObject[…]a content object
    {source1,source2,…}list of sources
  • Sources can be annotated. Every chunk from a given source will have the same annotation.
  • Possible ways to specify annotations include:
  • {source1val1,…}a list of sources and associated values
    {source1,…}{val1,…}a rule between sources and values
  • Accepted forms of vali include:
  • "string"string labels
    <|"tag1"v1,…|>an association of tags and metadata values
  • CreateSemanticSearchIndex supports the following options:
  • DistanceFunction EuclideanDistancethe distance function to use
    FeatureExtractor "MiniLM"how to extract features from text chunks
    GeneratedAssetLocation $GeneratedAssetLocationthe location of the index
    Method Automaticmethod details
    OverwriteTarget Automaticwhether to overwrite an existing location
    ProgressReporting$ProgressReportingwhether to report the progress of the computation
    WorkingPrecision "Real32"precision of floating-point calculations
  • Possible values for DistanceFunction include EuclideanDistance, SquaredEuclideanDistance, CosineDistance, JaccardDissimilarity and HammingDistance.
  • Possible values for FeatureExtractor include:
  • "MiniLM"a local model based on SentenceBERT
    LLMConfigurationan LLM-based sentence embedding
    fa custom extractor function
  • Custom extractors f must operate on a list of strings and produce a list of vectors of the same length.
  • Detailed options can be given using Method<|opt1val1|>. Possible values for opti are:
  • "ContextPadding"minimal overlap between chunks
    "MaximumItemLength"maximum length of a text chunk
    "MinimumItemLength"minimum length of a text chunk
    "SplitPattern"Automaticwhere to split long strings
  • The automatic "SplitPattern" tries to split a source text in paragraphs, newlines and words to create chunks between "MinimumItemLength" and "MaximumItemLength".
  • Possible settings for WorkingPrecision include:
  • "Integer8"signed 8-bit integers from -128 through 127
    "Real32"single-precision real (32-bit)
    "Real64"double-precision real (64-bit)

Examples

open all close all

Basic Examples  (2)

Create a new SemanticSearchIndex:

Wolfram Language code: index = CreateSemanticSearchIndex[ResourceData["Alice in Wonderland"], "Alice"]

Search in the text by semantic similarity:

Wolfram Language code: Short /@ SemanticSearch[index, "talking insect", MaxItems -> 4]

Create an index with multiple labeled sources:

Wolfram Language code: index = CreateSemanticSearchIndex[{ResourceData["Alice in Wonderland"] -> "Alice", ResourceData["Death in Venice"] -> "Venice"}]

Recover the label for the most similar items:

Wolfram Language code: SemanticSearch[index, "insects"]

Scope  (6)

Data Sources  (4)

Create an index from a string:

Wolfram Language code: CreateSemanticSearchIndex["Cats, with their mysterious charm and playful antics, effortlessly capture our hearts, prancing gracefully through sunbeams, curling up in cozy spots, and purring softly, reminding us of the joy they bring."]

Create an index from a file:

Wolfram Language code: CreateSemanticSearchIndex[File["ExampleData/USConstitution.txt"]]

Create an index from a URL:

Wolfram Language code: CreateSemanticSearchIndex[URL["http://www.wolfram.com"]]

Create an index with a specific name:

Wolfram Language code: CreateSemanticSearchIndex[ResourceData["Alice in Wonderland"], "Alice"]

Annotations  (2)

Annotate sources with a label:

Wolfram Language code: index = CreateSemanticSearchIndex[{ResourceData["Friends, Romans, Countrymen"] -> "Caesar", ResourceData["JFK Inaugural Address"] -> "JFK"}]

Each chunk inherits the correspondent source label:

Wolfram Language code: index["Label"]

The label is returned when performing search:

Wolfram Language code: SemanticSearch[index, "honour"]

Annotated sources with tagged metadata:

Wolfram Language code: CreateSemanticSearchIndex[{ResourceData["Alice in Wonderland"] -> <|"Author" -> "Lewis Carroll", "Date" -> 1865 |>, ResourceData["Friends, Romans, Countrymen"] -> <|"Author" -> "William Shakespeare", "Date" -> 1609|>}]

Specify the annotations in a separate Association:

Wolfram Language code: CreateSemanticSearchIndex[{ResourceData["Alice in Wonderland"], ResourceData["Friends, Romans, Countrymen"]} -> {<|"Author" -> "Lewis Carroll", "Date" -> 1865|>, <|"Author" -> "William Shakespeare", "Date" -> 1609|>}]

Options  (11)

DistanceFunction  (1)

Specify a custom distance function for the index:

Wolfram Language code: CreateSemanticSearchIndex[{"beautiful", "mysterious", "vibrant", "ancient"}, DistanceFunction -> CosineDistance]

By default, EuclideanDistance is used:

Wolfram Language code: CreateSemanticSearchIndex[{"beautiful", "mysterious", "vibrant", "ancient"}]["Database"]["DistanceFunction"]

FeatureExtractor  (2)

Train a custom feature extractor:

Wolfram Language code: fe = FeatureExtraction[TextWords@ResourceData["Declaration of Independence"]]

Use it to extract features from another text:

Wolfram Language code: CreateSemanticSearchIndex[ResourceData["A Far Country"], FeatureExtractor -> fe]

Use an LLM-based feature extractor:

Wolfram Language code: CreateSemanticSearchIndex[{"hi there"}, FeatureExtractor -> LLMConfiguration["Model" -> {"OpenAI", "text-embedding-3-small"}]]

GeneratedAssetLocation  (3)

Specify a custom location to store the index:

Wolfram Language code: index = CreateSemanticSearchIndex[ResourceData["Declaration of Independence"], GeneratedAssetLocation -> "CloudObject"]

Retrieve the location:

Wolfram Language code: index["Location"]

By default, the index is stored in a local object:

Wolfram Language code: CreateSemanticSearchIndex[RandomWord[10]]["Location"]

Store the vector index in a file:

Wolfram Language code: file = File[FileNameJoin[{$TemporaryDirectory, "testfile"}]]
Wolfram Language code: CreateSemanticSearchIndex[RandomWord[10], GeneratedAssetLocation -> file]

Retrieve the location:

Wolfram Language code: %["Location"]

Recreate the database from the file reference:

Wolfram Language code: SemanticSearchIndex[File["/private/var/folders/05/v_ct9frn7zv4vy6f2q18y2r80000gn/T/testfile"]]

Method  (2)

Create a text with multiple very short entries:

Wolfram Language code: text = "abacus: a calculator that performs arithmetic functions by manually sliding counters on rods or in groovesa tablet placed horizontally on top of the capital of a column as an aid in supporting the architrave abaft: at or near or toward the stern of a ship or tail of an airplanepreposition abalone: any of various large edible marine gastropods of the genus Haliotis having an ear-shaped shell with pearly interior abandoned: forsaken by owner or inhabitants free from constraint";

The entire text will be embedded as a single chunk:

Wolfram Language code: CreateSemanticSearchIndex[text]

Adjusting the minimum and maximum item length to chunk into more relevant sections:

Wolfram Language code: CreateSemanticSearchIndex[text, Method -> {"MinimumItemLength" -> 1, "MaximumItemLength" -> 64}]

Create several paragraphs of text:

Wolfram Language code: definitions = Map[StringJoin[#, ": ", WordDefinition[#]]&, RandomWord[50]]; Short[definitions]

Join them with nonstandard separators:

Wolfram Language code: text = StringRiffle[definitions, " !!! "];

The default automatic paragraph and sentence chunking will give poor results:

Wolfram Language code: First@CreateSemanticSearchIndex[text]["Items"]

Use a custom split pattern to create chunks at the custom dividers:

Wolfram Language code: First@CreateSemanticSearchIndex[text, Method -> <|"SplitPattern" -> StringExpression["!!!"], "MinimumItemLength" -> 4|>]["Items"]

OverwriteTarget  (2)

The index's automatic location is determined by its name:

Wolfram Language code: CreateSemanticSearchIndex[StringRiffle[RandomWord[10]], "myDB"]

With default OverwriteTargetAutomatic, a new index name is generated to avoid collisions:

Wolfram Language code: CreateSemanticSearchIndex[StringRiffle[RandomWord[10]], "myDB"]

To force overwriting, use OverwriteTargetTrue:

Wolfram Language code: CreateSemanticSearchIndex[StringRiffle[RandomWord[10]], "myDB", OverwriteTarget -> True]

Use OverwriteTargetFalse to perform a strict check:

Wolfram Language code: CreateSemanticSearchIndex[RandomWord[10], "myDB", OverwriteTarget -> False]

OverwriteTargetFalse will also prevent reusing the same index name in a different location:

Wolfram Language code: CreateSemanticSearchIndex[RandomWord[10], "myDB", GeneratedAssetLocation -> File["myDBfile"], OverwriteTarget -> False]

Create a file:

Wolfram Language code: CreateFile@File["myIndexfile"]

By default, existing files are not overwritten:

Wolfram Language code: CreateSemanticSearchIndex[RandomWord[10], GeneratedAssetLocation -> File["myIndexfile"]]

Use OverwriteTargetTrue to overwrite the existing file:

Wolfram Language code: CreateSemanticSearchIndex[RandomWord[10], GeneratedAssetLocation -> File["myIndexfile"], OverwriteTarget -> True]

WorkingPrecision  (1)

Specify a custom working precision for the embedding vectors:

Wolfram Language code: index = CreateSemanticSearchIndex[ResourceData["Shakespeare's Sonnets"], WorkingPrecision -> "Integer8"]

The working precision is stored in the index's vector database:

Wolfram Language code: index["Database"]

Retrieve the precision value:

Wolfram Language code: index["Database"]["WorkingPrecision"]

Applications  (2)

Create a reverse mapping between a word and its definitions:

Wolfram Language code: data = Map[StringRiffle[WordDefinition[#]] -> #&, WordList[]];

Build an index using the map:

Wolfram Language code: CreateSemanticSearchIndex[data, "definitions"]

Perform reverse lookup in a dictionary by matching the query against the definitions:

Wolfram Language code: SemanticSearch["definitions", "small land insect"]

Retrieve quotes from a book:

Wolfram Language code: index = CreateSemanticSearchIndex[ResourceData["Shakespeare's Sonnets"]]

Search for the quote without knowing its exact wording:

Wolfram Language code: SemanticSearch[index, "a day in the summer", MaxItems -> 1]

Properties & Relations  (1)

Create an index and retrieve the embeddings:

Wolfram Language code: text = "This is a sentence." semanticEmbeddings = Normal@CreateSemanticSearchIndex[text]["Embeddings"]; Short[semanticEmbeddings]

FeatureExtract can use the specified "SentenceVector" extractor to create similar embeddings:

Wolfram Language code: featureEmbeddings = FeatureExtract[{text}, "SentenceVector"]; Short@featureEmbeddings

Possible Issues  (2)

An input string is always interpreted as text:

Wolfram Language code: CreateSemanticSearchIndex["https://www.wolfram.com"]["Items"]

To follow the link, use the URL wrapper:

Wolfram Language code: CreateSemanticSearchIndex[URL["https://www.wolfram.com"]]["Items"]//First//Short

Use File to import file content:

Wolfram Language code: File
Wolfram Language code: CreateSemanticSearchIndex[File["ExampleData/USConstitution.txt"]]["Items"]//First//Short

Text with multiple small entries:

Wolfram Language code: text = "abacus: a calculator that performs arithmetic functions by manually sliding counters on rods or in groovesa tablet placed horizontally on top of the capital of a column as an aid in supporting the architrave abaft: at or near or toward the stern of a ship or tail of an airplanepreposition abalone: any of various large edible marine gastropods of the genus Haliotis having an ear-shaped shell with pearly interior abandoned: forsaken by owner or inhabitants free from constraint";

Chunking will not occur, as the length of the string is less than the default maximum:

Wolfram Language code: index = CreateSemanticSearchIndex[text, Method -> {"MinimumItemLength" -> 1}]

Lower the maximum item length to ensure chunking takes place:

Wolfram Language code: index = CreateSemanticSearchIndex[text, Method -> {"MinimumItemLength" -> 1, "MaximumItemLength" -> 64}]
Wolfram Research (2024), CreateSemanticSearchIndex, Wolfram Language function, https://reference.wolfram.com/language/ref/CreateSemanticSearchIndex.html (updated 2025).

Text

Wolfram Research (2024), CreateSemanticSearchIndex, Wolfram Language function, https://reference.wolfram.com/language/ref/CreateSemanticSearchIndex.html (updated 2025).

CMS

Wolfram Language. 2024. "CreateSemanticSearchIndex." Wolfram Language & System Documentation Center. Wolfram Research. Last Modified 2025. https://reference.wolfram.com/language/ref/CreateSemanticSearchIndex.html.

APA

Wolfram Language. (2024). CreateSemanticSearchIndex. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/CreateSemanticSearchIndex.html

BibTeX

@misc{reference.wolfram_2026_createsemanticsearchindex, author="Wolfram Research", title="{CreateSemanticSearchIndex}", year="2025", howpublished="\url{https://reference.wolfram.com/language/ref/CreateSemanticSearchIndex.html}", note=[Accessed: 13-June-2026]}

BibLaTeX

@online{reference.wolfram_2026_createsemanticsearchindex, organization={Wolfram Research}, title={CreateSemanticSearchIndex}, year={2025}, url={https://reference.wolfram.com/language/ref/CreateSemanticSearchIndex.html}, note=[Accessed: 13-June-2026]}