WOLFRAM

Wolfram Language & System Documentation Center

TokenWords

is an option for Read and related functions which gives a list of token words to be used to delimit words.

Details
Details and Options Details and Options
Examples  
Basic Examples  
See Also
Tech Notes
Related Guides
History
Cite this Page

TokenWords

TokenWords

is an option for Read and related functions which gives a list of token words to be used to delimit words.

Details

  • The setting for TokenWords is a list of strings which are used as delimiters for words to be read.
  • The delimiters specified by TokenWords are themselves returned as words.

Examples

Basic Examples  (1)

A text file:

Wolfram Language code: FilePrint["ExampleData/language"]

Read the file:

Wolfram Language code: ReadList["ExampleData/language", Word]

Read the file using token words to delimit words in the text:

Wolfram Language code: ReadList["ExampleData/language", Word, TokenWords -> {"+", "*"}]
Wolfram Research (1991), TokenWords, Wolfram Language function, https://reference.wolfram.com/language/ref/TokenWords.html.

Text

Wolfram Research (1991), TokenWords, Wolfram Language function, https://reference.wolfram.com/language/ref/TokenWords.html.

CMS

Wolfram Language. 1991. "TokenWords." Wolfram Language & System Documentation Center. Wolfram Research. https://reference.wolfram.com/language/ref/TokenWords.html.

APA

Wolfram Language. (1991). TokenWords. Wolfram Language & System Documentation Center. Retrieved from https://reference.wolfram.com/language/ref/TokenWords.html

BibTeX

@misc{reference.wolfram_2026_tokenwords, author="Wolfram Research", title="{TokenWords}", year="1991", howpublished="\url{https://reference.wolfram.com/language/ref/TokenWords.html}", note=[Accessed: 13-June-2026]}

BibLaTeX

@online{reference.wolfram_2026_tokenwords, organization={Wolfram Research}, title={TokenWords}, year={1991}, url={https://reference.wolfram.com/language/ref/TokenWords.html}, note=[Accessed: 13-June-2026]}