Wolfram Language & System Documentation Center

DefineInputStreamMethod

DefineInputStreamMethod["name",{"fname₁"function₁,"fname₂"function₂,… }]

defines a custom input stream method with the specified name, allowing the Wolfram Language to call the stream functions fname_i for opening and reading from an input stream.

Details and Options

Input stream methods are different ways to create InputStream objects. DefineInputStreamMethod allows new input stream methods to be added to the predefined list.
An input stream method can provide a stream from a new input source, filter the input (such as applying decryption or decompression), perform side effects during reading, or do a combination.
The following stream functions can be provided:

	"NameTestFunction"	whether a stream of the given name can be opened
	"ConstructorFunction"	open a new input stream using this method
	"CloseFunction"	close the stream and release its resources
	"ReadFunction"	read bytes from the stream
	"EndOfFileQFunction"	whether the stream has no more input to read
	"WaitForInputFunction"	wait until there is input to read
	"SeekFunction"	reposition the point in the stream from which input is read
	"StreamPositionFunction"	the current point in the stream from which input is read
	"StreamSizeFunction"	the number of bytes that can be read from the stream
	"SeekableQFunction"	indicate whether the stream method can reposition the point from which input is read
	"ErrorTextFunction"	indicate whether there has been an error
	"ClearErrorFunction"	clear the error indication
	"OptionChangesFunction"	modify the options returned by Options[stream]

Only the "ConstructorFunction" must be supplied to DefineInputStreamMethod. Other functions have a default definition if a function is not provided.
Each InputStream has a current state expression associated with it. The initial state expression is returned by "ConstructorFunction". Other functions take the current state as a parameter and return a new state value. This state expression is where a stream method can store information such as a handle to the underlying stream resource, the current stream position, the latest error message and other information needed by the stream method functions.
If a function returning a new state value is not provided, the default definition returns the given state unmodified.
An input stream method is used by OpenRead if the Method option uses the registered input stream method name, or when MethodAutomatic and the input stream method's "NameTestFunction" return True for the stream name.
With MethodAutomatic, OpenRead["name"] calls "NameTestFunction"f as f["name"] and the input stream method to construct the stream if f returns True.
If no "NameTestFunction" is supplied, the default implementation always returns False.
When OpenRead or another function opens an input stream, it calls "ConstructorFunction"f as f[name,caller,opts], where name is the stream name (e.g. a file path or URL). The caller is OpenRead or another function that opened the stream and can be used for issuing a message. The opts arguments are option rules. The function f is expected to return a list {success,state}, where success is True if the constructor was successful in opening the stream, and state is the initial value of the stream state.
When a stream is opened with Method{"method","name"value,...}, the rules after "method" are included in the opts rules passed to the "ConstructorFunction". This allows method-specific settings to be passed to the input stream method.
Close[stream] calls "CloseFunction"f as f[state]. This is the last function that will be called on an input stream, so it should perform any necessary resource cleanup.
If no "CloseFunction" is supplied, the default implementation does nothing.
Stream-reading functions such as Read and BinaryRead call "WaitForInputFunction"f as f[state] prior to reading and should return the new state. If the stream method is used for a situation where the input may be read intermittently, as when reading output from an external process, this method should be where the waiting should occur, so that the "ReadFunction" never waits for input.
If no "WaitForInputFunction" is supplied, the default function returns its input as fast as possible, without waiting. This is suitable for stream methods where no waiting is required.
Stream-reading functions such as Read and BinaryRead call "ReadFunction"f as f[state,n], where state is the current stream state and n is the maximum number of bytes to return. The function f is expected to return a list {{byte1,...},state}, where the first element is a list of no more than n byte values, and state is the new state expression.
If no "ReadFunction" is supplied, the default implementation always returns an empty list of bytes and an unchanged state expression.
If the "ReadFunction" returns fewer than the requested number of bytes, "EndOfFileQFunction"f is called as f[state]. It is expected to return a list {eof,state}, where eof is True if the input stream has reached the end of its input source, and False otherwise.
The default "EndOfFileQFunction" function returns {True,state}.
If the "ReadFunction" returns fewer bytes than requested and "EndOfFileQFunction" does not indicate the end of input has been reached, then "ErrorTextFunction"f is called as f[state]. The function f is expected to return a list {error,state}, where error is an error string, and state is the new state. If there is no error, then f should return Null instead of an error string.
If no "ErrorTextFunction" is supplied, the default implementation returns {Null,state}.
If the "ReadFunction" has returned an error string, then "ClearErrorFunction"f is called as f[state]. The function f should return a list {Null,state}. Typically, the stream method may store the error text in the state expression during a read, so "ClearErrorFunction" should return a new state with the error message cleared.
If no "ClearErrorFunction" is supplied, the default implementation returns {Null,state}.
StreamPosition will call "StreamPositionFunction"f as f[state]. The function f should return a list {n,state}, where n is the position of the current point in the open stream, measured as the number of bytes from the beginning. A position of -1 indicates that the stream position is unknown.
If no "StreamPositionFunction" is supplied, the default implementation returns a position of -1.
When opening a stream, "StreamSizeFunction"f is called as f[state]. The function f is expected to return a list {size, state}, where size is the total size of the input stream in bytes, if known, or -1 if it is not known, and state is the next state expression (which can be unchanged). If the stream size is known, it can result in faster reading.
If no "StreamSizeFunction" is supplied, the default implementation returns a size of -1.
When opening a stream, "SeekableQFunction"  f is called as f[state]. The function f is expected to return a list {flag,state}, where flag is True if the stream method can change its stream position with SetStreamPosition and False otherwise, and state is the new state expression (can be unchanged).
If no "SeekableQFunction" is supplied, the default implementation returns a flag of False.
SetStreamPosition calls "SeekFunction"f as f[state,pos], where pos is the new stream position. The position is either a non-negative integer indicating the number of bytes from the start of the stream or Infinity to indicate the end of the stream. The function f is expected to return a list {success,state}, where success indicates if the stream position was changed successfully, and state indicates the new stream state. The "SeekFunction" is only called if the stream method's "SeekableQFunction" had previously indicated that it supports changing the stream position.
If no "SeekFunction" is supplied, the default implementation returns a success flag of False.
The "OptionChangesFunction" is used to give the stream method a way to alter the value returned by Options[stream,Method] for cases where the options contain a secret, such as a password.
Options[stream] will call "OptionChangesFunction"f as f[state,opts]. The function f should return a list {newopts,state}, where newopts will be used as the Method option setting returned by Options.
If an input stream method with the same name exists, DefineInputStreamMethod will fail.
An input stream method can be removed with RemoveInputStreamMethod.

Examples

open all close all

Basic Examples (1)

Define an input stream method that reads from the stream name itself:

Wolfram Language code:

DefineInputStreamMethod["StringName", {
	"ConstructorFunction" -> ({True, <|"Pos" -> 1, "Text" -> ToCharacterCode[#]|>}&), 
	"ReadFunction" -> (With[{pos = #["Pos"]}, 
	Replace[Take[#["Text"], {pos, UpTo[pos + #2 - 1]}], 
	bytes_List :> {bytes, Append[#, "Pos" -> pos + Length[bytes]]}]
	]&)
	}]

Open a stream with some text to read as the name:

Wolfram Language code:

strm = OpenRead["Hello, world.
Here is text.", Method -> "StringName"]

Read lines from the stream:

Wolfram Language code: ReadList[strm, String]

Close the stream:

Wolfram Language code: Close[strm]

Scope (5)

Define an input stream method that uses "NameTestFunction" to open any stream with a name beginning with the prefix "text:":

Wolfram Language code:

DefineInputStreamMethod["TextName", {
	"NameTestFunction" -> StringStartsQ["text:"], 
	"ConstructorFunction" -> ({True, <|"Pos" -> 1, "Text" -> ToCharacterCode[StringDrop[#, 5]]|>}&), 
	"ReadFunction" -> (With[{pos = #["Pos"]}, 
	Replace[Take[#["Text"], {pos, UpTo[pos + #2 - 1]}], 
	bytes_List :> {bytes, Append[#, "Pos" -> pos + Length[bytes]]}]
	]&)
	}]

Open a stream with the "TextName" stream method by including the prefix "text:" in the stream name:

Wolfram Language code:

strm = OpenRead["text:Hello, world.
Here is text."]

Read lines from the stream:

Wolfram Language code: ReadList[strm, String]

Close the stream:

Wolfram Language code: Close[strm]

Define an input stream method for files that prints a message every time the stream is read from:

Wolfram Language code:

DefineInputStreamMethod["FileProgress", {
	"ConstructorFunction" -> Function[{name, caller, opts}, 
	Replace[OpenRead[name, BinaryFormat -> True], {
	s_InputStream :> {True, <|"Stream" -> s, "End" -> False|>}, 
	_ :> {False, Null}
	}]
	], 
	"CloseFunction" -> Function[state, Close[state["Stream"]]], 
	"ReadFunction" -> Function[{state, n}, 
	Replace[BinaryReadList[state["Stream"], "Byte", n], {
	{} :> {{}, Append[state, "End" -> True]}, 
	bytes_List :> (
	Print["Read ", Length[bytes], " bytes"];
	{bytes, state}
	)
	}]
	], 
	"EndOfFileQFunction" -> Function[state, {state["End"], state}]
	}]

Open a stream with this input method, noting that it is read from immediately to fill an internal buffer:

Wolfram Language code:

file = FileNameJoin[{$InstallationDirectory, "Documentation/English/System/ExampleData/USConstitution.txt"}];
strm = OpenRead[file, Method -> "FileProgress"]

Read from the stream, with progress messages printed as it reads:

Wolfram Language code: Short[ReadList[strm, String]]

Close the stream:

Wolfram Language code: Close[strm];

A stream method that does not define seek methods will work, but the kernel will need to use more memory to support certain operations. Define a stream method that prints on each read:

Wolfram Language code:

DefineInputStreamMethod["FileProgress", {
	"ConstructorFunction" -> Function[{name, caller, opts}, 
	Replace[OpenRead[name, BinaryFormat -> True], {
	s_InputStream :> {True, <|"Stream" -> s, "End" -> False|>}, 
	_ :> {False, Null}
	}]
	], 
	"CloseFunction" -> Function[state, Close[state["Stream"]]], 
	"ReadFunction" -> Function[{state, n}, 
	Replace[BinaryReadList[state["Stream"], "Byte", n], {
	{} :> {{}, Append[state, "End" -> True]}, 
	bytes_List :> (
	Print["Read ", Length[bytes], " bytes"];
	{bytes, state}
	)
	}]
	], 
	"EndOfFileQFunction" -> Function[state, {state["End"], state}]
	}]

This shows that more and more bytes are read on successive read calls:

Wolfram Language code:

st = OpenRead[FindLibrary["demo"], BinaryFormat -> True, Method -> "FileProgress"];
data = BinaryReadList[st];
Close[st];

Redefine the stream method by adding support for "SeekableQFunction", "StreamSize" and "SeekFunction":

Wolfram Language code:

RemoveInputStreamMethod["FileProgress"];
DefineInputStreamMethod["FileProgress", {
	"ConstructorFunction" -> Function[{name, caller, opts}, 
	Replace[OpenRead[name, BinaryFormat -> True], {
	s_InputStream :> {True, <|"Stream" -> s, "Size" -> FileByteCount[name], "End" -> False|>}, 
	_ :> {False, Null}
	}]
	], 
	"CloseFunction" -> Function[state, Close[state["Stream"]]], 
	"ReadFunction" -> Function[{state, n}, 
	Replace[BinaryReadList[state["Stream"], "Byte", n], {
	{} :> {{}, Append[state, "End" -> True]}, 
	bytes_List :> (
	Print["Read ", Length[bytes], " bytes"];
	{bytes, state}
	)
	}]
	], 
	"EndOfFileQFunction" -> Function[state, {state["End"], state}], 
	"SeekableQFunction" -> Function[state, {True, state}], 
	"StreamSizeFunction" -> Function[state, {state["Size"], state}], 
	"SeekFunction" -> Function[{state, newpos}, 
	SetStreamPosition[state["Stream"], newpos];
	{True, state}
	]
	}]

Now a fixed amount is read from the stream each time, limiting the memory demands of using the stream:

Wolfram Language code:

st = OpenRead[FindLibrary["demo"], BinaryFormat -> True, Method -> "FileProgress"];
data = BinaryReadList[st];
Close[st];

Define an input stream method that artificially introduces an error after reading a certain number of bytes:

Wolfram Language code:

exampleStreamContents = BinaryReadList[FileNameJoin[{$InstallationDirectory, "Documentation/English/System/ExampleData/USConstitution.txt"}]];
DefineInputStreamMethod["ErrorDemo", {
	"ConstructorFunction" -> ({True, <|"Pos" -> 1, "Text" -> exampleStreamContents, "Error" -> Null|>}&), 
	"ReadFunction" -> 
	(If[#["Pos"] > 18000, 
	(* artificially introduce an error *)
	{{}, Append[#, "Error" -> "Internal error detected."]}, 
	(* normal reading *)
	 With[{pos = #["Pos"]}, 
	Replace[Take[#["Text"], {pos, UpTo[pos + #2 - 1]}], 
	bytes_List :> {bytes, Append[#, "Pos" -> pos + Length[bytes]]}]
	]
	]&), 
	"EndOfFileQFunction" -> ({#["Pos"] >= Length[#["Text"]], #}&), 
	"ErrorTextFunction" -> ({#["Error"], #}&), 
	"ClearErrorFunction" -> ({Null, Append[#, "Error" -> Null]}&)
	}]

Open a stream from this method:

Wolfram Language code: strm = OpenRead["errordemo", Method -> "ErrorDemo"]

Read lines from the stream, which work up to a point at which the error message from the stream method is returned:

Wolfram Language code: Short[ReadList[strm, String]]

At this point the stream is considered at end of file:

Wolfram Language code: Read[strm, String]

The stream can be repositioned:

Wolfram Language code: SetStreamPosition[strm, 0]

The stream can be reread, thanks to the "ClearErrorFunction" removing the error marker from the state expression:

Wolfram Language code: Read[strm, String]

Define an input stream method that obscures the file being read by "rotating" each byte by a number, but that hides the exact number used from the stream options by using the "OptionChangesFunction":

Wolfram Language code:

DefineInputStreamMethod["ObscureFile", {
	"ConstructorFunction" -> Function[{name, caller, opts}, 
	Replace[OpenRead[name, BinaryFormat -> True], {
	s_InputStream :> {True, <|"Stream" -> s, "End" -> False, "Rotate" -> With[{n = Lookup[opts, "Rotate", 0]}, If[FromCharacterCode[#] === "
", #, Mod[# + n, 256]]&]|>}, 
	_ :> {False, Null}
	}]
	], 
	"CloseFunction" -> Function[state, Close[state["Stream"]]], 
	"OptionChangesFunction" -> Function[{state, options}, {FilterRules[options, Except["Rotate"]], state}], 
	"ReadFunction" -> Function[{state, n}, 
	Replace[BinaryReadList[state["Stream"], "Byte", n], {
	bytes_List :> {Map[state["Rotate"], bytes], Append[state, "End" -> bytes === {}]}
	}]
	], 
	"EndOfFileQFunction" -> Function[state, {state["End"], state}]
	}]

Open a stream with a rotation parameter of 7:

Wolfram Language code:

file = FileNameJoin[{$InstallationDirectory, "Documentation/English/System/ExampleData/USConstitution.txt"}];
strm = OpenRead[file, Method -> {"ObscureFile", "Rotate" -> 7}]

Read from the stream:

Wolfram Language code: Read[strm, String]

When the stream options are checked, the "Rotate" parameter has been hidden via the "OptionChangesFunction":

Wolfram Language code: Options[strm]

Close the stream:

Wolfram Language code: Close[strm];

Top

More Learning

Tech Support

Wolfram Solutions

Wolfram Solutions For Education

Get Started

Grow Your Skills

Work with Us

Educational Programs for Adults

Educational Programs for Youth

Read

DefineInputStreamMethod

Details and Options

Examples

Basic Examples (1)

Scope (5)

Text

CMS

APA

BibTeX

BibLaTeX

DefineInputStreamMethod

Details and Options

Examples

Basic Examples (1)

Scope (5)

See Also

Tech Notes

Related Guides

History

Text

CMS

APA

BibTeX

BibLaTeX