DataFrameWriter

sink Operator

Represents an operator that writes each data frame in the sequence to an Apache Arrow file using an ArrowWriter.

Inputs & Outputs

Writes all of the data frames in the sequence to an Apache Arrow file.

The sequence of BufferedDataFrames to write.

An observable sequence that is identical to the source sequence but where there is an additional side effect of writing the frames to an Apache Arrow file.

Writes all of the data frames in the sequence to an Apache Arrow file.

The sequence of DataFrames to write.

An observable sequence that is identical to the source sequence but where there is an additional side effect of writing the frames to an Apache Arrow file.

Properties

Property	Type	Description
EnableCompression	bool	Gets or sets a value indicating whether to enable compression when writing to the Arrow file. When enabled, data is compressed using the Zstandard compression algorithm before writing. This reduces file size at the cost of additional CPU overhead on read and write.
FileName	string	Gets or sets the name of the file on which to write the elements.
Suffix	PathSuffix	Gets or sets the suffix used to generate file names. None = 0 FileCount = 1 Timestamp = 2
Buffered	bool	Gets or sets a value indicating whether element writing should be buffered. If true, the write commands will be queued in memory as fast as possible and will be processed by the writer in a different thread. Otherwise, writing will be done in the same thread in which notifications arrive.
Overwrite	bool	Gets or sets a value indicating whether to overwrite the output file if it already exists.