File Cleanup Policies

The commons configuration for Connect File Pulse.

The connector can be configured with a specific FileCleanupPolicy implementation.

The cleanup policy can be configured with the below connect property :

ConfigurationDescriptionTypeDefaultImportance
fs.cleanup.policy.classThe fully qualified name of the class which is used to cleanup filesclass-high

Available Cleanup Policies

DeleteCleanPolicy

This policy deletes all files regardless of their final status (completed or failed).

To enable this policy, the property fs.cleanup.policy.class must be configured to :

io.streamthoughts.kafka.connect.filepulse.fs.clean.DeleteCleanupPolicy

LogCleanPolicy

This policy prints into logs some information after files completion.

To enable this policy, the property fs.cleanup.policy.class must be configured to :

io.streamthoughts.kafka.connect.filepulse.fs.clean.LogCleanupPolicy 

LocalMoveCleanupPolicy

This policy attempts to move atomically files to configurable target directories.

To enable this policy, the property fs.cleanup.policy.class must be configured to :

io.streamthoughts.kafka.connect.filepulse.fs.clean.LocalMoveCleanupPolicy

Configuration

ConfigurationDescriptionTypeDefaultImportance
cleaner.output.failed.pathTarget directory for file proceed with failurestring.failurehigh
cleaner.output.succeed.pathTarget directory for file proceed successfullystring.successhigh

Implementing your own policy

Last modified April 6, 2022: docs: fix missing archive (8c1e46d0)