File Cleanup Policies

The commons configuration for Connect File Pulse.

The connector can be configured with a specific FileCleanupPolicy implementation.

The cleanup policy can be configured with the below connect property :

ConfigurationDescriptionTypeDefaultImportance
fs.cleanup.policy.classThe fully qualified name of the class which is used to cleanup filesclass-high

Available Cleanup Policies

DeleteCleanPolicy

This policy deletes all files regardless of their final status (completed or failed).

To enable this policy, the property fs.cleanup.policy.class must configured to :

io.streamthoughts.kafka.connect.filepulse.clean.DeleteCleanupPolicy

Configuration

no configuration

LogCleanPolicy

This policy prints to logs some information after files completion.

To enable this policy, the property fs.cleanup.policy.class must configured to :

io.streamthoughts.kafka.connect.filepulse.clean.LogCleanupPolicy 

Configuration

no configuration

MoveCleanPolicy

This policy attempts to move atomically files to configurable target directories.

To enable this policy, the property fs.cleanup.policy.class must configured to :

io.streamthoughts.kafka.connect.filepulse.clean.MoveCleanupPolicy

Configuration

ConfigurationDescriptionTypeDefaultImportance
cleaner.output.failed.pathTarget directory for file proceed with failurestring.failurehigh
cleaner.output.succeed.pathTarget directory for file proceed successfullystring.successhigh

Implementing your own policy