File Cleanup Policies

The commons configuration for Connect File Pulse.

The connector can be configured with a specific FileCleanupPolicy implementation.

The cleanup policy can be configured with the below connect property :

Configuration	Description	Type	Default	Importance
`fs.cleanup.policy.class`	The fully qualified name of the class which is used to cleanup files	class	-	high

Available Cleanup Policies

This policy deletes all files regardless of their final status (completed or failed).

To enable this policy, the property fs.cleanup.policy.class must configured to :

io.streamthoughts.kafka.connect.filepulse.fs.clean.DeleteCleanupPolicy

This policy prints to logs some information after files completion.

To enable this policy, the property fs.cleanup.policy.class must configured to :

io.streamthoughts.kafka.connect.filepulse.fs.clean.LogCleanupPolicy

This policy attempts to move atomically files to configurable target directories.

To enable this policy, the property fs.cleanup.policy.class must configured to :

io.streamthoughts.kafka.connect.filepulse.fs.clean.LocalMoveCleanPolicy

This policy only works when using the LocalFSDirectoryListing.

Configuration	Description	Type	Default	Importance
`cleaner.output.failed.path`	Target directory for file proceed with failure	string	.failure	high
`cleaner.output.succeed.path`	Target directory for file proceed successfully	string	.success	high