File Cleanup Policies

The commons configuration for Connect File Pulse.

The connector can be configured with a specific FileCleanupPolicy implementation.

The cleanup policy can be configured with the below connect property :

ConfigurationDescriptionTypeDefaultImportance
fs.cleanup.policy.classThe fully qualified name of the class which is used to cleanup filesclass-high

Generic Cleanup Policies

DeleteCleanPolicy

This policy deletes all files regardless of their final status (completed or failed).

To enable this policy, the property fs.cleanup.policy.class must be configured to :

io.streamthoughts.kafka.connect.filepulse.fs.clean.DeleteCleanupPolicy

LogCleanPolicy

This policy prints into logs some information after files completion.

To enable this policy, the property fs.cleanup.policy.class must be configured to :

io.streamthoughts.kafka.connect.filepulse.fs.clean.LogCleanupPolicy 

Cleanup Policies: Local Filesystem

LocalMoveCleanupPolicy

This policy attempts to move atomically files to configurable target directories.

To enable this policy, the property fs.cleanup.policy.class must be configured to :

io.streamthoughts.kafka.connect.filepulse.fs.clean.LocalMoveCleanupPolicy

Configuration

ConfigurationDescriptionTypeDefaultImportance
cleaner.output.failed.pathTarget directory for file proceed with failurestring.failurehigh
cleaner.output.succeed.pathTarget directory for file proceed successfullystring.successhigh

Cleanup Policies: Amazon

AmazonMoveCleanupPolicy

This policy moves S3 objects atomically files to configurable target directories.

To enable this policy, the property fs.cleanup.policy.class must be configured to :

io.streamthoughts.kafka.connect.filepulse.fs.clean.AmazonS3MoveCleanupPolicy
ConfigurationDescriptionTypeDefaultImportance
fs.cleanup.policy.move.success.aws.bucket.nameThe name of the destination S3 bucket for success objects (optional)stringBucket name of the source S3 ObjectHIGH
fs.cleanup.policy.move.success.aws.prefix.pathThe prefix to be used for defining the key of an S3 object to move into the destination bucket.stringHIGH
fs.cleanup.policy.move.failure.aws.bucket.nameThe name of the destination S3 bucket for failure objects (optional)stringBucket name of the source S3 ObjectHIGH
fs.cleanup.policy.move.failure.aws.prefix.pathThe prefix to be used for defining the key of an S3 object to move into the destination bucket.stringHIGH
aws.s3.default.object.storage.classThe AWS storage class to associate with an S3 object when it is copied by the connector (e.g., during a move operation). Accepted values are: STANDARD, GLACIER, REDUCED_REDUNDANCY, STANDARD_IA,ONEZONE_IA,INTELLIGENT_TIERING,DEEP_ARCHIVEstringLOW

Implementing your own policy

Last modified April 19, 2022: docs: update AWS config properties (d8b176bc)