Basic Configuration

The commons configuration for deploying a File Pulse connector.

Commons configuration

Whatever the kind of files you are processing a connector should always be configured with the below properties. Those configuration are described in detail in subsequent chapters.

ConfigurationDescriptionTypeDefaultImportance
fs.scanner.classThe fully qualified name of the class which is used to scan file systemclassio.streamthoughts.kafka.connect.filepulse.scanner.local.LocalFSDirectoryWalkermedium
fs.cleanup.policy.classThe fully qualified name of the class which is used to cleanup filesclass-high
fs.scan.directory.pathThe input directory to scanstring-high
fs.scan.interval.msTime interval (in milliseconds) at wish to scan input directorylong10000high
fs.scan.filtersFilters use to list eligible input fileslist-medium
filtersList of filters aliases to apply on each data (order is important)list-medium
internal.kafka.reporter.topicName of the internal topic used by tasks and connector to report and monitor file progression.classconnect-file-pulse-statushigh
internal.kafka.reporter.bootstrap.serversA list of host/port pairs uses by the reporter for establishing the initial connection to the Kafka cluster.string-high
task.reader.classThe fully qualified name of the class which is used by tasks to read input filesclassio.streamthoughts.kafka.connect.filepulse.reader.RowFileReaderhigh
offset.strategyA separated list of attributes, using + as a character separator, to be used for uniquely identifying an input file; must be one of [name, path, lastModified, inode, hash] (e.g: name+hash). Note that order doesn’t matter.stringpath+namehigh
topicThe default output topic to writestring-high

Prior to Connect FilePulse 1.3.x (deprecated)

ConfigurationDescriptionTypeDefaultImportance
internal.kafka.reporter.idThe reporter identifier to be used by tasks and connector to report and monitor file progression (default null). This property must only be set for users that have run a connector in version prior to 1.3.x to ensure backward-compatibility (when set, must be unique for each connect instance).string-high