Developer Guide

Learn about the concepts and the functionalities of the Connect File Pulse Plugin.

The Developer Guide section helps you learn about the functionalities of the File Pulse Connector and the concepts File Pulse uses to process and transform your data, and helps you obtain a deeper understanding of how File Pulse Connector works.


Installation

How to install Connect File Pulse

Configuration

The common configurations for deploying a File Pulse connector.

FileSystem Listing

Learn how to configure Connect FilePulse for listing files from local or remote storage system.

FileListFilter

File Readers

Learn how to configure Connect FilePulse for a specific file format.

Identifying Files

Learn how Kafka Connect FilePulse uniquely identifies files.

Filter Chain Definition

Learn how to define complex pipelines to transform and structure your data before integration into Kafka.

Accessing Data and Metadata

Learn how to use the Simple Connect Expression Language (SCeL) for accessing and manipulating data.

Conditional Execution

Learn how to conditionally execute a transformation filter.

Handling Failures

Learn how to handle failures thrown while processing of records by the filter chain.

Processing Filters

The list of available transformation filters.

Tracking File Status

Learn how Connect FilePulse tracks the processing of each input file.

File Cleanup Policies

The commons configuration for Connect File Pulse.