AWS S3

Learn how to configure the AmazonS3FileSystemListing to read files from AWS S3.

The AmazonS3FileSystemListing class can be used for listing objects that exist in a specific Amazon S3 bucket.

How to use it ?

Use the following property in your Connector’s configuration:

fs.listing.class=io.streamthoughts.kafka.connect.filepulse.fs.AmazonS3FileSystemListing

Configuration

The following table describes the properties that can be used to configure the AmazonS3FileSystemListing:

ConfigurationDescriptionTypeDefaultImportance
aws.access.key.idAWS Access Key ID AWSstring-HIGH
aws.secret.access.keyAWS Secret Access Keystring-HIGH
aws.secret.session.tokenAWS Secret Session Tokenstring-HIGH
aws.credentials.provider.classThe AWSCredentialsProvider to use if no access key id and secret access key is configured.classcom.amazonaws.auth.EnvironmentVariableCredentialsProviderLOW
aws.s3.regionThe AWS S3 Region, e.g. us-east-1stringRegions.DEFAULT_REGION.getName()MEDIUM
aws.s3.service.endpointAWS S3 custom service endpoint.string-MEDIUM
aws.s3.path.style.access.enabledConfigures the client to use path-style access for all requests.string-MEDIUM
aws.s3.bucket.nameThe name of the Amazon S3 bucket.string-HIGH
aws.s3.bucket.prefixThe prefix to be used for restricting the listing of the objects in the bucketstring-MEDIUM
aws.s3.default.object.storage.classThe AWS storage class to associate with an S3 object when it is copied by the connector (e.g., during a move operation). Accepted values are: STANDARD, GLACIER, REDUCED_REDUNDANCY, STANDARD_IA,ONEZONE_IA,INTELLIGENT_TIERING,DEEP_ARCHIVEstringLOW
aws.s3.backoff.delay.msThe base back-off time (milliseconds) before retrying a request.int100MEDIUM
aws.s3.backoff.max.delay.msThe maximum back-off time (in milliseconds) before retrying a request.int20_000MEDIUM
aws.s3.backoff.max.retriesThe maximum number of retry attempts for failed retryable requests.int`3MEDIUM