fluent-bit s3 output plugin

Windows binaries are available in release pages.

This plugin works with fluent-bit's go plugin interface. You can use fluent-bit-go-s3 to ship logs into AWS S3.

The configuration typically looks like:

fluent-bit --> AWS S3

Usage

$ fluent-bit -e /path/to/built/out_s3.so -c fluent-bit.conf

Or,

$ make build-image

and then, specify configuration parameters as environment variables:

$ docker run -it -e="FLUENT_BIT_ACCESS_KEY_ID=yourawsaccesskey" \
                 -e="FLUENT_BIT_SECRET_ACCESS_KEY=yourawsaccesssecret" \
                 -e="FLUENT_BIT_BUCKET_NAME=yourbucketname" \
                 -e="FLUENT_BIT_S3_PREFIX=yours3prefix" \
                 -e="FLUENT_BIT_REGION=awsregion" \
                 -e="FLUENT_BIT_SUFFIX_ALGORITHM=algorithm" \
                 cosmo0920/fluent-bit-go-s3[:built-image-version]

Using docker image from docker hub.

$ docker pull cosmo0920/fluent-bit-go-s3[:latest or other tag]

Other released images are available in DockerHub's fluent-bit-go-s3 image tags.

Or, using helm:

helm install [YOURRELEASENAME] ./helm/fluent-bit

Prerequisites

• Go 1.11+
• gcc (for cgo)
• make

Building

$ make

Configuration Options

Example:

Add this section to fluent-bit.conf:

[Output]
    Name s3
    Match *
    # Credential    /path/to/sharedcredentialfile
    AccessKeyID     yourawsaccesskeyid
    SecretAccessKey yourawssecretaccesskey
    Bucket          yourbucketname
    S3Prefix        yours3prefixname
    SuffixAlgorithm sha256
    Region          us-east-1
    Compress        gzip
    # Endpoint parameter is mainly used for minio.
    # Endpoint      http://localhost:9000
    # TimeFormat    20060102/15
    # TimeZone      Asia/Tokyo

Credentials

By default AWS credentials are loaded from their usual providers. See AWS CLI Configuration and Credential File Settings.

If the default credentials are not desired, this plugin supports the following credential providers.

Shared Credentials

Create the following file which includes credentials:

[default]
aws_access_key_id = YOUR_AWS_ACCESS_KEY_ID
aws_secret_access_key = YOUR_AWS_SECRET_ACCESS_KEY

Then, specify the following parameter in fluent-bit configuration:

Credential    /path/to/sharedcredentialfile

Static Credentials

Specify the following parameters in fluent-bit configuration:

AccessKeyID     yourawsaccesskeyid
SecretAccessKey yourawssecretaccesskey

Useful links

• fluent-bit-go