All Projects → felipefrizzo → terraform-aws-kinesis-firehose

felipefrizzo / terraform-aws-kinesis-firehose

Licence: MIT license
This code creates a Kinesis Firehose in AWS to send CloudWatch log data to S3.

Programming Languages

HCL
1544 projects
python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to terraform-aws-kinesis-firehose

Parquet Cpp
Apache Parquet
Stars: ✭ 339 (+1256%)
Mutual labels:  big-data, parquet
Parquet Mr
Apache Parquet
Stars: ✭ 1,278 (+5012%)
Mutual labels:  big-data, parquet
Parquet Dotnet
🏐 Apache Parquet for modern .NET
Stars: ✭ 276 (+1004%)
Mutual labels:  big-data, parquet
Amazon S3 Find And Forget
Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)
Stars: ✭ 115 (+360%)
Mutual labels:  big-data, parquet
Eel Sdk
Big Data Toolkit for the JVM
Stars: ✭ 140 (+460%)
Mutual labels:  big-data, parquet
Bigdata Playground
A complete example of a big data application using : Kubernetes (kops/aws), Apache Spark SQL/Streaming/MLib, Apache Flink, Scala, Python, Apache Kafka, Apache Hbase, Apache Parquet, Apache Avro, Apache Storm, Twitter Api, MongoDB, NodeJS, Angular, GraphQL
Stars: ✭ 177 (+608%)
Mutual labels:  big-data, parquet
Parquet Format
Apache Parquet
Stars: ✭ 800 (+3100%)
Mutual labels:  big-data, parquet
Drill
Apache Drill is a distributed MPP query layer for self describing data
Stars: ✭ 1,619 (+6376%)
Mutual labels:  big-data, parquet
Gaffer
A large-scale entity and relation database supporting aggregation of properties
Stars: ✭ 1,642 (+6468%)
Mutual labels:  big-data, parquet
Parquetviewer
Simple windows desktop application for viewing & querying Apache Parquet files
Stars: ✭ 145 (+480%)
Mutual labels:  big-data, parquet
Awkward 0.x
Manipulate arrays of complex data structures as easily as Numpy.
Stars: ✭ 216 (+764%)
Mutual labels:  big-data, parquet
img2dataset
Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.
Stars: ✭ 1,173 (+4592%)
Mutual labels:  big-data
terraform-provider-nexus
Terraform provider for Sonatype Nexus
Stars: ✭ 74 (+196%)
Mutual labels:  terraform-provider
automile-php
Automile offers a simple, smart, cutting-edge telematics solution for businesses to track and manage their business vehicles.
Stars: ✭ 28 (+12%)
Mutual labels:  big-data
FlameStream
Distributed stream processing model and its implementation
Stars: ✭ 14 (-44%)
Mutual labels:  big-data
terraform-provider-sumologic
Terraform provider for Sumo Logic
Stars: ✭ 24 (-4%)
Mutual labels:  terraform-provider
Real Time Social Media Mining
DevOps pipeline for Real Time Social/Web Mining
Stars: ✭ 22 (-12%)
Mutual labels:  big-data
lubeck
High level linear algebra library for Dlang
Stars: ✭ 57 (+128%)
Mutual labels:  big-data
terraform-aws-cognito-user-pool
A Terraform module to create and manage Cognito User Pools (Simple and Secure User Sign-Up, Sign-In, and Access Control) on Amazon Web Services (AWS). https://aws.amazon.com/cognito
Stars: ✭ 46 (+84%)
Mutual labels:  terraform-aws
terraform-provider-ignition
Terraform Ignition provider
Stars: ✭ 37 (+48%)
Mutual labels:  terraform-provider

terraform-aws-kinesis-firehose

This code creates a Kinesis Firehose in AWS to send CloudWatch log data to S3.

Usage

module "kinesis-firehose" {
  source                                = "git::https://github.com/felipefrizzo/terraform-aws-kinesis-firehose.git?ref=master"
  region                                = "aws_region"
  kinesis_firehose_stream_name          = "stream_name"
  kinesis_firehose_stream_backup_prefix = "bucket_backup_prefix"
  bucket_name                           = "bucket_name"
  root_path                             = false
  lambda_function_name                  = "lambda_function_name"
  lambda_function_file_name             = "kinesis-firehose-cloudwatch-logs-json-processor-python"
  glue_catalog_database_name            = "glue_catalog_database_name"
  glue_catalog_table_name               = "glue_catalog_table_name"
  glue_catalog_table_columns            = {
    "column_name" = {
      name = "column_name"
      type = "column_type"
    }
  }
  cloudwatch_subscription_filter_name   = "cloudwatch_subscription_filter_name"
  cloudwatch_log_group_name             = "cloudwatch_log_group_name"
  cloudwatch_filter_pattern             = "cloudwatch_filter_pattern"
}

Inputs

Name Description Type Default Required
kinesis_firehose_stream_name Name to be use on kinesis firehose stream (e.g. poc_logs) string - yes
kinesis_firehose_stream_backup_prefix The prefix name to use for the kinesis backup (e.g. backup_prefix) string `` no
bucket_name Bucket name string - yes
root_path The path where the lambda function file is located is root or module path (e.g. true) boolean `` no
lambda_function_name Lambda function name (e.g. lambda_kinesis) string - yes
lambda_function_file_name Lambda function file name string - yes
glue_catalog_database_name Glue catalog database name string - yes
glue_catalog_table_name Glue catalog database table name string - yes
glue_catalog_table_columns A map of object of table columns map <map> yes
cloudwatch_subscription_filter_name Subscription filter name string - yes
cloudwatch_log_group_name Cloudwatch log group name string - yes
cloudwatch_filter_pattern Cloudwatch filter pattern string - yes
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].