Secor: Persist Kafka Logs to Amazon S3, Google Cloud Storage and Openstack Swift
GitHub - pinterest/secor: Secor is a service implementing Kafka log persistence
Pinterest Secor

Secor is a service persisting Kafka logs to Amazon S3, Google Cloud Storage, Microsoft Azure Blob Storage and Openstack Swift.
Key features

strong consistency: as long as Kafka is not dropping messages (e.g., due to aggressive cleanup policy) before Secor is able to read them, it is guaranteed that each message will be saved in exactly one S3 file. This property is not compromised by the notorious temporal inconsistency of S3 caused by the eventual consistency model,
fault tolerance: any component of Secor is allowed to crash at any given point without compromising data integrity,
load distribution: Secor may be distributed across multiple machines,
horizontal scalability: scaling the system out to handle more load is as easy as starting extra Secor processes. Reducing the resource footprint can be achieved by killing any of the running Secor processes. Neither ramping up nor down has any impact on data consistency,
output partitioning: Secor parses incoming messages and puts them under partitioned s3 paths to enable direct import into systems like Hive. day,hour,minute level partitions are supported by secor
configurable upload policies: commit points controlling when data is persisted in S3 are configured through size-based and time-based policies (e.g., upload data when local buffer reaches size of 100MB and at least once per hour),
monitoring: metrics tracking various performance properties are exposed through Ostrich, Micrometer and optionally exported to OpenTSDB / statsD,
customizability: external log message parser may be loaded by updating the configuration,
event transformation: external message level transformation can be done by using customized class.
Qubole interface: Secor connects to Qubole to add finalized output partitions to Hive tables.

Release Notes
Release Notes for past versions can be found in RELEASE.md.
Setup/Configuration Guide
Setup/Configuration instruction is available in README.setup.md.
Secor configuration for Kubernetes/GKE environment
Extra Setup instruction for Kubernetes/GKE environment is available in README.kubernetes.md.
Detailed design
Design details are available in DESIGN.md.
License
Secor is distributed under Apache License, Version 2.0.
Maintainers

Pawel Garbacki
Henry Cai

Contributors

Andy Kramolisch
Brenden Matthews
Lucas Zago
James Green
Praveen Murugesan
Zack Dever
Leo Woessner
Jerome Gagnon
Taichi Nakashima
Lovenish Goyal
Ahsan Nabi Dar
Ashish Kumar
Ashwin Sinha
Avi Chad-Friedman

Companies who use Secor

Airbnb
Appsflyer
Branch
Coupang
Credit Karma
GO-JEK
Nextperf
PayTM
Pinterest
Rakuten
Robinhood
Simplaex
Skyscanner
Strava
TiVo
VarageSale
Viacom
Wego
Yelp
Zalando
Zapier

Help
If you have any questions or comments, you can reach us at secor-users@googlegroups.com
