Log Courier is a tool created to transmit log files speedily and securely to remote Logstash instances for processing whilst using small amounts of local resources. The project is an enhanced fork of Logstash Forwarder 0.3.1 with many enhancements and behavioural improvements.
Log Courier implements the following features:
- Tail log files, following rotations and resuming at the last offset on restart
- Read from standard input for lightweight shipping of a program's output
- Extra event fields, arrays and hashes on a per file basis
- Fast and secure transmission of logs using TLS with both server and client certificate verification
- Multiline codec to combine multiple lines into single events prior to shipping
- A ruby gem to enable fast and secure transmission of logs between Logstash instances
- Transmission of logs via CurveZMQ to multiple receivers simultaneously (optional, requires ZeroMQ 4+)
To build with the optional ZMQ support use the following.
git clone https://github.com/driskell/log-courier
cd log-courier
make with=zmq
Otherwise, simply run make standalone as follows.
git clone https://github.com/driskell/log-courier
cd log-courier
make
The log-courier program can then be found in the 'bin' folder.
A genkey utility can also be found in 'bin' when ZMQ support is built. This utility will generate CurveZMQ key pair configurations for you.
If you receive errors, try using gmake instead.
Details instructions on how to integrate with Logstash can be found on the Logstash Integration page.
To quickly create a self-signed SSL certificate, run make selfsigned. This
will prompt for the certificate information; most of which can be anything or
left as the default except 'Common Name', that should be set to the exact same
hostname you will use in log-courier's 'servers' configuration. This ensures
that certificate validation passes successfully. You will find the generated
.key and .crt files inside the 'bin' folder.
If you will be connecting via IP address, the certificate will need extra
information to pass validation. Open spec/lib/openssl.cnf in your favourite
editor and look for #subjectAltName = IP:1.1.1.1, remove the pound prefix,
set the IP address, and run make selfsigned again.
The log-courier command accepts the following command line options.
-config="": The config file to load
-cpuprofile="": write cpu profile to file
-from-beginning=false: Read new files from the beginning, instead of the end
-idle-flush-time=5s: Maximum time to wait for a full spool before flushing anyway
-log-to-syslog=false: Log to syslog instead of stdout
-spool-size=1024: Maximum number of events to spool before a flush is forced