Gollum is a n:m multiplexer that gathers messages from different sources and broadcasts them to a set of destinations.
There are a few basic terms used throughout Gollum:
- “Consumers” read data from other services
- “Producers” write data to other services
- “Streams” route data between consumers and producers
- A “message” is a set of data passed between consumers and producers
- “Formatters” can transform the content of messages
- “Filters” can block/pass messages based on their content
Writing a custom plugin does not require you to change any additional code besides your new plugin file.
A how-to-use documentation can be found on read the docs. Developers should use the godoc pages to get started. Plugin documentation is generated from the plugin source code. So if you feel that something is missing a look into the code may help.
Consumers (reading data)
Consoleread from stdin.
Fileread from a file (like tail).
Httpread http requests.
Kafkaread from a Kafka topic.
Kinesisread from a Kinesis stream.
ProfilerGenerate profiling messages.
Proxyuse in combination with a proxy producer to enable two-way communication.
Socketread from a socket (gollum specific protocol).
Syslogdread from a socket (syslogd protocol).
SystemDread from the SystemD journal.
Producers (writing data)
Consolewrite to stdin or stdout.
ElasticSearchwrite to elasticsearch via http/bulk.
Filewrite to a file. Supports log rotation and compression.
Firehosewrite data to a Firehose stream.
HTTPRequestHTTP request forwarder.
InfluxDBsend data to an InfluxDB server.
Kafkawrite to a Kafka topic.
Kinesiswrite data to a Kinesis stream.
Proxytwo-way communication proxy for simple protocols.
S3write data to Amazon S3 stream.
Scribesend messages to a Facebook scribe server.
Socketsend messages to a socket (gollum specific protocol).
Spoolingwrite messages to disk and retry them later.
Websocketsend messages to a websocket.
Broadcastsend to all producers in a stream.
Randomsend to a random producer in a stream.
RoundRobinswitch the producer after each send in a round robin fashion.
Routeconvert streams to one or multiple others
Formatters (modifying data)
Base64Encodeencode messages to base64.
Base64Decodedecode messages from base64.
Clearclears a message
CollectdToInflux08convert CollectD 0.8 data to InfluxDB compatible values.
CollectdToInflux09convert CollectD 0.9 data to InfluxDB compatible values.
CollectdToInflux10convert CollectD 0.10 data to InfluxDB compatible values.
ExtractJSONextracts a single field from a JSON object.
Envelopeadd a prefix and/or postfix string to a message.
Forwardwrite the message without modifying it.
Hostnameprepend the current machine’s hostname to a message.
Identifierhash the message to generate a (mostly) unique id.
JSONwrite the message as a JSON object. Messages can be parsed to generate fields.
ProcessJSONModify fields of a JSON encoded message.
ProcessTSVModify fields of a TSV encoded message.
Runlengthprepend the length of the message.
Sequenceprepend the sequence number of the message.
SplitToJSONtokenize a message and put the values into JSON fields.
StreamNameprepend the name of a stream to the payload.
StreamRevertroute a message to the previous stream (e.g. after it has been routed).
StreamRouteroute a message to another stream by reading a prefix.
Timestampprepend a timestamp to the message.
Filters (filtering data)
Alllets all message pass.
JSONblocks or lets json messages pass based on their content.
Noneblocks all messages.
Rateblocks messages that go over a given messages per second rate.
RegExpblocks or lets messages pass based on a regular expression.
Streamblocks or lets messages pass based on their stream name.
Installation from source requires the installation of the Go toolchain.
Gollum supports the Go 1.5 vendor experiment that is automatically enabled when using the provided makefile. With Go 1.6 and later you can also use
go build directly without additional modifications.
Builds with Go 1.4 or earlier versions are not officially supported and might require additional steps and modifications.
$ make $ ./gollum --help
You can use the make file coming with gollum to trigger cross platform builds.
Make will produce ready to deploy .zip files with the corresponding platform builds inside the dist folder.
To test gollum you can make a local profiler run with a predefined configuration:
$ gollum -c config/profile.conf -ps -ll 3
By default this test profiles the theoretic maximum throughput of 256 Byte messages.
You can enable different producers in that config to test the write performance of these producers, too.
Configuration files are written in the YAML format and have to be loaded via command line switch. Each plugin has a different set of configuration options which are currently described in the plugin itself, i.e. you can find examples in the GoDocs.
Use a given configuration file.
Print this help message.
Set the loglevel [0-3]. Higher levels produce more messages as in 0=Errors, 1=Warnings, 2=Notes, 3=Debug.
Port to use for metric queries. Set 0 to disable.
Number of CPUs to use. Set 0 for all CPUs.
Write the process id into a given file.
Write CPU profiler results to a given file.
Write heap profile results to a given file.
Write msg/sec measurements to log.
Print detailed version report and quit.
Test a given configuration file and exit.
Write trace results to a given file.
Print version information and quit.
Mac OS X
The easiest way to install go is by using homebrew:
brew install go
Download Go from the golang website and unzip it to e.g. /usr/local/go.
You have to set the GOROOT environment variable to the folder you chose:
If you do not already have a GOPATH set up you need to create one.
The location is free of choice, we prefer to put it into each users home folder:
mkdir -p ~/go export GOPATH=$(HOME)/go
You can download gollum via
go get github.com/trivago/gollum or clone it directly into your GOPATH.
If you choose this way you need to download your dependencies directly from that folder
mkdir -p $(GOPATH)/src/github.com/trivago cd $(GOPATH)/src/github.com/trivago git clone https://github.com/trivago/gollum.git cd gollum
Building gollum is as easy as
When using Go 1.5 make sure to enable the go vendor experiment by setting
export GO15VENDOREXPERIMENT=1 or use
If you want to do cross platform builds use
make all or specify one of the following platforms instead of “all”:
current build for current OS (default)
freebsd build for FreeBSD
linux build for Linux x64
mac build for MacOS X
pi build for Linux ARM
win build for Windows
There are also supplementary targets for make:
clean clean all artifacts created by the build process
test run unittests
vendor install gvt and update all dependencies
aws build for Linux x64 and generate an Elastic Beanstalk package
If you want to use native plugins (contrib/native) you will have to enable the corresponding imports in the file contrib/loader.go. Doing so will disable the possibility to do cross-platform builds for most users.
The repository contains a
Dockerfile which enables you to build and run gollum inside a Docker container.
$ docker build -t trivago/gollum . $ docker run -it --rm trivago/gollum -c config/profile.conf -ps -ll 3
To use your own configuration you could run:
$ docker run -it --rm -v /path/to/config.conf:/etc/gollum/gollum.conf:ro trivago/gollum -c /etc/gollum/gollum.conf
This project is released under the terms of the Apache 2.0 license.