I've had the opportunity to dig into fluentd and how to build a data pipeline. Here are some of my tips for anyone getting started.

Setup your environment

There are some things that can slow you down right out the gate, like max file descriptors being too low. Make sure you do the pre-installation steps so you avoid slowness out the gate.

Understand buffering

One of the first things you run into while developing is strange throughput with plugins that include buffering. Make sure you know how the buffering works. Many plugins use this capability and if you don't configure it correctly, it will buffer everything and then push it all through at the end. Making you wait to see your code work.

Essential knowledge

Read up on the configuration file syntax because about half way down it describes the list of directives which describe the essential types of plugins including: source, match and filter

Before diving into 3rd party plugins that you'll need to connect to things like mongo, s3, mysql, etc... Read up on the built-in plug-ins as they are essential to get any job done. You can find the docs on them here

• input plugins
• output plugins
• filter plugins
• parser plugins
• formatter plugins

These plugins are you fundamental tools for processing your data into something useful.