Deploy a logs management infrastructure (Elasticsearch+Kibana+Fluentd) with Ansible

| 8 Comments

0.00 avg. rating (0% score) - 0 votes

During the last weeks I started to play with Elasticsearch, Fluentd and Kibana. I made a documentation to help on deploying it easily.

As you may know, I’m an Ansible fan, so I made Ansible playbooks to deploy a complete infrastructure (server and clients). They will deploy this kind of architecture:

On the client side, Fluentd clients will get syslog and Nginx logs, to send them to the Fluentd server. On the server side, a Fluentd receiver will be there to get data from other Fluentd clients. It will then push it to the Elasticsearch server. The Elasticsearch server is located on the same server than the Fluentd receiver to make it simpler. An autopurge can be configured directly from the playbook using curator tool. To finish, Kibana is installed on the server to make it simpler as well.

You can find those playbooks on my GitHub or on Ansible Galaxy.

Is it complex to configure Ansible playbooks it? I made my best to make it as quick as possible and as customizable as possible (for the context and my needs). What you need to do is first getting playbooks like this (site.yml):

Then you have to configure/adapt the mandatory options to your needs for fluentd_clients (fluentd_client.yml):

And like this for the log server (kibana.yml):

Then launch the Ansible playbook, it will install everything by itself:

At the end, the web interface of kibana will be ready and all your fluentd clients will redirect their syslog and default nginx configuration to ElasticSearch.

Note: I’ve used my Nginx playbook, but you can use any other one to setup kibana interface as this is made in AngularJS.

Author: Deimos

I'm a passionate DevOps. I love transmit my skills and I love working on high availability infrastructures/technologies.

8 Comments

  1. I’m missing your nginx role in ansible galaxy. Could you provide one, because I’d like to test the whole setup. For now I’d like to install it everything on a single host (kibana + elasticsearch, fluentd) + redirect localhost syslog + nginx to this.

  2. Peux-tu expliquer pourquoi tu es parti sur Fluentd plutôt que ELK (donc Logstash) ? Ton post https://blog.deimos.fr/2014/05/13/logstash-vs-fluentd/ tient toujours ?

    J’avoue ne pas savoir sur quel pied jongler, le créateur de Logstash étant chez Elasticsearch et le fait que toute la stack soit chez eux (ELK) je me demande si Fluentd ne va pas se retrouver délaissé mais surtout isolé.

    • Oui mon post tient toujours. Sur Logstash il y a un problème de maturité du produit aujourd’hui. Trop de bugs, crashs, changements fondamentaux d’une version a une autre…bref on sent que le soft est en train de prendre ses marques ce qui est une bonne choses. Mais a l’heure d’aujourd’hui, je pense qu’il n’est pas assez mature pour être mis en production.

      L’autre aspect important est qu’il se focalise majoritairement sur ElasticSearch, ce qui est normal pour les raisons que tu cite. Mais que se passe t’il si tu veux envoyer sur autre chose que de l’ES ? As tu regardé le nombre de possibilité en sortie de Fluentd par rapport a Logstash ? Renseigne toi, tu comprendras.

      Pour ce qui est de Fluentd, il est massivement utilisé aujourd’hui. Je pense qu’il a encore de belles années devant lui. Et je pense également qu’un jour viendra ou le trio ES+Kibana+Logstash fera partie d’une solution globale ou les gens ne se poseront plus la question de quoi utiliser pour collecter leurs logs.

      J’espère avoir été clair sur ma réponse 🙂

  3. Tu as été très clair et tu m’as même convaincu, je n’ai plus qu’à jouer avec Ansible, c’est du beurre chez toi 😉

  4. Why use rsyslog that sends logs to localhost where fluentd agent serves like forwarder and sends logs to the aggregation server?
    m-m-m, kurzgesagt, why rsyslog->fluentd->fluentd_>elasticsearch instead of rsyslog->fluentd->elasticsearch?

    • Hi, the reason is in fact simple. Fluentd on localhost permit to have a local caching system. That mean if Fluentd located on ElasticSearch fall down or is in maintenance, you’ll loose some data during the down time if you only use rsyslog. While having Fluentd locally will send logs back to the other Fluentd when it will be up and running again.

      Hope my explanation is clear 🙂

Laisser une réponse