Open-source log data collector
> why logs?
- for compliance (auditing, company, business)
- for security (transparency, monitoring, administer)
- for debugging (find bugs by analyzing all application logs)
> challenges collecting and consuming logs
apps will write logs to files, it will take thousands of years to read all and trying to look for similar times and trying to navigate through different log formats.
to overcome this, we can make the apps write logs directly into a log database (elasticsearch) then visualize this data. this presents a solution but also there is a load of things to do such that the app developer then need to add a library for elasticsearch and configure it to connect to elastic then send those logs and also to configure the proper format. not just that, what about the third-party apps that is also used for the project such as postgresql, nginx service? how in the world can we get their logs??
so a nice solution is by using a third-party app called fluentd, it collects data from an array of data sources and process it into a unified format. not only that! you can use the data again for alerting, analysis, archiving, etc.
> Fluentd solution to the challenges, how it works?
fluentd is deployed into a cluster and starts collecting logs from all the apps. the logs that fluentd collected must be of different forms and formats (json, nginx, postgres xml formats). So fluentd will process and reformat to a unified format. on top of that, fluentd can send to any destination. example, you want to specifically send python app logs to only go to mongoDB for data analysis while other app logs go to elasticsearch or kafka. this is all possible by routing. route with tags!
> configuring Fluentd
first step: install fluentd in k8s as a daemon set. A daemon set is a component that runs on each k8s node. If you have 5 node, then they will have a fluentd pod running on them.
second step: configure using Fluentd configuration file. fluentd config is abit hard but is very powerful in terms of processing and formating.
