Logstash custom filters

Logstash filters include plain ruby https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html which allows to run custom ruby code to manage events.

Suppose we have the following java stack trace where we need to pull java class names coming after “Caused by: “

  at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
  at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
  at com.company.application_service(app.java:50)
  at com.company.application_service.(app.java:43)
  at com.company.controller.app(app.java:49)
... 34 more

Here is an entire filter snippet including ruby filter.

grok {
  match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} +%{LOGLEVEL} +\[%{WORD}\] +%{TIME} +%{LOGLEVEL:loglevel} +\[%{WORD:component}\] +%{GREEDYDATA:message}" }
  overwrite => [ "message" ]
ruby {
  code => "
    event['caused_by'] = event['message'].scan(/Caused by: ([\w\.]+)\n/).flatten;

Ruby code parses the entire event message finding all occurrences of java class names following “Caused by: “, flattens the array and puts it into new field ‘caused_by’.