The ELK (ElasticSearch+LogStash +Kibana ) stack is maybe the best method to monitor your environment in a simple visually applying way.
My task was to monitor the parameters of a Jboss/Wildfly application server where my enterprise application is hosted.
There are several guides that describe how to deploy the basic stack, I am not going to insist on that. In the following steps I just list some interesting tricks discovered while implementing the setup.
STEP 1: Feed the Jboss/Wildfly logs to LogStash
To add logging from Jboss to logstash follow the following tutorial.
http://wildfly.org/news/2015/07/25/Wildfly-And-ELK/
STEP 2: Tune the server or container where ElasticSearch runs.
ElasticSearch need a running of the max map count:
The vm_max_map_count kernel setting needs to be set to at least 262144 for production use. Depending on your platform:
In Linux
The vm_map_max_count setting should be set permanently in /etc/sysctl.conf:
$ grep vm.max_map_count /etc/sysctl.conf
vm.max_map_count=262144
To apply the setting on a live system type: sysctl -w vm.max_map_count=262144
STEP 3: Port allocation error in logstash
You may get the following error in LogStash logs:
ERROR
logstash_1 | [2017-05-18T14:20:57,206][ERROR][logstash.inputs.tcp ] Could not start TCP server: Address in use {:host=>”0.0.0.0″, :port=>5000}
The problem is that there were two configuration files used by logstash.
root@srv-syslog:~# locate central.conf
/etc/logstash/conf.d/central.conf
/etc/logstash/conf.d/central.conf.save
STEP 4: Filter logs with LogSpout
I found this to be very useful:
https://github.com/looplab/logspout-logstash