VMware vCenter Log Insight is a VMware analytics product introduced one year ago. Part of VMware vCenter family, Log Insight delivers automated log management through log analytics, aggregation, and search, extending VMware’s leadership in analytics to log data. The new Log Insight 2.0 version is much faster, more effective and usable:
- 8X faster data collection
- 6X query performance But also able to scale up and out in several ways:
- Increase nodes (up to 6) to an existing Log Insight installation.
- 2TB of live searchable data per node.
- High availability: no single point of failure for log ingestion.
- Load balanced via external load balancer. more about this in this post.
- Single UI for distributed queries and single management interface.
VMware Log Insight supports receipt and ingestion of Syslog messages that are sent over UDP, TCP, TCP with SSL encryption and via API. I’m going to be using this in the lab to collect all the syslogs from VMware vSphere hosts and Windows VM’s thru agent.
Deploy two or more Log Insight nodes (workers)
I deployed two new Log Insight Appliance in my Lab. Both Extra Small Configuration (2CPU / 4Gb memory). Configured a static IP and make sure the disk is Thick Eager Zeroed (much faster writes). Browse to the Log Insight website: https://<ip-address>. Follow the initial setup website and set admin password, e-mail adress, relay (if you have) and Finish the setup.
After the Initial setup there are no logs imported. Let’s proceed to install a second node where we choose to Join the first installed node. enter the FQDN of the first Log Insight node you installed.
Install Kemp Load Balancer
We are going to install one Kemp Load Balancer to have a single point of entry for all logs. You can setup 2 load balancers to have HA features if you wish. (HA config document)
- Go to http://kemptechnologies.com/server-load-balancing-appliances/virtual-loadbalancer/vlm-download and download the appropriate Kemp Virtual LoadMaster.
- Follow the prompts to create a Progress ID after the download begins.
- Import the VLM OVF into your VMware Infrastructure.
- Follow the instructions in the Licensing Feature Description document.
- I created new DNS records to access the load balancer thru FQDN.
Update LB and Configure Kemp for Log Insight
The Log Insight add-on pack is required and this can be acquired by posting a General request in the Kemp Help Center Community: https://support.kemptechnologies.com/hc/en-us/requests/new.
It will also be available for direct download later in September from the tools section of Kemp’s website at
https://support.kemptechnologies.com/hc/en-us/categories/200294835
To install the Log Insight Add-On on the Virtual LoadMaster, please follow the listed steps: 1- Navigate back to System Configuration > System Administration > Update Software. Browse to the ‘addon’ file and click on “Install Addon Package”.
2- Click “OK” on the resulting dialog box 3- Navigate to System Configuration > System Administration > System Reboot. Click on “Reboot”.
NOTE: Question marks in the top ribbon will indicate that you’ve lost access and the VLM is rebooting. Don’t click “Continue” so that the console automatically reloads upon completion of reboot.
4- Navigate back to System Configuration > System Administration > Update Software. You should now see that the “Log_Insight” package is set to 7.1-19-536.
Configure the Load Balancer
Download this LoadMaster Deployment Guide – VMware vCenter Log Insight Manager document. A number of Virtual Services will need to be created for the LoadMaster to work effectively with Log Insight.
Refer to the downloaded document from section 2.2 for detailed, follow the step-by-step instructions to fully configure the Load Balancer.
The MOST important value of the solution comes from the fact that you can get even distribution across the cluster of Log Insight nodes and this is not possible natively anytime syslog is sent over any other transport than UDP.
The features that the add-on pack enables is a special service type called “Log Insight”:
also check out the setting called “Log Interval Split” that controls how many messages should be directed to 1 server before moving to the next
Results
We can see all the syslog output send to both Log Insight Nodes. Spreading the load exactly 50%.
We can see all the real servers (Log Inside workers) online.
All events are distributed evenly on the Log Insight Nodes.
My 2ct
This setup is a great affordable way of a building a great Enterprise Log Analytics environment that can massively scale and is also High Available. I love the Log Insight Content Packs. Check it out for yourself!