Graylog central logging

I am a huge fan of Log Analytics (formerly Operations Management Suite). I’ve blogged about OMS and using it previously. But, in a home lab environment where I have all the compute and storage on site, I just can’t justify the costs of sending to LA. In a business environment it’s well worth doing and taking the cost. As a result, I wanted to look at another way to collect and aggregate log files etc. Greylog keeps coming up as recommended on Reddit so I thought I would take a look.

It’s available as an OVA deployment which suits my ESXi environment nicely. Where possible I wanted to use prebuilt appliances. It just makes life easier if everything is just ready to go.

I had to deploy the OVA file direct to the ESXi host rather than through vCenter as there is an error that appears when going through vCenter. With the appliance deployed, I was able to note the username and passwords that are applicable to the appliance. First thing to do was log in to the console and set up the IP address. This was done using the Netplan YAML file which was a new experience. I manually created the DNS hostname in the AD DNS zone as well.

Within the main web interface the first thing I had to do was add an input. I went with GELF TCP. At the moment, I set it as a global input and followed the default settings.

Next task was to get events going in to Graylog. To do this, I used the NXLog Community edition which is free to download and use. I installed it on the Windows 10 machine I have so that I could work out the configuration file settings. I have settled on the following for the NXLog.conf file:

define ROOT     C:\Program Files (x86)\nxlog
define CERTDIR  %ROOT%\cert
define CONFDIR  %ROOT%\conf
define LOGDIR   %ROOT%\data
define LOGFILE  %LOGDIR%\nxlog.log
LogFile %LOGFILE%
Moduledir %ROOT%\modules
CacheDir  %ROOT%\data
Pidfile   %ROOT%\data\nxlog.pid
SpoolDir  %ROOT%\data
<Extension _syslog>
    Module      xm_syslog
</Extension>
<Extension _charconv>
    Module      xm_charconv
    AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
</Extension>
<Extension _exec>
    Module      xm_exec
</Extension>
<Extension _fileop>
    Module      xm_fileop
    # Check the size of our log file hourly, rotate if larger than 5MB
    <Schedule>
        Every   1 hour
        Exec    if (file_exists('%LOGFILE%') and \
                   (file_size('%LOGFILE%') >= 5M)) \
                    file_cycle('%LOGFILE%', 8);
    </Schedule>
    # Rotate our log file every week on Sunday at midnight
    <Schedule>
        When    @weekly
        Exec    if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
    </Schedule>
</Extension>
<Extension _gelf>
    Module      xm_gelf
</Extension>
<Input in>
    Module      im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id='1'>
                <Select Path='Application'>*</Select>
                <Select Path='Security'>*</Select>
                <Select Path='System'>*</Select>
                <Select Path='Microsoft-Windows-GroupPolicy/Operational'>*</Select>
                <Select Path='Directory Service'>*</Select>
                <Select Path='Microsoft-Windows-NTLM/Operational'>*</Select>
                <Select Path='Microsoft-Windows-SMBServer/Audit'>*</Select>
            </Query>
        </QueryList>
    </QueryXML>
</Input>
<Output out>
    Module     om_tcp
    Host       e-Graylog.lab.marklewis.blog
    Port       12201
    OutputType GELF_TCP
    Exec $Hostname = hostname_fqdn();
</Output>
<Route 1>
    Path      in => out
</Route>

It seems to be working for now. Changes I’ve made are adding the GELF extension, Adding the Input, Output and Route sections. I’ve found the above seems to work on all the Windows machines. For each individual role (DC, ICA etc) I will have to add the specific event log in to the select statement. This is because when the service starts if the log doesn’t exist it doesn’t skip it, rather aborts. I chose to use the Select option because there is a limit on the number of logs that NXLog can import. When not filtering I got the following errors

WARNING Due to a limitation in the Windows EventLog subsystem, a query cannot contain more than 256 sources.
WARNING The following sources are omitted to avoid exceeding the limit in the generated query:  Microsoft-Windows-EapMethods-Sim/Operational Microsoft-Windows-EapMethods-RasTls/Operational Microsoft-Windows-EapMethods-RasChap/Operational Microsoft-Windows-EapHost/Operational Microsoft-Windows-DxgKrnl-Operational Microsoft-Windows-DxgKrnl-Admin Microsoft-Windows-DSC/Operational Microsoft-Windows-DSC/Admin Microsoft-Windows-DiskDiagnosticResolver/Operational Microsoft-Windows-DiskDiagnosticDataCollector/Operational Microsoft-Windows-DiskDiagnostic/Operational Microsoft-Windows-DirectoryServices-Deployment/Operational Microsoft-Windows-Diagnostics-Performance/Operational Microsoft-Windows-Diagnostics-Networking/Operational Microsoft-Windows-Diagnosis-ScriptedDiagnosticsProvider/Operational Microsoft-Windows-Diagnosis-Scripted/Operational Microsoft-Windows-Diagnosis-Scripted/Admin Microsoft-Windows-Diagnosis-Scheduled/Operational Microsoft-Windows-Diagnosis-PLA/Operational Microsoft-Windows

So, in order to prevent this error and to reduce the amount of noise, I have applied the select filters. Because I will have to copy the config file, I am going to do a PowerShell script to deploy the agent. The PowerShell script is as follows

$NXLogEXE = 'C:\Program Files (x86)\nxlog\nxlog.exe'
If(!(Test-Path $NXLogEXE))
{
    $NXLogMSI = '\\lab.marklewis.blog\NETLOGON\NXLog\nxlog-ce-2.10.2150.msi'
    $NXLogConf = '\\lab.marklewis.blog\NETLOGON\NXLog\nxlog.conf'
    $NXLogConfLocal = 'C:\Program Files (x86)\nxlog\conf\nxlog.conf'
    Start-Process msiexec.exe -ArgumentList "/i $NXLogMSI /q /n" -Wait
    Copy-Item $NXLogConf $NXLogConfLocal
    Start-Service NXLog
}

I know, I shouldn’t store my MSI file in the NETLOGON share, this will change eventually. But for now, it gets the agent deployed.

At this point, I decided to remove the Enterprise plugins given that they are something I will not be using. Logging in to the appliance and navigating to

/usr/share/graylog-server/plugin

I removed the two files

graylog-plugin-enterprise-3.2.2.jar
graylog-plugin-enterprise-integrations-3.2.2.jar

Reboot the server and the Enterprise features will no longer be available to you. This is fine as I only ever plan to use the Open Source version of this.

I’ve still got a way to go with the Graylog deployment but at this point, I have a a working deployment. I can now play around with the data, look at alerts and see what’s what. I will keep playing around with this and see what things I can get working.