Monitoring 6000+ hosts in Zabbix A Pseudo-DevOps journey About me - - PowerPoint PPT Presentation

Monitoring 6000+ hosts in Zabbix

A Pseudo-DevOps journey

About me

š Senior Systems Engineer Tools and Automation Kinetic IT @ Department of Education š Co founder of Passive Eye Ltd š Open Source contributor

• Dept. Education

š ~800 schools š ~400,000 end users š ~5000 SOE servers @ schools š ~1500 heterogeneous servers @ central office š Hub-spoke topology š Vast geographic distribution

Problem definition

š Multiple, disconnected monitoring tools

š Poor coverage š Lack of correlation š Duplication of effort š Inconsistent practice š Difficulty measuring SLAs

Requirements

š Single pane of glass š Scalability š Extensibility š Ease of use š Low costs licensing

Enter Zabbix

š All-in-one š Support for diverse devices š Small footprint and scalable architecture š Extensible API š Configuration UI š Free and open source + support

How Zabbix works

š Primary server š Database š Frontend š Proxy Servers š Agents š Passive devices

proxy proxy web

How Zabbix works

š Items and LLD š Active and passive checks š Hosts and templates š Triggers, Events and Actions š Graphs, Screens and Maps

Planning

š Scrum + Jira Agile

Building

š Discrete environments in Vagrant š Infrastructure as code š Discrete feature branches š Monolithic source repo

http://nvie.com/posts/a-successful-git-branching-model/

Puppet code

Testing

š Bamboo š Cucumber

Deployment

š Server, DB, Web š No Proxies (so far)

Performance

Integrations

š Active Directory š CMDB š Service Management š ICT Dashboard

Autonomy

š Host registration š Low level discovery š User provisioning š Remediation scripts š Data housekeeping š Incidents and escalations

Template hierarchy

š Host š Class š Templates š Items, etc.

Windows monitoring

š OOB Support:

š WMI queries š Performance counter probes š Event Log monitor š Service state

Windows monitoring

š Customisations

š Hostname casing and format š Service discovery š Performance counter discovery š Failover Cluster discovery š Persistent disk and volume identification

Windows monitoring

š Tools

š MSI installer package š Test PowerShell script š Performance Counter template builder

Windows monitoring

š Convert-PerfCountToZabbixTemplate.ps1

š Counter Set > Application š Multi-instance counter > Discovery Rule + Prototypes š Single-instance counter > Item check

> ./Convert-PerfCountersToZabbixTemplate.ps1

• CounterSet Processor | Out-File template.xml

Linux monitoring

š Modules

š Extensions for Linux kernel š PostgreSQL

š Packaging š Test script

SNMP monitoring

š CRAC š UPS š Dell iDRAC š IPS š Mail gateways

SNMP monitoring

š mib2zabbix.pl

š Tree nodes > Applications š OID Tables > Discovery Rules + Prototypes š OID Scalars > Item checks š OID Traps > Item + Trigger + snmptt config š Enums > Value Maps

$ ./mib2zabbix.pl --template –oid=.1.3.6.1.2.1.25 --name=“Host resources”

Application monitoring

š Microsoft Exchange š Microsoft SCCM š Microsoft SQL Server š Microsoft Active Directory š PostgreSQL Server š EMC Avamar š HP BPM š Squid Proxy š Zabbix Server š …

Risk mitigation

š Document in code š Source control š Clearly defined interfaces š Quality gates š Upstream contribution š Change the hiring criteria to avoid SPOF

Agent stress test

š Critical to finding:

š Memory leaks š Race conditions š Impact on system š Regressions

š Validate efficiency improvements

It’s no magic bullet…

š Data aggregation š Visualizations š Alert Scripts

Future

š Zabbix v3 upgrade š Better engagement from ITOps š More devices and apps š More automation š Better use of data š Enterprise Integration Patterns š Cloud monitor

DevOps?

š Meta-software š Agile delivery š Infrastructure As Code š Continuous Integration š Theory of Constraints

Contrib

š PostgreSQL monitoring https://github.com/cavaliercoder/libzbxpgsql š Agent benchmarking https://github.com/cavaliercoder/zabbix_agent_bench š Windows MSI package https://github.com/cavaliercoder/zabbix-msi š Golang module adapter https://github.com/cavaliercoder/g2z

Windows Counters

š Performance Counter IDs are non-persistent

š Today G: is PhysicalDisk(3 G:), tomorrow it is PhysicalDisk(5)

š Graphs and alerts break š Mapping is not practical in scripting APIs

Windows physical disks

š Mutable performance counter ID: PhysicalDisk(0 C:) š Index (‘0’) changes on reboot, swap, failover, etc. š The drive letter (‘C:’) is undocumented š MBR Signatures and GPT GUIDs are more persistent

C: Q:

Windows physical disks

š Q: What runtime counter index maps to desired MBR/GUID? š Identify via MBR Signature or GPT GUID DeviceIOControl (IOCTL_DISK_GET_DRIVE_LAYOUT_EX) š Get device index (\\.\PHYSICALDRIVE<i>) DeviceIOControl (IOCTL_STORAGE_GET_DEVICE_NUMBER) š Iterate PhysicalDisk counters (ignore drive letter) PdhEnumObjectItems

Sample code: https://github.com/cavaliercoder/sysinv/blob/master/diskinfo.cpp

Windows Volumes

š Performance counter ID: LogicalDisk(C:|HarddiskVolumeN) š Drive letter is mutable š N is mutable š Volume GUIDs or Serials are more persistent

Windows Volumes

š Q: Which runtime counter ID matches Volume GUID? š Find volumes with ID FindNextVolume š Compare GUID/Sig against name returned by GetVolumeInformation š Enumerate LogicalDisk counters with PdhEnumObjectItems š Test mount paths (N:) returned by GetVolumePathNamesForVolumeName š Test DOS Device Path (\Device\HarddiskVolumeN) returned by QueryDosDevice

Sample code: https://github.com/cavaliercoder/sysinv/blob/master/diskinfo.cpp

Windows Failover Clusters

š Disks move between nodes š Node disks are visible on cluster IP š IDs and drive letters change

Windows Failover Clusters

š Q: Is a MBR/GUID listed as a cluster resource? š Cluster API uses MBR Signature or GPT GUID! š Enumerate “Physical Disk” resources in cluster with ClusterEnum š Add a discovery parameter for clustered/non-clustered disks

Sample code: https://github.com/cavaliercoder/sysinv/blob/master/cluster.cpp

Thank you!

Ryan Armstrong

š Blog: cavaliercoder.com š Twitter: @cavaliercoder š GitHub: cavaliercoder