Monitoring Swift OpenStack Summit, Austin 2016 Adam Takvam, Sr. - - PowerPoint PPT Presentation

Monitoring Swift

OpenStack Summit, Austin 2016

Adam Takvam, Sr. Systems Engineer Martin Lanner, Engagement Manager April 28, 2016

Overview

• Problems
• Usage intelligence
• Capacity planning
• Operational health
• Audit trails
• Background
• Methods: logs + system metrics
• Interpretation of metrics
• Actions: thresholds + alerting
• Swift key monitoring concepts
• What to monitor?
• How to monitor
• Monitoring methods - demos
• Logging: ELK
• Trending/Forecasting:

Prometheus + Grafana

• System monitoring: Zabbix

It’s Linux!

Properties of Swift

• Distributed system
• Extremely durable through replication or Erasure Coding
• No single point of failure
• Even distribution of data
• Resilient
• Self-healing capabilities
• Can take a lot of abuse and negligence

Anatomy of a Monitoring Solution

• Agent: Gathers metrics on a host and either

pushed or advertises them

• Logstash
• Prometheus Node Exporter
• Zabbix Agent
• Nagios NRPE
• Aggregation Engines: Collects metrics from

agents and provides an API with access to aggregated metric values

• Nagios
• Zabbix
• Elasticsearch
• Prometheus
• Visualizer: Renders graphs in a human-friendly

format for easy comprehension of system state

• Kibana
• Grafana
• Alerting: Uses metric thresholds to trigger

alerts when metrics fall out of an acceptable range

• AlertManager
• PagerDuty

Forms of Monitoring

• System utilization: CPU, memory, disk

I/O, network, auditing cycles, replicator timing

• Performance: Transaction latency
• Errors: Invalid requests or states
• Outages: Service failures
• Feature usage: Understand CRUD
• perations and traffic patterns
• Audit trail: Who did what when?

Monitoring Lifecycle

• Measurement
• Reporting
• Characterization
• Thresholds
• Alerting
• Root cause analysis
• Remediation
• Manual
• Automated

Developing a Monitoring Strategy

Examples of monitoring methods

• ELK: Usage intelligence
• Who?
• Agents
• HTTP response codes
• Errors
• Audit trails
• Prometheus: Capacity planning
• Data growth
• Trending analytics
• Zabbix: Operational health
• Network
• CPU
• RAM

Key concepts for monitoring Swift

• Cluster full
• df
• Data growth
• Capacity planning
• Networking
• Availability
• Saturation
• Proxy state
• CPU
• /healthcheck
• Auditing cycles
• Replication cycle timing

Load balancer health checks against Swift proxy servers

demo@demo:~$ curl http://swift.swiftstack.oss/healthcheck OK

• Most load balancers run ICMP checks against all IPs in its pool by default
• Also, consider configuring the load balancer to run TCP checks against Swift’s

/healthcheck endpoint

Example:

Audit trails with ELK

Object size distribution

Distribution of CRUD operations over time

Zabbix triggers for Swift

Zabbix node memory usage

Zabbix drive utilization events

Disk I/O

Object Replicator Operations

Prometheus + Grafana trending and forecasting

Alerting

ALERT StorageCritical24Hours IF sum(predict_linear(node_filesystem_free{ job='swiftstack',mountpoint=~"/srv/node/.*” }[1d]), 24*3600) < sum(node_filesystem_size{ job="swiftstack",mountpoint=~"/srv/node/.*” }) * 0.2 FOR 1h LABELS { group="storage_admin“ severity="critical“ }

Translation: Send a critical alert to all members of the storage_admin group if the total available storage capacity is projected to be less than 20% of the total storage capacity within the next 24 hours and that forecast has held true for at least 1 hour, recalculating every 5 minutes (per server config / not shown).

Example:

Q&A / Demo

Thank you!