Never Lose a Syslog Message Alexander Bluhm bluhm@openbsd.org - - PowerPoint PPT Presentation

▶
never lose a syslog message
SMART_READER_LITE
LIVE PREVIEW

Never Lose a Syslog Message Alexander Bluhm bluhm@openbsd.org - - PowerPoint PPT Presentation

Sep 05, 2022 398 likes •742 views

Motivation Starting Position Local Improvements Remote Logging Conclusion Never Lose a Syslog Message Alexander Bluhm bluhm@openbsd.org September 24, 2017 Motivation Starting Position Local Improvements Remote Logging Conclusion Agenda

slide-1
SLIDE 1

Motivation Starting Position Local Improvements Remote Logging Conclusion

Never Lose a Syslog Message

Alexander Bluhm

bluhm@openbsd.org

September 24, 2017

slide-2
SLIDE 2

Motivation Starting Position Local Improvements Remote Logging Conclusion

Agenda

1

Motivation

2

Starting Position

3

Local Improvements

4

Remote Logging

5

Conclusion

slide-3
SLIDE 3

Motivation Starting Position Local Improvements Remote Logging Conclusion

Why reliable logging?

system analysis attacker tries to prevent log required by common criteria

slide-4
SLIDE 4

Motivation Starting Position Local Improvements Remote Logging Conclusion

What can go wrong?

UDP for remote logs UNIX datagram for local logs file descriptors chroot environment timestamps and time zones

slide-5
SLIDE 5

Motivation Starting Position Local Improvements Remote Logging Conclusion

Agenda

1

Motivation

2

Starting Position

3

Local Improvements

4

Remote Logging

5

Conclusion

slide-6
SLIDE 6

Motivation Starting Position Local Improvements Remote Logging Conclusion

Traditional Message Flow

program syslog(LOG ERR, "message %d", 7) libc priority, timestamp, sprintf, send kernel /dev/log syslogd recv, log file, send UDP

slide-7
SLIDE 7

Motivation Starting Position Local Improvements Remote Logging Conclusion

Priority, Facility, Level, Severity, Options

  • penlog("ftpd", LOG PID|LOG CONS, LOG FTP)

syslog(LOG INFO, "%s logged in", user) #define LOG FTP (11<<3) /* ftp daemon */ #define LOG INFO 6 /* informational */ <94>Sep 24 09:35:00 ftpd[4711]: bluhm logged in

slide-8
SLIDE 8

Motivation Starting Position Local Improvements Remote Logging Conclusion

Agenda

1

Motivation

2

Starting Position

3

Local Improvements

4

Remote Logging

5

Conclusion

slide-9
SLIDE 9

Motivation Starting Position Local Improvements Remote Logging Conclusion

/dev/log

Problems with /dev/log UNIX socket needs file descriptor use LOG NDELAY reconnect after SIGHUP syslogd needs UNIX socket in chroot needs pledge("unix") LOG CONS is even worse

slide-10
SLIDE 10

Motivation Starting Position Local Improvements Remote Logging Conclusion

sendsyslog

New system call sendsyslog(2) int sendsyslog(const void *msg, size t len, int flags) sendsyslog("<94>Sep 24 09:36:23 ftpd[4711]: bluhm logged in", 47, LOG CONS)

slide-11
SLIDE 11

Motivation Starting Position Local Improvements Remote Logging Conclusion

Using sendsyslog

Syslogd does create socketpair register one end with ioctl(LIOCSFD) receive form other end Kernel does send to syslogd’s socketpair write to console if necessary ktrace if activated count errors

slide-12
SLIDE 12

Motivation Starting Position Local Improvements Remote Logging Conclusion

Error Handling

void syslog(int prio, const char *msg, ...) libc cannot return error program cannot log error Kernel sendsyslog can do it count failures when sending to syslogd write message to syslog when it works again sendsyslog: dropped 2 messages, error 57

slide-13
SLIDE 13

Motivation Starting Position Local Improvements Remote Logging Conclusion

Libc Timestamp

Timestamp from syslog(3) needs /etc/localtime in every chroot no year no time zone no indication of daylight saving time insufficient precision does not work for kernel messages Sep 24 09:37:42

slide-14
SLIDE 14

Motivation Starting Position Local Improvements Remote Logging Conclusion

Syslogd Timestamp

Timestamp added by syslogd timestamp is optional in received message syslogd adds it if missing libc does not generate it syslogd -Z generates ISO format in UTC use millisecond precision 2017-09-24T07:38:59.333Z

slide-15
SLIDE 15

Motivation Starting Position Local Improvements Remote Logging Conclusion

Logging without Libc

System call sendsyslog allows logging from signal handler at memcpy overlap from stack protector handler from ld.so dynamic linker

slide-16
SLIDE 16

Motivation Starting Position Local Improvements Remote Logging Conclusion

dmesg Overflow

Detect dmesg overflow in log file ring buffer with kernel logs syslogd reads from /dev/klog messages may overwrite special kernel message at gap <4>klog: dropped 1243 bytes, message buffer full

slide-17
SLIDE 17

Motivation Starting Position Local Improvements Remote Logging Conclusion

Agenda

1

Motivation

2

Starting Position

3

Local Improvements

4

Remote Logging

5

Conclusion

slide-18
SLIDE 18

Motivation Starting Position Local Improvements Remote Logging Conclusion

Possibilities

syslogd sender loghost receiver loghost tty user wall console memory buffer pipe file process kernel

slide-19
SLIDE 19

Motivation Starting Position Local Improvements Remote Logging Conclusion

Local Methods

syslogd process kernel sendsyslog /dev/log UNIX socket /dev/klog

slide-20
SLIDE 20

Motivation Starting Position Local Improvements Remote Logging Conclusion

Remote Methods

syslogd sender loghost receiver loghost UDP TCP TLS IPv4 IPv6 UDP TCP TLS IPv4 IPv6

slide-21
SLIDE 21

Motivation Starting Position Local Improvements Remote Logging Conclusion

UDP Format

single UDP packet max 1180 bytes <94>Sep 24 10:07:13 80.154.94.47 ftpd[4711]: bluhm logged in

slide-22
SLIDE 22

Motivation Starting Position Local Improvements Remote Logging Conclusion

TCP Format

no proper RFC 6587 new line delimiter

  • r NUL delimiter
  • r octet counting

60 <94>Sep 24 10:08:52 80.154.94.47 ftpd[4711]: bluhm logged in

slide-23
SLIDE 23

Motivation Starting Position Local Improvements Remote Logging Conclusion

TLS Format

  • ctet counting

must support 2048 bytes should support 8192 bytes libevent and libtls

slide-24
SLIDE 24

Motivation Starting Position Local Improvements Remote Logging Conclusion

Provide Server Certificate

syslogd sender loghost

  • T host:port

/etc/ssl/host.crt /etc/ssl/private/host.key syslogd must provide server certificate sender can identify syslogd attacker cannot see messages

slide-25
SLIDE 25

Motivation Starting Position Local Improvements Remote Logging Conclusion

Validate Client Certificate

syslogd sender loghost

  • T host:port

client certificate

  • K CAfile

sender may provide client certificate syslogd can identify sender attacker cannot inject messages

slide-26
SLIDE 26

Motivation Starting Position Local Improvements Remote Logging Conclusion

Validate Server Certificate

syslogd receiver loghost @tls://host:port

  • C CAfile

/etc/ssl/cert.pem server certificate syslogd must know server CA hostname must be in server certificate syslogd can identify receiver attacker cannot see messages turn off with -V

slide-27
SLIDE 27

Motivation Starting Position Local Improvements Remote Logging Conclusion

Provide Client Certificate

syslogd receiver loghost @tls://host:port

  • c cert file
  • k key file

syslogd may provide client certificate receiver can identify syslogd attacker cannot inject messages

slide-28
SLIDE 28

Motivation Starting Position Local Improvements Remote Logging Conclusion

TCP/TLS Errors

debug incoming connections log connection errors count dropped messages suppress “last message repeated” syslogd[17361]: dropped 2 messages to remote loghost

slide-29
SLIDE 29

Motivation Starting Position Local Improvements Remote Logging Conclusion

Agenda

1

Motivation

2

Starting Position

3

Local Improvements

4

Remote Logging

5

Conclusion

slide-30
SLIDE 30

Motivation Starting Position Local Improvements Remote Logging Conclusion

OpenBSD Message Flow

program syslog(LOG ERR, "message %d", 7) libc priority, sprintf, syscall kernel sendsyslog, error handling syslogd recv, timestamp, log file, send TLS

slide-31
SLIDE 31

Motivation Starting Position Local Improvements Remote Logging Conclusion

Run and Log Reliably

no fatal errors count dropped messages TCP and TLS transport libevent safe signal handlers file descriptor exhaustion privsep with re-exec pledge child and parent

slide-32
SLIDE 32

Motivation Starting Position Local Improvements Remote Logging Conclusion

Tests

180 regression tests for almost everything config, start, log, stop, check stderr, client, server, file, pipe, console, user, ktrace, fstat

slide-33
SLIDE 33

Motivation Starting Position Local Improvements Remote Logging Conclusion

TODO

initialization errors to file continue after file system full log memory buffer overflow move format from RFC 3164 to 5424 fix bug found by mpi@openbsd ivadasz@dragonfly likes kernel timestamps

slide-34
SLIDE 34

Motivation Starting Position Local Improvements Remote Logging Conclusion

Questions

?