Libabigail & ABI compatibility Taming the runtime linking - PowerPoint PPT Presentation

Libabigail & ABI compatibility Taming the runtime linking problem Ben Woodard Consulting Engineer Dodji Seketeli Tools Engineer Aug 2016

Problems due to ABI Compatibility There are several problems cause by ABI compatibility problems either real or feared. ● Linking objects compiled by different compilers not recommended ○ Unforeseen problem at runtime ○ Need to compile libs for all compilers ● Library Version problems ○ Match library version with application ○ Match compiler 2 INSERT DESIGNATOR, IF NEEDED

Lots of compilers We use a lot of compilers... ● GCC ● Intel ● PGI ● XL ● LLVM 3 INSERT DESIGNATOR, IF NEEDED

Lots of compilers We use a lot of compilers...and a lot of different version of the compilers. ● GCC ○ 4.4.7, 4.7.2, 4.8.2, 4.9.2, 5.3, 6.1.0 ● Intel ○ 9.1, 1[0-4].[01], 1[56] ● PGI ○ 7.[01], [89].0, 10.[2,9], 11.{1,10}, 12.[18], 13.{1,2,6,10}, 14.{3,7,10}, 15. {1,5,7,10}, 16.[13] ● XL ○ ??? ● LLVM ○ 3.[4-8] 4 INSERT DESIGNATOR, IF NEEDED

Runtime Linking Problems Linking objects compiled by different compilers can sometimes cause unexplained problems. Why? ● ABI is supposed to make this work properly. ○ This does work to some extent with system libraries ○ Other libraries may hit corner cases especially C++ ● We are talking about ABI not API ○ API directly affects this. 5 INSERT DESIGNATOR, IF NEEDED

Red Hat’s Problem We make API and ABI guarantees to customers ● API and ABI should not change within a major release of RHEL ● There are some specific exceptions ● Manual verification very time consuming ● Occasionally mistakes have been made. ● Fedora doesn’t want to rebuild everything all the time ● Future projects in the devops world 6 INSERT DESIGNATOR, IF NEEDED

What is ABI? • ABI is all the things that can affect inter-object linking • It is not just what is written in the standard for Architecture and language • A first approximation is ELF and DWARF • ?? 7 INSERT DESIGNATOR, IF NEEDED

Libabigail Libabigail reads ELF, DWARF, & XML to build an IR of the ABI ● ELF, DWARF -> IR of ABI -> can be saved as XML ○ Other formats theoretically possible e.g. Mach-O or Windows ○ Not just DWARF to DWARF comparison ■ Allows comparison between compilers Saved ABI XML represents all ABI artifacts ■ ○ Compare saved ABI to ELF, DWARF for new object ■ Used to maintain ABI stability ■ No need to keep old objects around. 8 INSERT DESIGNATOR, IF NEEDED

Higher level Libabigail utilities Libabigail is a C++ library and is easily to write utilities using it ● abidw - dump the ABI to stdout ○ Used to save copies ABI ○ “abidw --abidiff elfobject.so” compares an object to itself. Used to test libabigail ● abidiff - compare the ABI of one ELF or ABI XML file to another. ○ Compare two objects or saved ABI XML files. ○ “abidiff elffile1.so elffile2.so ○ “abidiff saved.abixml elffile.so 9 INSERT DESIGNATOR, IF NEEDED

Higher level Libabigail tools (cont) Libabigail is a C++ library and is easily to write utilities using it ● abicompat - does a changed ABI matter to the program ○ If the only changed functions and variables are not used the program, then it isn’t a problem. ○ Does not detect functional changes ● abipkgdiff - compares analogous objects from related packages or directories 10 INSERT DESIGNATOR, IF NEEDED

Goals: ● Eliminate the need for versions of library for each and every compiler version. ● e.g. GCC 4.4.7 4.7 4.9 5.2 … ● spack can assume rather rebuilding or discard duplicates ● Reduce the need for versions of library compiled with each compiler ● e.g. GCC vs. Intel vs. Clang… ● This is still a ways off ● Detect library API and ABI changes ● Educate developers on library versioning ● Let runtime linker do most of the work 11 INSERT DESIGNATOR, IF NEEDED

Ultimate Goals: ● Improve the quality of ELF, DWARF generated by all compilers ● Identify and resolve areas where platform ABI is not specified or incompletely specified. ● Fulfill the promise of ABI by providing safe inter-compiler cross linking ● Even for C++ ● Maybe even for Fortran 12 INSERT DESIGNATOR, IF NEEDED

Current Status This past year was foundational ● Stabilized libabigail fixing many crashes and performance pathologies. ● Now properly capturing and recording ABI artifacts. ● In the process of learning how to deal with compiler idioms ● type synonyms ● void * ● Working through issues with SPACK the build system ● Finding, reporting and/or resolving compiler bugs ● ELF bugs - improperly exported functions (PGI) ● Compiler specific internal functions in ELF (Intel) ● Type qualifiers omitted e.g. restrict (Intel) ● Double … aka DW_TAG_unspecified_parameters (GCC) ● _Bool vs. unsigned char (Intel) 13 INSERT DESIGNATOR, IF NEEDED

Upcoming work Much is left to do ● Continue to work with SPACK developers to broaden range of applications tested ● Currently mostly C ● C++ is the next big challenge ● Templates ● Very precise ABI requirements ● Fortran ● Not sure we’ve captured all ABI artifacts yet ● Performance trumps all ABI 14 INSERT DESIGNATOR, IF NEEDED

THANK YOU plus.google.com/+RedHat facebook.com/redhatinc linkedin.com/company/red-hat twitter.com/RedHatNews youtube.com/user/RedHatVideos

Compare ABI When referencing code snippets, use the template below. Resize the snippet box to the appropriate size for your text. Type code snippet here 16 INSERT DESIGNATOR, IF NEEDED