SLIDE 1
LTO: History and work to be done Rafael Avila de Esp ndola World - - PowerPoint PPT Presentation
LTO: History and work to be done Rafael Avila de Esp ndola World - - PowerPoint PPT Presentation
LTO: History and work to be done Rafael Avila de Esp ndola World Wide Studios - Sony Computer Entertainment 2014-04-07 Very quick history overview Very quick history overview We had llvm specific tools (llvm-nm, llvm-ar, llvm-ld, .
SLIDE 2
SLIDE 3
Very quick history overview
◮ We had llvm specific tools (llvm-nm, llvm-ar, llvm-ld, . . . )
SLIDE 4
Very quick history overview
◮ We had llvm specific tools (llvm-nm, llvm-ar, llvm-ld, . . . ) ◮ We wrote plugins for native tools (libLTO.dylib, LLVMgold.so)
SLIDE 5
Very quick history overview
◮ We had llvm specific tools (llvm-nm, llvm-ar, llvm-ld, . . . ) ◮ We wrote plugins for native tools (libLTO.dylib, LLVMgold.so) ◮ We are writing new tools (new llvm-nm and llvm-ar, lld, . . . )
SLIDE 6
Very quick history overview
◮ We had llvm specific tools (llvm-nm, llvm-ar, llvm-ld, . . . ) ◮ We wrote plugins for native tools (libLTO.dylib, LLVMgold.so) ◮ We are writing new tools (new llvm-nm and llvm-ar, lld, . . . ) ◮ Are we going back and forth?!?
SLIDE 7
Very quick history overview
◮ We had llvm specific tools (llvm-nm, llvm-ar, llvm-ld, . . . ) ◮ We wrote plugins for native tools (libLTO.dylib, LLVMgold.so) ◮ We are writing new tools (new llvm-nm and llvm-ar, lld, . . . ) ◮ Are we going back and forth?!? ◮ Yes!
SLIDE 8
Very quick history overview
◮ We had llvm specific tools (llvm-nm, llvm-ar, llvm-ld, . . . ) ◮ We wrote plugins for native tools (libLTO.dylib, LLVMgold.so) ◮ We are writing new tools (new llvm-nm and llvm-ar, lld, . . . ) ◮ Are we going back and forth?!? ◮ Yes! ◮ Probably the correct choice at every point in time.
SLIDE 9
Early tools
SLIDE 10
Early tools
◮ They were llvm only.
SLIDE 11
Early tools
◮ They were llvm only. ◮ Easy to write. Important for a small (at the time) project.
SLIDE 12
Early tools
◮ They were llvm only. ◮ Easy to write. Important for a small (at the time) project. ◮ llvm-link was “95% persent working” in October 2001.
SLIDE 13
Early tools
◮ They were llvm only. ◮ Easy to write. Important for a small (at the time) project. ◮ llvm-link was “95% persent working” in October 2001. ◮ Combine with opt and we already had LTO!
SLIDE 14
Evolution of the early tools
SLIDE 15
Evolution of the early tools
◮ Adding llvm-link, opt and llc to a build is a pain.
SLIDE 16
Evolution of the early tools
◮ Adding llvm-link, opt and llc to a build is a pain. ◮ We added gccld to help in January 2002. It hides the extra
build steps. Produces a shell script that runs lli.
SLIDE 17
Evolution of the early tools
◮ Adding llvm-link, opt and llc to a build is a pain. ◮ We added gccld to help in January 2002. It hides the extra
build steps. Produces a shell script that runs lli.
◮ Libraries can be created with llvm-ar since August 2003.
SLIDE 18
Evolution of the early tools
◮ Adding llvm-link, opt and llc to a build is a pain. ◮ We added gccld to help in January 2002. It hides the extra
build steps. Produces a shell script that runs lli.
◮ Libraries can be created with llvm-ar since August 2003. ◮ Native codegen added to gccld in September 2003).
SLIDE 19
Evolution of the early tools
◮ Adding llvm-link, opt and llc to a build is a pain. ◮ We added gccld to help in January 2002. It hides the extra
build steps. Produces a shell script that runs lli.
◮ Libraries can be created with llvm-ar since August 2003. ◮ Native codegen added to gccld in September 2003). ◮ gccld is renamed llvm-ld in r16305 September 2004.
SLIDE 20
Some limitations remained
SLIDE 21
Some limitations remained
◮ They don’t know about native object files. Have to be
conservative.
SLIDE 22
Some limitations remained
◮ They don’t know about native object files. Have to be
conservative.
◮ The archive symbol table only lists llvm members.
SLIDE 23
Some limitations remained
◮ They don’t know about native object files. Have to be
conservative.
◮ The archive symbol table only lists llvm members. ◮ To assemble and link, llvm-ld runs gcc.
SLIDE 24
Some limitations remained
◮ They don’t know about native object files. Have to be
conservative.
◮ The archive symbol table only lists llvm members. ◮ To assemble and link, llvm-ld runs gcc. ◮ Semantics not exactly right.
SLIDE 25
Some limitations remained
◮ They don’t know about native object files. Have to be
conservative.
◮ The archive symbol table only lists llvm members. ◮ To assemble and link, llvm-ld runs gcc. ◮ Semantics not exactly right. ◮ Slow.
SLIDE 26
Some limitations remained
◮ They don’t know about native object files. Have to be
conservative.
◮ The archive symbol table only lists llvm members. ◮ To assemble and link, llvm-ld runs gcc. ◮ Semantics not exactly right. ◮ Slow. ◮ Build changes are still required.
SLIDE 27
Some limitations remained
◮ They don’t know about native object files. Have to be
conservative.
◮ The archive symbol table only lists llvm members. ◮ To assemble and link, llvm-ld runs gcc. ◮ Semantics not exactly right. ◮ Slow. ◮ Build changes are still required. ◮ Hacks everywhere.
SLIDE 28
The native tools
SLIDE 29
The native tools
◮ Native tools (ld, ar, ranlib . . . ) know the command line args.
SLIDE 30
The native tools
◮ Native tools (ld, ar, ranlib . . . ) know the command line args. ◮ They know all semantic corner cases.
SLIDE 31
The native tools
◮ Native tools (ld, ar, ranlib . . . ) know the command line args. ◮ They know all semantic corner cases. ◮ They know the library format.
SLIDE 32
The native tools
◮ Native tools (ld, ar, ranlib . . . ) know the command line args. ◮ They know all semantic corner cases. ◮ They know the library format. ◮ Build systems are already using them!
SLIDE 33
The native tools
◮ Native tools (ld, ar, ranlib . . . ) know the command line args. ◮ They know all semantic corner cases. ◮ They know the library format. ◮ Build systems are already using them! ◮ They know all there is to know about LTO, except llvm IR.
SLIDE 34
The plugin era
SLIDE 35
The plugin era
◮ Idea: Use the native tools, but have them ask llvm for help.
SLIDE 36
The plugin era
◮ Idea: Use the native tools, but have them ask llvm for help. ◮ First implemented in ld64.
SLIDE 37
The plugin era
◮ Idea: Use the native tools, but have them ask llvm for help. ◮ First implemented in ld64. ◮ tools/lto added in August 2006.
SLIDE 38
The plugin era
◮ Idea: Use the native tools, but have them ask llvm for help. ◮ First implemented in ld64. ◮ tools/lto added in August 2006. ◮ gold got plugin support in September 2008.
SLIDE 39
The plugin era
◮ Idea: Use the native tools, but have them ask llvm for help. ◮ First implemented in ld64. ◮ tools/lto added in August 2006. ◮ gold got plugin support in September 2008. ◮ tools/gold added in February 2009.
SLIDE 40
The plugin era
◮ Idea: Use the native tools, but have them ask llvm for help. ◮ First implemented in ld64. ◮ tools/lto added in August 2006. ◮ gold got plugin support in September 2008. ◮ tools/gold added in February 2009. ◮ bfd (nm, ar) gets plugin support in May 2009.
SLIDE 41
The plugin era
◮ Idea: Use the native tools, but have them ask llvm for help. ◮ First implemented in ld64. ◮ tools/lto added in August 2006. ◮ gold got plugin support in September 2008. ◮ tools/gold added in February 2009. ◮ bfd (nm, ar) gets plugin support in May 2009. ◮ bfd (ld) gets plugin support in October 2010.
SLIDE 42
The plugin era
◮ Idea: Use the native tools, but have them ask llvm for help. ◮ First implemented in ld64. ◮ tools/lto added in August 2006. ◮ gold got plugin support in September 2008. ◮ tools/gold added in February 2009. ◮ bfd (nm, ar) gets plugin support in May 2009. ◮ bfd (ld) gets plugin support in October 2010. ◮ We can now do LTO by setting CFLAGS, CXXFLAGS and
LDFLAGS!
SLIDE 43
The llvm project around 2009
SLIDE 44
The llvm project around 2009
◮ llvm had become a good compiler toolkit:
SLIDE 45
The llvm project around 2009
◮ llvm had become a good compiler toolkit:
◮ High quality.
SLIDE 46
The llvm project around 2009
◮ llvm had become a good compiler toolkit:
◮ High quality. ◮ Liberal license.
SLIDE 47
The llvm project around 2009
◮ llvm had become a good compiler toolkit:
◮ High quality. ◮ Liberal license. ◮ Modular.
SLIDE 48
The llvm project around 2009
◮ llvm had become a good compiler toolkit:
◮ High quality. ◮ Liberal license. ◮ Modular. ◮ . . .
SLIDE 49
The llvm project around 2009
◮ llvm had become a good compiler toolkit:
◮ High quality. ◮ Liberal license. ◮ Modular. ◮ . . .
◮ That is not enough!
SLIDE 50
The llvm project around 2009
◮ llvm had become a good compiler toolkit:
◮ High quality. ◮ Liberal license. ◮ Modular. ◮ . . .
◮ That is not enough! ◮ These are also desirabilities of
SLIDE 51
The llvm project around 2009
◮ llvm had become a good compiler toolkit:
◮ High quality. ◮ Liberal license. ◮ Modular. ◮ . . .
◮ That is not enough! ◮ These are also desirabilities of
◮ Assemblers
SLIDE 52
The llvm project around 2009
◮ llvm had become a good compiler toolkit:
◮ High quality. ◮ Liberal license. ◮ Modular. ◮ . . .
◮ That is not enough! ◮ These are also desirabilities of
◮ Assemblers ◮ Linkers
SLIDE 53
The llvm project around 2009
◮ llvm had become a good compiler toolkit:
◮ High quality. ◮ Liberal license. ◮ Modular. ◮ . . .
◮ That is not enough! ◮ These are also desirabilities of
◮ Assemblers ◮ Linkers ◮ Debuggers
SLIDE 54
The llvm project around 2009
◮ llvm had become a good compiler toolkit:
◮ High quality. ◮ Liberal license. ◮ Modular. ◮ . . .
◮ That is not enough! ◮ These are also desirabilities of
◮ Assemblers ◮ Linkers ◮ Debuggers ◮ . . .
SLIDE 55
The llvm project around 2009
◮ llvm had become a good compiler toolkit:
◮ High quality. ◮ Liberal license. ◮ Modular. ◮ . . .
◮ That is not enough! ◮ These are also desirabilities of
◮ Assemblers ◮ Linkers ◮ Debuggers ◮ . . .
◮ We need a toolchain toolkit!
SLIDE 56
The project expands to other areas
SLIDE 57
The project expands to other areas
◮ lib/MC added in June 2009.
SLIDE 58
The project expands to other areas
◮ lib/MC added in June 2009. ◮ libcxx added in r103490 May 2010.
SLIDE 59
The project expands to other areas
◮ lib/MC added in June 2009. ◮ libcxx added in r103490 May 2010. ◮ lldb added in r105617 June 2010.
SLIDE 60
The project expands to other areas
◮ lib/MC added in June 2009. ◮ libcxx added in r103490 May 2010. ◮ lldb added in r105617 June 2010. ◮ lib/Object added in r119107 November 2010.
SLIDE 61
The project expands to other areas
◮ lib/MC added in June 2009. ◮ libcxx added in r103490 May 2010. ◮ lldb added in r105617 June 2010. ◮ lib/Object added in r119107 November 2010. ◮ lld added in r146598 December 2011 (was named lold).
SLIDE 62
The new tools and LTO
SLIDE 63
The new tools and LTO
◮ We are building a complete toolchain.
SLIDE 64
The new tools and LTO
◮ We are building a complete toolchain. ◮ Lets make sure it supports LTO!
SLIDE 65
The new tools and LTO
◮ We are building a complete toolchain. ◮ Lets make sure it supports LTO! ◮ Very different tools form the original tools:
SLIDE 66
The new tools and LTO
◮ We are building a complete toolchain. ◮ Lets make sure it supports LTO! ◮ Very different tools form the original tools:
◮ They are not llvm only. ELF, COFF and MachO work.
SLIDE 67
The new tools and LTO
◮ We are building a complete toolchain. ◮ Lets make sure it supports LTO! ◮ Very different tools form the original tools:
◮ They are not llvm only. ELF, COFF and MachO work. ◮ In archives, IR files go in the same symbol table.
SLIDE 68
The new tools and LTO
◮ We are building a complete toolchain. ◮ Lets make sure it supports LTO! ◮ Very different tools form the original tools:
◮ They are not llvm only. ELF, COFF and MachO work. ◮ In archives, IR files go in the same symbol table. ◮ Implement the same semantics and options as the native tools.
SLIDE 69
The new tools and LTO
◮ We are building a complete toolchain. ◮ Lets make sure it supports LTO! ◮ Very different tools form the original tools:
◮ They are not llvm only. ELF, COFF and MachO work. ◮ In archives, IR files go in the same symbol table. ◮ Implement the same semantics and options as the native tools. ◮ Should still be able to do LTO by setting CFLAGS,
CXXFLAGS and LDFLAGS.
SLIDE 70
Recent developments
Decided to start with llvm-ar and make it generic.
SLIDE 71
The first expected items
SLIDE 72
The first expected items
◮ lvm-ar was rewritten to use lib/Object.
SLIDE 73
The first expected items
◮ lvm-ar was rewritten to use lib/Object. ◮ Removed lib/Archive.
SLIDE 74
The first expected items
◮ lvm-ar was rewritten to use lib/Object. ◮ Removed lib/Archive. ◮ llvm-ar could then create native symbol tables!
SLIDE 75
The first expected items
◮ lvm-ar was rewritten to use lib/Object. ◮ Removed lib/Archive. ◮ llvm-ar could then create native symbol tables! ◮ And they include native object formats!
SLIDE 76
The first expected items
◮ lvm-ar was rewritten to use lib/Object. ◮ Removed lib/Archive. ◮ llvm-ar could then create native symbol tables! ◮ And they include native object formats! ◮ Pretty much done by July 2013.
SLIDE 77
The first expected items
◮ lvm-ar was rewritten to use lib/Object. ◮ Removed lib/Archive. ◮ llvm-ar could then create native symbol tables! ◮ And they include native object formats! ◮ Pretty much done by July 2013. ◮ Then I “only” needed to add IR support to lib/Object.
SLIDE 78
Yak shaving, part 1
SLIDE 79
Yak shaving, part 1
◮ The symbol table in archives uses the final (mangled) name.
SLIDE 80
Yak shaving, part 1
◮ The symbol table in archives uses the final (mangled) name. ◮ llvm-ar should not depend on lib/Target.
SLIDE 81
Yak shaving, part 1
◮ The symbol table in archives uses the final (mangled) name. ◮ llvm-ar should not depend on lib/Target. ◮ The mangler had to move from lib/Target to lib/IR.
SLIDE 82
Yak shaving, part 1
◮ The symbol table in archives uses the final (mangled) name. ◮ llvm-ar should not depend on lib/Target. ◮ The mangler had to move from lib/Target to lib/IR. ◮ And the mangling had to be specified in DataLayout.
SLIDE 83
Yak shaving, part 1
◮ The symbol table in archives uses the final (mangled) name. ◮ llvm-ar should not depend on lib/Target. ◮ The mangler had to move from lib/Target to lib/IR. ◮ And the mangling had to be specified in DataLayout. ◮ The DataLayout specs were out of sync in llvm and clang.
SLIDE 84
Yak shaving, part 1
◮ The symbol table in archives uses the final (mangled) name. ◮ llvm-ar should not depend on lib/Target. ◮ The mangler had to move from lib/Target to lib/IR. ◮ And the mangling had to be specified in DataLayout. ◮ The DataLayout specs were out of sync in llvm and clang. ◮ Move completed in Jan 2014.
SLIDE 85
Yak shaving, part 2
SLIDE 86
Yak shaving, part 2
◮ The ObjectFile interface is huge.
SLIDE 87
Yak shaving, part 2
◮ The ObjectFile interface is huge. ◮ Has to handle symbols, sections, segments, relocations, . . . .
SLIDE 88
Yak shaving, part 2
◮ The ObjectFile interface is huge. ◮ Has to handle symbols, sections, segments, relocations, . . . . ◮ For IR, only symbols make sense.
SLIDE 89
Yak shaving, part 2
◮ The ObjectFile interface is huge. ◮ Has to handle symbols, sections, segments, relocations, . . . . ◮ For IR, only symbols make sense. ◮ Added a new SymbolicFile with a simpler interface.
SLIDE 90
Yak shaving, part 2
◮ The ObjectFile interface is huge. ◮ Has to handle symbols, sections, segments, relocations, . . . . ◮ For IR, only symbols make sense. ◮ Added a new SymbolicFile with a simpler interface. ◮ An ObjectFile is a SymbolicFile.
SLIDE 91
Yak shaving, part 2
◮ The ObjectFile interface is huge. ◮ Has to handle symbols, sections, segments, relocations, . . . . ◮ For IR, only symbols make sense. ◮ Added a new SymbolicFile with a simpler interface. ◮ An ObjectFile is a SymbolicFile. ◮ Finally implemented IRObjectFile in February 2014.
SLIDE 92
Yak shaving, part 3
SLIDE 93
Yak shaving, part 3
◮ We could then mangle a GlobalValue’s name using
DataLayout!
SLIDE 94
Yak shaving, part 3
◮ We could then mangle a GlobalValue’s name using
DataLayout!
◮ But DataLayout is an ImmutablePass.
SLIDE 95
Yak shaving, part 3
◮ We could then mangle a GlobalValue’s name using
DataLayout!
◮ But DataLayout is an ImmutablePass. ◮ And llvm-ar has no pass manager :-(
SLIDE 96
Yak shaving, part 3
◮ We could then mangle a GlobalValue’s name using
DataLayout!
◮ But DataLayout is an ImmutablePass. ◮ And llvm-ar has no pass manager :-( ◮ Split DataLayout into DataLayoutPass and DataLayout.
SLIDE 97
Yak shaving, part 3
◮ We could then mangle a GlobalValue’s name using
DataLayout!
◮ But DataLayout is an ImmutablePass. ◮ And llvm-ar has no pass manager :-( ◮ Split DataLayout into DataLayoutPass and DataLayout. ◮ Split finished in February 2014.
SLIDE 98
Yak shaving, part 3
◮ We could then mangle a GlobalValue’s name using
DataLayout!
◮ But DataLayout is an ImmutablePass. ◮ And llvm-ar has no pass manager :-( ◮ Split DataLayout into DataLayoutPass and DataLayout. ◮ Split finished in February 2014. ◮ The symbol tables are now correct and include IR files!
SLIDE 99
Work to be done
SLIDE 100
Finish llvm-ar
SLIDE 101
Finish llvm-ar
◮ We need to finish it so that it can be used on any build.
SLIDE 102
Finish llvm-ar
◮ We need to finish it so that it can be used on any build. ◮ Support for non-gnu format is missing.
SLIDE 103
Finish llvm-ar
◮ We need to finish it so that it can be used on any build. ◮ Support for non-gnu format is missing. ◮ BFD format should make it possible to align objects!
SLIDE 104
Finish llvm-ar
◮ We need to finish it so that it can be used on any build. ◮ Support for non-gnu format is missing. ◮ BFD format should make it possible to align objects! ◮ Thin archives are missing.
SLIDE 105
Finish llvm-ar
◮ We need to finish it so that it can be used on any build. ◮ Support for non-gnu format is missing. ◮ BFD format should make it possible to align objects! ◮ Thin archives are missing. ◮ File handling could use open, ftruncate, mmap.
SLIDE 106
Finish llvm-ar
◮ We need to finish it so that it can be used on any build. ◮ Support for non-gnu format is missing. ◮ BFD format should make it possible to align objects! ◮ Thin archives are missing. ◮ File handling could use open, ftruncate, mmap. ◮ Hopefully no yaks.
SLIDE 107
Some inefficiencies of IR handling in llvm-ar
SLIDE 108
Some inefficiencies of IR handling in llvm-ar
◮ llvm-ar needs the name of global symbols.
SLIDE 109
Some inefficiencies of IR handling in llvm-ar
◮ llvm-ar needs the name of global symbols. ◮ llvm-ar uses lib/Object to find out.
SLIDE 110
Some inefficiencies of IR handling in llvm-ar
◮ llvm-ar needs the name of global symbols. ◮ llvm-ar uses lib/Object to find out. ◮ lib/Object doesn’t know what it is being used
for.
SLIDE 111
Some inefficiencies of IR handling in llvm-ar
◮ llvm-ar needs the name of global symbols. ◮ llvm-ar uses lib/Object to find out. ◮ lib/Object doesn’t know what it is being used
for.
◮ lib/Object uses lib/IR to find out.
SLIDE 112
Some inefficiencies of IR handling in llvm-ar
◮ llvm-ar needs the name of global symbols. ◮ llvm-ar uses lib/Object to find out. ◮ lib/Object doesn’t know what it is being used
for.
◮ lib/Object uses lib/IR to find out. ◮ lib/IR doesn’t know what it is being used for.
SLIDE 113
Some inefficiencies of IR handling in llvm-ar
◮ llvm-ar needs the name of global symbols. ◮ llvm-ar uses lib/Object to find out. ◮ lib/Object doesn’t know what it is being used
for.
◮ lib/Object uses lib/IR to find out. ◮ lib/IR doesn’t know what it is being used for. ◮ lib/IR parses the entire file.
SLIDE 114
Some inefficiencies of IR handling in llvm-ar
◮ llvm-ar needs the name of global symbols. ◮ llvm-ar uses lib/Object to find out. ◮ lib/Object doesn’t know what it is being used
for.
◮ lib/Object uses lib/IR to find out. ◮ lib/IR doesn’t know what it is being used for. ◮ lib/IR parses the entire file. ◮ Using a lazy module helps, but still reads metadata.
SLIDE 115
Making lib/Object lazier
SLIDE 116
Making lib/Object lazier
◮ Reading only the names and DataLayout is a special case.
SLIDE 117
Making lib/Object lazier
◮ Reading only the names and DataLayout is a special case. ◮ Implement it directly with lib/Bitcode.
SLIDE 118
Making lib/Object lazier
◮ Reading only the names and DataLayout is a special case. ◮ Implement it directly with lib/Bitcode. ◮ llvm-ar would never even create a Module.
SLIDE 119
Making lib/Object lazier
◮ Reading only the names and DataLayout is a special case. ◮ Implement it directly with lib/Bitcode. ◮ llvm-ar would never even create a Module. ◮ Code that needs more info can create a Module and query.
SLIDE 120
Making lib/Object lazier
◮ Reading only the names and DataLayout is a special case. ◮ Implement it directly with lib/Bitcode. ◮ llvm-ar would never even create a Module. ◮ Code that needs more info can create a Module and query.
◮ llvm-nm: Not performance critical.
SLIDE 121
Making lib/Object lazier
◮ Reading only the names and DataLayout is a special case. ◮ Implement it directly with lib/Bitcode. ◮ llvm-ar would never even create a Module. ◮ Code that needs more info can create a Module and query.
◮ llvm-nm: Not performance critical. ◮ LTO: Can delay creating a Module in some cases.
SLIDE 122
How linkers view LTO
SLIDE 123
LTO according to ld64
SLIDE 124
LTO according to ld64
◮ ld64 uses libLTO to create Modules for all IR files.
SLIDE 125
LTO according to ld64
◮ ld64 uses libLTO to create Modules for all IR files. ◮ ld64 requests the list of all symbols in each module
SLIDE 126
LTO according to ld64
◮ ld64 uses libLTO to create Modules for all IR files. ◮ ld64 requests the list of all symbols in each module
◮ Really all symbols, even uses and defs in inline assembly.
SLIDE 127
LTO according to ld64
◮ ld64 uses libLTO to create Modules for all IR files. ◮ ld64 requests the list of all symbols in each module
◮ Really all symbols, even uses and defs in inline assembly. ◮ Asks whether a symbol can be hidden form the symbol table.
SLIDE 128
LTO according to ld64
◮ ld64 uses libLTO to create Modules for all IR files. ◮ ld64 requests the list of all symbols in each module
◮ Really all symbols, even uses and defs in inline assembly. ◮ Asks whether a symbol can be hidden form the symbol table. ◮ C++ inline function whose address is not taken.
SLIDE 129
LTO according to ld64
◮ ld64 uses libLTO to create Modules for all IR files. ◮ ld64 requests the list of all symbols in each module
◮ Really all symbols, even uses and defs in inline assembly. ◮ Asks whether a symbol can be hidden form the symbol table. ◮ C++ inline function whose address is not taken. ◮ Requires loading all function bodies.
SLIDE 130
LTO according to ld64
◮ ld64 uses libLTO to create Modules for all IR files. ◮ ld64 requests the list of all symbols in each module
◮ Really all symbols, even uses and defs in inline assembly. ◮ Asks whether a symbol can be hidden form the symbol table. ◮ C++ inline function whose address is not taken. ◮ Requires loading all function bodies.
◮ Does symbol resolution.
SLIDE 131
LTO according to ld64
◮ ld64 uses libLTO to create Modules for all IR files. ◮ ld64 requests the list of all symbols in each module
◮ Really all symbols, even uses and defs in inline assembly. ◮ Asks whether a symbol can be hidden form the symbol table. ◮ C++ inline function whose address is not taken. ◮ Requires loading all function bodies.
◮ Does symbol resolution. ◮ Decides which symbols llvm should keep.
SLIDE 132
LTO according to ld64
◮ ld64 uses libLTO to create Modules for all IR files. ◮ ld64 requests the list of all symbols in each module
◮ Really all symbols, even uses and defs in inline assembly. ◮ Asks whether a symbol can be hidden form the symbol table. ◮ C++ inline function whose address is not taken. ◮ Requires loading all function bodies.
◮ Does symbol resolution. ◮ Decides which symbols llvm should keep. ◮ libLTO creates a MachO file.
SLIDE 133
LTO according to gold
SLIDE 134
LTO according to gold
◮ Requests the list of global symbols of each file.
SLIDE 135
LTO according to gold
◮ Requests the list of global symbols of each file.
◮ Defined/Undefined/Weak/. . .
SLIDE 136
LTO according to gold
◮ Requests the list of global symbols of each file.
◮ Defined/Undefined/Weak/. . . ◮ Visibility
SLIDE 137
LTO according to gold
◮ Requests the list of global symbols of each file.
◮ Defined/Undefined/Weak/. . . ◮ Visibility ◮ Comdat Key
SLIDE 138
LTO according to gold
◮ Requests the list of global symbols of each file.
◮ Defined/Undefined/Weak/. . . ◮ Visibility ◮ Comdat Key
◮ Does symbol resolution. May close some files.
SLIDE 139
LTO according to gold
◮ Requests the list of global symbols of each file.
◮ Defined/Undefined/Weak/. . . ◮ Visibility ◮ Comdat Key
◮ Does symbol resolution. May close some files. ◮ Notifies the plugin of the result for each symbol.
SLIDE 140
LTO according to gold
◮ Requests the list of global symbols of each file.
◮ Defined/Undefined/Weak/. . . ◮ Visibility ◮ Comdat Key
◮ Does symbol resolution. May close some files. ◮ Notifies the plugin of the result for each symbol.
◮ Symbol is used from an ELF or BC file.
SLIDE 141
LTO according to gold
◮ Requests the list of global symbols of each file.
◮ Defined/Undefined/Weak/. . . ◮ Visibility ◮ Comdat Key
◮ Does symbol resolution. May close some files. ◮ Notifies the plugin of the result for each symbol.
◮ Symbol is used from an ELF or BC file. ◮ Symbol is used by the symbol table.
SLIDE 142
LTO according to gold
◮ Requests the list of global symbols of each file.
◮ Defined/Undefined/Weak/. . . ◮ Visibility ◮ Comdat Key
◮ Does symbol resolution. May close some files. ◮ Notifies the plugin of the result for each symbol.
◮ Symbol is used from an ELF or BC file. ◮ Symbol is used by the symbol table. ◮ Symbol was preempted.
SLIDE 143
LTO according to gold
◮ Requests the list of global symbols of each file.
◮ Defined/Undefined/Weak/. . . ◮ Visibility ◮ Comdat Key
◮ Does symbol resolution. May close some files. ◮ Notifies the plugin of the result for each symbol.
◮ Symbol is used from an ELF or BC file. ◮ Symbol is used by the symbol table. ◮ Symbol was preempted. ◮ . . .
SLIDE 144
LTO according to gold
◮ Requests the list of global symbols of each file.
◮ Defined/Undefined/Weak/. . . ◮ Visibility ◮ Comdat Key
◮ Does symbol resolution. May close some files. ◮ Notifies the plugin of the result for each symbol.
◮ Symbol is used from an ELF or BC file. ◮ Symbol is used by the symbol table. ◮ Symbol was preempted. ◮ . . .
◮ Plugin decides which symbols to keep and creates an ELF file.
SLIDE 145
LTO according to gold
◮ Requests the list of global symbols of each file.
◮ Defined/Undefined/Weak/. . . ◮ Visibility ◮ Comdat Key
◮ Does symbol resolution. May close some files. ◮ Notifies the plugin of the result for each symbol.
◮ Symbol is used from an ELF or BC file. ◮ Symbol is used by the symbol table. ◮ Symbol was preempted. ◮ . . .
◮ Plugin decides which symbols to keep and creates an ELF file. ◮ Only now do we need a Module!
SLIDE 146
LTO according to gold
◮ Requests the list of global symbols of each file.
◮ Defined/Undefined/Weak/. . . ◮ Visibility ◮ Comdat Key
◮ Does symbol resolution. May close some files. ◮ Notifies the plugin of the result for each symbol.
◮ Symbol is used from an ELF or BC file. ◮ Symbol is used by the symbol table. ◮ Symbol was preempted. ◮ . . .
◮ Plugin decides which symbols to keep and creates an ELF file. ◮ Only now do we need a Module! ◮ Can trim the Module before IR linking!
SLIDE 147
Reorganizing lib/LTO, tools/gold and tools/lto
SLIDE 148
Reorganizing lib/LTO, tools/gold and tools/lto
◮ lib/LTO logic needed by any linker doing lto.
SLIDE 149
Reorganizing lib/LTO, tools/gold and tools/lto
◮ lib/LTO logic needed by any linker doing lto.
◮ Codegen can add calls to memcpy, don’t drop it.
SLIDE 150
Reorganizing lib/LTO, tools/gold and tools/lto
◮ lib/LTO logic needed by any linker doing lto.
◮ Codegen can add calls to memcpy, don’t drop it. ◮ MCStreamer to find asm symbols.
SLIDE 151
Reorganizing lib/LTO, tools/gold and tools/lto
◮ lib/LTO logic needed by any linker doing lto.
◮ Codegen can add calls to memcpy, don’t drop it. ◮ MCStreamer to find asm symbols.
◮ tools/gold
SLIDE 152
Reorganizing lib/LTO, tools/gold and tools/lto
◮ lib/LTO logic needed by any linker doing lto.
◮ Codegen can add calls to memcpy, don’t drop it. ◮ MCStreamer to find asm symbols.
◮ tools/gold
◮ Implemented on top of lib/Object and lib/LTO.
SLIDE 153
Reorganizing lib/LTO, tools/gold and tools/lto
◮ lib/LTO logic needed by any linker doing lto.
◮ Codegen can add calls to memcpy, don’t drop it. ◮ MCStreamer to find asm symbols.
◮ tools/gold
◮ Implemented on top of lib/Object and lib/LTO. ◮ No longer uses a C api.
SLIDE 154
Reorganizing lib/LTO, tools/gold and tools/lto
◮ lib/LTO logic needed by any linker doing lto.
◮ Codegen can add calls to memcpy, don’t drop it. ◮ MCStreamer to find asm symbols.
◮ tools/gold
◮ Implemented on top of lib/Object and lib/LTO. ◮ No longer uses a C api. ◮ Will serve to test LTO strategies.
SLIDE 155
Reorganizing lib/LTO, tools/gold and tools/lto
◮ lib/LTO logic needed by any linker doing lto.
◮ Codegen can add calls to memcpy, don’t drop it. ◮ MCStreamer to find asm symbols.
◮ tools/gold
◮ Implemented on top of lib/Object and lib/LTO. ◮ No longer uses a C api. ◮ Will serve to test LTO strategies.
◮ tools/lto
SLIDE 156
Reorganizing lib/LTO, tools/gold and tools/lto
◮ lib/LTO logic needed by any linker doing lto.
◮ Codegen can add calls to memcpy, don’t drop it. ◮ MCStreamer to find asm symbols.
◮ tools/gold
◮ Implemented on top of lib/Object and lib/LTO. ◮ No longer uses a C api. ◮ Will serve to test LTO strategies.
◮ tools/lto
◮ Free to be just the api used by ld64 again.
SLIDE 157
Metadata issues
SLIDE 158
Metadata issues
◮ We have support for lazy loading function bodies.
SLIDE 159
Metadata issues
◮ We have support for lazy loading function bodies. ◮ Debug metadata is larger than code and not lazy loaded.
SLIDE 160
Metadata issues
◮ We have support for lazy loading function bodies. ◮ Debug metadata is larger than code and not lazy loaded. ◮ It is uniqued at the context level. Many nodes end up dead.
SLIDE 161
Metadata issues
◮ We have support for lazy loading function bodies. ◮ Debug metadata is larger than code and not lazy loaded. ◮ It is uniqued at the context level. Many nodes end up dead. ◮ Making Module own metadata would help gold, hurt lld.
SLIDE 162
Metadata issues
◮ We have support for lazy loading function bodies. ◮ Debug metadata is larger than code and not lazy loaded. ◮ It is uniqued at the context level. Many nodes end up dead. ◮ Making Module own metadata would help gold, hurt lld. ◮ Lazy loading it during IR linking should be possible.
SLIDE 163
Metadata issues
◮ We have support for lazy loading function bodies. ◮ Debug metadata is larger than code and not lazy loaded. ◮ It is uniqued at the context level. Many nodes end up dead. ◮ Making Module own metadata would help gold, hurt lld. ◮ Lazy loading it during IR linking should be possible. ◮ Maybe do a GC pass? Slower than having the module own it.
SLIDE 164
Metadata issues
◮ We have support for lazy loading function bodies. ◮ Debug metadata is larger than code and not lazy loaded. ◮ It is uniqued at the context level. Many nodes end up dead. ◮ Making Module own metadata would help gold, hurt lld. ◮ Lazy loading it during IR linking should be possible. ◮ Maybe do a GC pass? Slower than having the module own it. ◮ Might still need to find a denser representation.
SLIDE 165
Making LTO useful
SLIDE 166
Making LTO useful
◮ One optimization decision can prevent other optimizations.
SLIDE 167
Making LTO useful
◮ One optimization decision can prevent other optimizations.
◮ Inlining C in B can prevent inlining B in A.
SLIDE 168
Making LTO useful
◮ One optimization decision can prevent other optimizations.
◮ Inlining C in B can prevent inlining B in A.
◮ Some optimization order is also important.
SLIDE 169
Making LTO useful
◮ One optimization decision can prevent other optimizations.
◮ Inlining C in B can prevent inlining B in A.
◮ Some optimization order is also important.
◮ Try to find loop bounds before unrolling.
SLIDE 170
Making LTO useful
◮ One optimization decision can prevent other optimizations.
◮ Inlining C in B can prevent inlining B in A.
◮ Some optimization order is also important.
◮ Try to find loop bounds before unrolling.
◮ LLVM is fairly well tuned for these.
SLIDE 171
Making LTO useful
◮ One optimization decision can prevent other optimizations.
◮ Inlining C in B can prevent inlining B in A.
◮ Some optimization order is also important.
◮ Try to find loop bounds before unrolling.
◮ LLVM is fairly well tuned for these. ◮ Under the assumption it will never see the code again :-(
SLIDE 172
Making LTO useful
◮ One optimization decision can prevent other optimizations.
◮ Inlining C in B can prevent inlining B in A.
◮ Some optimization order is also important.
◮ Try to find loop bounds before unrolling.
◮ LLVM is fairly well tuned for these. ◮ Under the assumption it will never see the code again :-( ◮ We should probably drop some passes with -flto (loop-unroll,
vectorize).
SLIDE 173
Making LTO useful
◮ One optimization decision can prevent other optimizations.
◮ Inlining C in B can prevent inlining B in A.
◮ Some optimization order is also important.
◮ Try to find loop bounds before unrolling.
◮ LLVM is fairly well tuned for these. ◮ Under the assumption it will never see the code again :-( ◮ We should probably drop some passes with -flto (loop-unroll,
vectorize).
◮ Others (inliner) should know that -flto is being used.
SLIDE 174
Profiling and LTO
SLIDE 175
Profiling and LTO
◮ We need to get basic LTO working well first.
SLIDE 176
Profiling and LTO
◮ We need to get basic LTO working well first. ◮ Lots of possibilities, have to be careful to not over engineer.
SLIDE 177
Profiling and LTO
◮ We need to get basic LTO working well first. ◮ Lots of possibilities, have to be careful to not over engineer. ◮ Should we support fat binaries?
SLIDE 178
Profiling and LTO
◮ We need to get basic LTO working well first. ◮ Lots of possibilities, have to be careful to not over engineer. ◮ Should we support fat binaries?
◮ Do a first build using the ELF half.
SLIDE 179
Profiling and LTO
◮ We need to get basic LTO working well first. ◮ Lots of possibilities, have to be careful to not over engineer. ◮ Should we support fat binaries?
◮ Do a first build using the ELF half. ◮ Do a sampling run.
SLIDE 180
Profiling and LTO
◮ We need to get basic LTO working well first. ◮ Lots of possibilities, have to be careful to not over engineer. ◮ Should we support fat binaries?
◮ Do a first build using the ELF half. ◮ Do a sampling run. ◮ Only LTO relink with the profile info.
SLIDE 181
Profiling and LTO
◮ We need to get basic LTO working well first. ◮ Lots of possibilities, have to be careful to not over engineer. ◮ Should we support fat binaries?
◮ Do a first build using the ELF half. ◮ Do a sampling run. ◮ Only LTO relink with the profile info.
◮ Partial LTO.
SLIDE 182
Profiling and LTO
◮ We need to get basic LTO working well first. ◮ Lots of possibilities, have to be careful to not over engineer. ◮ Should we support fat binaries?
◮ Do a first build using the ELF half. ◮ Do a sampling run. ◮ Only LTO relink with the profile info.
◮ Partial LTO.
◮ Just cherry pick hot functions for LTO.
SLIDE 183
Profiling and LTO
◮ We need to get basic LTO working well first. ◮ Lots of possibilities, have to be careful to not over engineer. ◮ Should we support fat binaries?
◮ Do a first build using the ELF half. ◮ Do a sampling run. ◮ Only LTO relink with the profile info.
◮ Partial LTO.
◮ Just cherry pick hot functions for LTO. ◮ And use the others from the ELF part (-ffunction-sections).
SLIDE 184
Profiling and LTO
◮ We need to get basic LTO working well first. ◮ Lots of possibilities, have to be careful to not over engineer. ◮ Should we support fat binaries?
◮ Do a first build using the ELF half. ◮ Do a sampling run. ◮ Only LTO relink with the profile info.
◮ Partial LTO.
◮ Just cherry pick hot functions for LTO. ◮ And use the others from the ELF part (-ffunction-sections).
◮ LIPO style build also possible.
SLIDE 185
Profiling and LTO
◮ We need to get basic LTO working well first. ◮ Lots of possibilities, have to be careful to not over engineer. ◮ Should we support fat binaries?
◮ Do a first build using the ELF half. ◮ Do a sampling run. ◮ Only LTO relink with the profile info.
◮ Partial LTO.
◮ Just cherry pick hot functions for LTO. ◮ And use the others from the ELF part (-ffunction-sections).
◮ LIPO style build also possible.
◮ In the second build, read some funtions from other IR files.
SLIDE 186
Profiling and LTO
◮ We need to get basic LTO working well first. ◮ Lots of possibilities, have to be careful to not over engineer. ◮ Should we support fat binaries?
◮ Do a first build using the ELF half. ◮ Do a sampling run. ◮ Only LTO relink with the profile info.
◮ Partial LTO.
◮ Just cherry pick hot functions for LTO. ◮ And use the others from the ELF part (-ffunction-sections).
◮ LIPO style build also possible.
◮ In the second build, read some funtions from other IR files. ◮ Add them as available-externally to the current TU.