Clang provides a few options to generate timing report. Among them,-ftime-report and -ftime-trace can be used toanalyze the performance of Clang's internal passes.
-fproc-stat-report records time and memory on spawnedprocesses (ld, and gas if-fno-integrated-as).
-ftime-trace, introduced in 2019, generates Clangtiming information in the Chrome Trace Event format (JSON). The formatsupports nested events, providing a rich view of the front end.
-ftime-report: The option name is borrowed fromGCC.
This post focuses on the traditional -ftime-report,which uses a line-based textual format.
Understanding-ftime-report output
The output consists of information about multiple timer groups. Thelast group spans the largest interval and encompasses timing data fromother groups.
Up to Clang 19, the last group is called "Clang front-end timereport". You would see something like the following.
The "Clang front-end timer" timer measured the time spent inclang::FrontendAction::Execute, which includes lexing,parsing, semantic analysis, LLVM IR generation, optimization, andmachine code generation. However, "Code Generation Time" and "LLVM IRGeneration Time" belonged to the default timer group "MiscellaneousUngrouped Timers". This caused confusion for many users. For example, https://aras-p.info/blog/2019/01/12/Investigating-compile-times-and-Clang-ftime-report/elaborates on the issues.
To address the ambiguity, I revamped the output in Clang 20.
1 2 3 4 5 6 7 8 9 10 11 12
... ===-------------------------------------------------------------------------=== Clang time report ===-------------------------------------------------------------------------=== Total Execution Time: 0.7685 seconds (0.7686 wall clock)
The last group has been renamed and changed to cover a longerinterval within the invocation. It provides timing information for fourstages:
Front end: Includes lexing, parsing, semantic analysis, andmiscellnaenous tasks not captured by the subsequent timers.
LLVM IR generation: The time spent in generating LLVM IR.
LLVM IR optimization: The time consumed by LLVM's IR optimizationpipeline.
Machine code generation: The time taken to generate machine code orassembly from the optimized IR.
The -ftime-report output further elaborates on thesestages through additional groups:
"Pass execution timing report" (first instance): A subset of the"Optimizer" group, providing detailed timing for individual optimizationpasses.
"Analysis execution timing report": A subset of the first "Passexecution timing report". In LLVM's new pass manager, analyses areexecuted as part of pass invocations.
"Pass execution timing report" (second instance): A subset of the"Machine code generation" group. (This group's name should be updatedonce the legacy pass manager is no longer used for IRoptimization.)
"Instruction Selection and Scheduling": This group appears whenSelectionDAG is utilized and is part of the "Instruction Selection"timer within the second "Pass execution timing report".
When -ftime-report=per-run-pass is specified, a timer iscreated for each pass object. This can result in significant output,especially for modules with numerous functions, as each pass will bereported multiple times.
Clang internals
As clang -### -c -ftime-report shows, clangDriverforwards -ftime-report to Clang cc1. Within cc1, thisoption sets the codegen flagclang::CodeGenOptions::TimePasses. This flag enables ethuses of llvm::Timer objects to measure the execution timeof specific code blocks.
From Clang 20 onwards, the placement of the timers can be understoodthrough the following call tree.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
cc1_main ExecuteCompilerInvocation // "Front end" minus the following timers ... all kinds of initialization CompilerInstance::ExecuteAction FrontendAction::BeginSourceFile FrontendAction::Execute FrontendAction::ExecutionAction ASTFrontendAction::ExecuteAction ParseAST BackendConsumer::HandleTranslationUnit clang::emitBackendOutput EmitAssemblyHelper::emitAssembly RunOptimizationPipeline // "Optimizer" RunCodegenPipeline // "Machine code generation" FrontendAction::EndSourceFile
The measured interval does not cover the whole invocation. integratedcc1 clang -c -ftime-report a.c
LLVM internals
LLVM/lib/Support/Time.cpp implements the timer feature.Timer belongs to a TimerGroup.Timer::startTimer and Timer::stopTimergenerate a TimeRecord. Inclang/tools/driver/cc1_main.cpp,llvm::TimerGroup::printAll(llvm::errs()); dumps theseTimerGroup and TimeRecord information tostderr.
There are a few cl::opt options
sort-timers (default: true): sort the timers in a groupin descending wall time.
track-memory: record increments or decrements in mallocstatistics. In glibc 2.33 and above, this utilizesmallinfo2::unordblks.
info-output-file: dump output to the specifiedfile.
I have been busy creating posts, authoring a total of 31 blog posts(including this one). 7 posts resonated on Hacker News, garnering over50 points. (https://news.ycombinator.com/from?site=maskray.me).
I have also revised many posts initially written between 2020 and2024.
I made 5 commits to the project, including the addition of the x86inline asm constraint "Ws". you can read more about that in my earlierpost Rawsymbol names in inline assembly.
I believe that modernizing code review and test infrastructure willenhance the contributor experience and attract more contributors.
Official maintainer status on the MC layer and binary utilities
My involvement with LLVM 18 and 19
Key Points:
TODO
Added a script update_test_body.pyto generate elaborated IR and assembly tests (#89026)
MC
Made some MCand assembler improvements in LLVM 19
Fixed some intrusive changes to the generic code due to AIX andz/OS.
Made llvm-mc better as an assemblerand disassembler
Light ELF
Implementeda compact relocation format for ELF
AArch64mapping symbol size optimization
Enabled StackSafetyAnalysis for AddressSanitizer to removeinstrumentations on stack-allocated variables that are guaranteed to besafe from memory access bugs
Bail out if MemIntrinsic length is -1
Bail out when calling ifunc
Added the Clang cc1 option--output-asm-variant= and cleaned up internals of itsfriends (x86-asm-syntax).
llvm/ADT/Hashing.hstability
llvm/ADT/Hashing.h stability
To facilitate improvements, llvm/ADT/Hashing.h promisedto be non-deteriministic so that users could not depend on exact hashvalues. However, the values were actually deterministic unlessset_fixed_execution_hash_seed was called. A lot of internalcode incorrectly relied on the stability ofhash_value/hash_combine/hash_combine_range. I have fixedthem and landed https://github.com/llvm/llvm-project/pull/96282 to makethe hash value non-deteriministic inLLVM_ENABLE_ABI_BREAKING_CHECKS builds.
lld/ELF
lld/ELF is quite stable. I have made some maintenance changes. Asusual, I wrote the ELF port's release notes for the two releases. See lld 18 ELF changes and lld 19 ELF changes fordetail.
Linux kernel
Contributed 4 commits.
ccls
I finally removed support for LLVM 7, 8, and 9. The latest release https://github.com/MaskRay/ccls/releases/tag/0.20241108has some nice features.
didOpen: sort index requests. When you open A/B/foo.cc, files under"A/B/" and "A/" will be prioritized during the initial indexing process,leading to a quicker response time.
Support for older these LLVM versions 7, 8, and 9 has beendropped.
LSP semantic tokens are now supported. See usage guidehttps://maskray.me/blog/2024-10-20-ccls-and-lsp-semantic-tokens usage(including rainbow semantic highlighting)
textDocument/switchSourceHeader (LSP extension) is nowsupported.
Misc
Reported 12 feature requests or bugs to binutils.
objdump -R: dump SHT_RELR relocations?
gas arm aarch64: missing mapping symbols $d in the absence of alignment directives
gas: Extend .loc directive to emit a label
Compressed .strtab and .symtab
gas: Support \+ in .rept/.irp/.irpc directives
ld: Add CLASS to allow separate section matching and referring
gas/ld: Implicit addends for non-code sections
binutils: Support CREL relocation format
ld arm: global/weak non-hidden symbols referenced by R_ARM_FUNCDESC are unnecessarily exported
ld arm: fdpic link segfaults on R_ARM_GOTOFFFUNCDESC referencing a hidden symbol
ld arm: fdpic link may have null pointer dereference in allocate_dynrelocs_for_symbol
objcopy: add --prefix-symbols-remove
Reported 2 feature requests to glibc
Feature request: special static-pie capable of loading the interpreter from a relative path
In debuggers, stepping into a function with arguments that involvefunction calls may step into the nested function calls, even if they aresimple and uninteresting, such as those found in the C++ STL.
intmain(){ auto i = make_unique<int>(3); vector v{1,2}; foo(*i, v.back()); // step into }
When GDB stops at the foo call, the step(s) command will step into std::vector::backand std::unique_ptr::operator*. While you can executefinish (fin) and then execute sagain, it's time-consuming and distracting, especially when dealing withcomplex argument expressions.
% g++ -g a.cc -o a % gdb ./a ... (gdb) s std::vector<int, std::allocator<int> >::back (this=0x7fffffffddd0) at /usr/include/c++/14.2.1/bits/stl_vector.h:1235 1235 back() _GLIBCXX_NOEXCEPT (gdb) fin Run till exit from #0 std::vector<int, std::allocator<int> >::back (this=0x7fffffffddd0) at /usr/include/c++/14.2.1/bits/stl_vector.h:1235 0x00005555555566f8 in main () at a.cc:13 13 foo(*i, v.back()); Value returned is $1 = (__gnu_cxx::__alloc_traits<std::allocator<int>, int>::value_type &) @0x55555556c2d4: 2 (gdb) s std::unique_ptr<int, std::default_delete<int> >::operator* (this=0x7fffffffddc0) at /usr/include/c++/14.2.1/bits/unique_ptr.h:447 447 __glibcxx_assert(get() != pointer()); (gdb) fin Run till exit from #0 std::unique_ptr<int, std::default_delete<int> >::operator* (this=0x7fffffffddc0) at /usr/include/c++/14.2.1/bits/unique_ptr.h:447 0x0000555555556706 in main () at a.cc:13 13 foo(*i, v.back()); Value returned is $2 = (int &) @0x55555556c2b0: 3 (gdb) s foo (i=3, j=2) at a.cc:7 7 printf("%d %d\n", i, j);
This problem was tracked as a feature request in 2003: https://sourceware.org/bugzilla/show_bug.cgi?id=8287.Fortunately, GDB provides the skipcommand to skip functions that match a regex or filenames that matcha glob (GDB 7.12 feature). You can skip all demangled function namesthat start with std::.
1
skip -rfu ^std::
Alternatively, you can executeskip -gfi /usr/include/c++/*/bits/* to skip these libstdc++files.
Important note:
The skip command's file matching behavior uses thefnmatch function with the FNM_FILE_NAMEflag. This means the wildcard character (*) won't matchslashes. So, skip -gfi /usr/* won't exclude/usr/include/c++/14.2.1/bits/stl_vector.h.
I proposed to dropthe FNM_FILE_NAME flag. If this is accepted, I will beable to skip a project directory with
1
skip -gfi */include/llvm/ADT/*
instead of
1
skip -gfi /home/ray/llvm/llvm/include/llvm/ADT/*
User functionscalled by skipped functions
When a function (let's call it "A") is skipped during debugging, anyuser-defined functions that are called by "A" will also be skipped.
For example, consider the following code snippet:
1 2 3
std::vector<int> a{1, 2}; if (std::all_of(a.begin(), a.end(), predicate)) { }
If std::all_of is skipped due to a skipcommand, predicate called within std::all_ofwill also be skipped when you execute s at the ifstatement.
LLDB
By default, LLDB avoids stepping into functions whose names startwith std:: when you use the s(step, thread step-in) command. This behavioris controlled by a setting:
1 2 3 4
(lldb) settings show target.process.thread.step-avoid-regexp target.process.thread.step-avoid-regexp (regex) = ^std:: (lldb) set sh target.process.thread.step-avoid-libraries target.process.thread.step-avoid-libraries (file-list) =
target.process.thread.step-avoid-libraries can be usedto skip functions defined in a library.
While the command settings set is long, you can shortenit to set set.
Visual Studio
Visual Studio provides a debugging feature JustMy Code that automatically steps over calls to system,framework, and other non-user code.
It also supports a Step Into Specific command, whichseems interesting.
The implementation inserts a call to__CheckForDebuggerJustMyCode at the start of every userfunction. The function(void __CheckForDebuggerJustMyCode(const char *flag)) takesa global variable defined in the .msvcjmc section anddetermines whether the debugger should stop.
This LLDB feature request has a nice description: https://github.com/llvm/llvm-project/issues/61152.
For the all_of example, the feature can possibly allowthe debugger to stop at test.
1 2 3
std::vector<int> a{1, 2}; if (std::all_of(a.begin(), a.end(), test)) { }
Fuchsia zxdb
The Fuchsia debugger "zxdb" provides a command "ss"similar to Visual Studio's "Step Into Specific".
On https://x.com/settings/, clickMore -> Settings and privacy -> Download an archive of your data.Wait for a message from x.com: "@XXX your X data is ready" Download thearchive.
1
cp data/tweets.js tweets.ts
Change the first line from window.YTD.tweets.part0 = [to let part0 = [, and append
Both compiler developers and security researchers have builtdisassemblers. They often prioritize different aspects. Compilertoolchains, benefiting from direct contributions from CPU vendors, tendto offer more accurate and robust decoding. Security-focused tools, onthe other hand, often excel in user interface design.
For quick disassembly tasks, rizinprovides a convenient command-line interface.
1 2 3
% rz-asm -a x86 -b 64 -d 4829c390 sub rbx, rax nop
-a x86 can be omitted.
llvm-mc
Within the LLVM ecosystem, llvm-objdump serves as a drop-inreplacement for the traditional GNU objdump, leveraging instructioninformation from LLVM's TableGen files(llvm/lib/Target/*/*.td). Another LLVM tool, llvm-mc, wasoriginally designed for internal testing of the Machine Code (MC) layer,particularly the assembler and disassembler components. There arenumerous RUN: llvm-mc ... tests withinllvm/test/MC. Despite its internal origins, llvm-mc isoften distributed as part of the LLVM toolset, making it accessible tousers.
However, using llvm-mc for simple disassembly tasks can becumbersome. It requires explicitly prefixing hexadecimal byte valueswith 0x:
Let's break down the options used in this command:
--triple=x86_64: This specifies the targetarchitecture. If your LLVM build's default target triple is alreadyx86_64-*-*, this option can be omitted.
--output-asm-variant=1:LLVM, like GCC, defaults to AT&T syntax for x86 assembly. Thisoption switches to the Intel syntax. See lhmouse/mcfgthread/wiki/Intel-syntaxif you prefer the Intel syntax in compiler toolchains.
--cdis: Introduced in LLVM 18, this option enablescolored disassembly. In older LLVM versions, you have to use--disassemble.
I have contributed patches to remove.text and allow disassemblingraw bytes without the 0x prefix. You can now use the--hex option:
% disasm 4829c390 sub rbx, rax nop % disasm $'4829 c3\n# comment\n90' sub rbx, rax nop
The --hex option conveniently ignores whitespace and#-style comments within the input.
Atomic blocks
llvm-mc handles decoding failures by skipping a number of bytes, asdetermined by the target-specificllvm::MCDisassembler::getInstruction. To treat a sequenceof bytes as a single unit during disassembly, enclose them within[].
(I've contributed a change to LLVM 20 that removesthe previously printed .text directive.)
llvm-objdump
For address information, llvm-mc falls short. We need to turn tollvm-objdump to get that detail. Here is a little fish script that takesraw hex bytes as input, converts them to a binary format(xxd -r -p), and then creates an ELF relocatable file(llvm-objcopy -I binary) targeting the x86-64 architecture.Finally, llvm-objdump with the -D flag disassembles thedata section (.data) containing the converted binary.
#!/usr/bin/env fish argparse a/arch= att r -- $argv; or return 1 if test -z "$_flag_arch"; set _flag_arch x86_64; end set opt --triple=$_flag_arch if test -z "$_flag_att" && string match -rq 'i.86|x86_64' $_flag_arch; set -a opt -M intel; end if test -n "$_flag_r"; set -a opt --no-leading-addr; set -a opt --no-show-raw-insn; end
switch $_flag_arch case arm; set bfdname elf32-littlearm case aarch64; set bfdname elf64-littleaarch64 case ppc32; set bfdname elf32-powerpc case ppc32le; set bfdname elf32-powerpcle case ppc64; set bfdname elf64-powerpc case ppc64le; set bfdname elf64-powerpcle case riscv32; set bfdname elf32-littleriscv case riscv64; set bfdname elf64-littleriscv case 'i?86'; set bfdname elf32-i386 case x86_64; set bfdname elf64-x86-64 case '*'; echo unknown arch >&2; return 1 end llvm-objdump -D -j .data $opt (echo $argv | xxd -r -p | llvm-objcopy -I binary -O $bfdname - - | psub) | sed '1,/<_binary__stdin__start>:/d'
Both compiler developers and security researchers have builtdisassemblers. They often prioritize different aspects. Compilertoolchains, benefiting from direct contributions from CPU vendors, tendto offer more accurate and robust decoding. Security-focused tools, onthe other hand, often excel in user interface design.
For quick disassembly tasks, rizinprovides a convenient command-line interface.
1 2 3
% rz-asm -a x86 -b 64 -d 4829c390 sub rbx, rax nop
-a x86 can be omitted.
Within the LLVM ecosystem, llvm-objdump serves as a drop-inreplacement for the traditional GNU objdump, leveraging instructioninformation from LLVM's TableGen files(llvm/lib/Target/*/*.td). Another LLVM tool, llvm-mc, wasoriginally designed for internal testing of the Machine Code (MC) layer,particularly the assembler and disassembler components. There arenumerous RUN: llvm-mc ... tests withinllvm/test/MC. Despite its internal origins, llvm-mc isoften distributed as part of the LLVM toolset, making it accessible tousers.
However, using llvm-mc for simple disassembly tasks can becumbersome. It requires explicitly prefixing hexadecimal byte valueswith 0x:
Let's break down the options used in this command:
--triple=x86_64: This specifies the targetarchitecture. If your LLVM build's default target triple is alreadyx86_64-*-*, this option can be omitted.
--output-asm-variant=1:LLVM, like GCC, defaults to AT&T syntax for x86 assembly. Thisoption switches to the Intel syntax. See lhmouse/mcfgthread/wiki/Intel-syntaxif you prefer the Intel syntax in compiler toolchains.
--cdis: Introduced in LLVM 18, this option enablescolored disassembly. In older LLVM versions, you have to use--disassemble.
I have contributed patches to remove.text and allow disassemblingraw bytes without the 0x prefix. You can now use the--hex option:
The Google C++ Style is widely adopted by projects. It contains abrace omission guideline in Loopingand branching statements:
For historical reasons, we allow one exception to the above rules:the curly braces for the controlled statement or the line breaks insidethe curly braces may be omitted if as a result the entire statementappears on either a single line (in which case there is a space betweenthe closing parenthesis and the controlled statement) or on two lines(in which case there is a line break after the closing parenthesis andthere are no braces).
1 2 3 4 5 6 7 8 9
// OK - fits on one line. if (x == kFoo) { returnnewFoo(); }
// OK - braces are optional in this case. if (x == kFoo) returnnewFoo();
// OK - condition fits on one line, body fits on another. if (x == kBar) Bar(arg1, arg2, arg3);
In clang-format's predefined Google style for C++, there are tworelated style options:
The two options cause clang-format to aggressively join lines for thefollowing code:
1 2 3 4 5 6 7 8
for (int x : a) foo(x);
while (cond()) foo(x);
if (x) foo(x);
As a heavy debugger user, I find this behavior cumbersome.
1 2 3 4 5 6 7
// clang-format --style=Google #include<vector> voidfoo(int v){} intmain(){ std::vector<int> a{1, 2, 3}; for (int x : a) foo(x); // breakpoint }
When GDB stops at the for loop, how can I step into theloop body? Unfortunately, it's not simple.
If I run step, GDB will dive into the implementationdetail of the range-based for loop. It will stop at thestd::vector::begin function. Stepping out and executingstep again will stop at the std::vector::endfunction. Stepping out and executing step another time willstop at the operator!= function of the iterator type. Hereis an interaction example with GDB:
(gdb) n 5 for (int x : a) foo(v); (gdb) s std::vector<int, std::allocator<int> >::begin (this=0x7fffffffdcc0) at /usr/include/c++/14.2.1/bits/stl_vector.h:873 873 begin() _GLIBCXX_NOEXCEPT (gdb) fin Run till exit from #0 std::vector<int, std::allocator<int> >::begin (this=0x7fffffffdcc0) at /usr/include/c++/14.2.1/bits/stl_vector.h:873 0x00005555555561d5 in main () at a.cc:5 5 for (int x : a) foo(v); Value returned is $1 = 1 (gdb) s std::vector<int, std::allocator<int> >::end (this=0x7fffffffdcc0) at /usr/include/c++/14.2.1/bits/stl_vector.h:893 893 end() _GLIBCXX_NOEXCEPT (gdb) fin Run till exit from #0 std::vector<int, std::allocator<int> >::end (this=0x7fffffffdcc0) at /usr/include/c++/14.2.1/bits/stl_vector.h:893 0x00005555555561e5 in main () at a.cc:5 5 for (int x : a) foo(v); Value returned is $2 = 0 (gdb) s __gnu_cxx::operator!=<int*, std::vector<int, std::allocator<int> > > (__lhs=1, __rhs=0) at /usr/include/c++/14.2.1/bits/stl_iterator.h:1235 1235 { return __lhs.base() != __rhs.base(); } (gdb) fin Run till exit from #0 __gnu_cxx::operator!=<int*, std::vector<int, std::allocator<int> > > (__lhs=1, __rhs=0) at /usr/include/c++/14.2.1/bits/stl_iterator.h:1235 0x0000555555556225 in main () at a.cc:5 5 for (int x : a) foo(v); Value returned is $3 = true (gdb) s __gnu_cxx::__normal_iterator<int*, std::vector<int, std::allocator<int> > >::operator* (this=0x7fffffffdca0) at /usr/include/c++/14.2.1/bits/stl_iterator.h:1091 1091 { return *_M_current; } (gdb) fin Run till exit from #0 __gnu_cxx::__normal_iterator<int*, std::vector<int, std::allocator<int> > >::operator* (this=0x7fffffffdca0) at /usr/include/c++/14.2.1/bits/stl_iterator.h:1091 0x00005555555561f7 in main () at a.cc:5 5 for (int x : a) foo(v); Value returned is $4 = (int &) @0x55555556b2b0: 1
You can see that this can significantly hinder the debugging process,as it forces the user to delve into uninteresting function calls of therange-based for loop.
In contrast, when the loop body is on the next line, we can just runnext to skip the three uninteresting function calls:
1 2
for (int x : a) // next foo(x); // step
The AllowShortIfStatementsOnASingleLine style option issimilar. While convenient for simple scenarios, it can sometimes hinderdebuggability.
For the following code, it's not easy to skip the c()and d() function calls if you just want to step intofoo(v).
1
if (c() && d()) foo(v);
Many developers, mindful of potential goto fail-likeissues, often opt to include braces in their code. clang-format'sdefault style can further reinforce this practice.
1 2 3 4 5 6 7
// clang-format does not join lines. if (v) { foo(v); } for (int x : a) { foo(x); }
Other predefined styles
clang-format's Chromium style is a variant of the Google style anddoes not have the aforementioned problem. The LLVM style, and manystyles derived from it, do not have the problem either.
Go, Odin, and Rust require {} for if statements but omit(), striking a balance between clarity and conciseness.C/C++'s required ()` makes opt-in braces feel a bit verbose.
LLD, the LLVM linker, is a matureand fast linker supporting multiple binary formats (ELF, Mach-O,PE/COFF, WebAssembly). Designed as a standalone program, the code baserelies heavily on global state, making it less than ideal for libraryintegration. As outlined in RFC:Revisiting LLD-as-a-library design, two main hurdles exist:
Fatal errors: they exit the process without returning control to thecaller. This was actually addressed for most scenarios in 2020 byutilizing llvm::sys::Process::Exit(val, /*NoCleanup=*/true)and CrashRecoveryContext (longjmp under thehood).
Global variable conflicts: shared global variables do not allow twoconcurrent invocation.
I understand that calling a linker API could be convenient,especially when you want to avoid shipping another executable (which canbe large when you link against LLVM statically). However, I believe thatinvoking LLD as a separate process remains the recommended approach.There are several advantages:
Build system control: Build systems gain greater control overscheduling and resource allocation for LLD. In an edit-compile-linkcycle, the link could need more resources and threading is moreuseful.
Better parallelism management
Global state isolation: LLVM's global state (primarilycl::opt and ManagedStatic) is isolated.
While spawning a new process offers build system benefits, the issueof global state usage within LLD remains a concern. This is a factor toconsider, especially for advanced use cases. Here are global variablesin the LLD 15 code base.
In 2021, global variables were removed fromlld/Common.
The COFF port followed suite, eliminating most of its globalvariables.
Inspired by theseadvancements, I conceived a plan to eliminate globalvariables from the ELF port. In 2022, as part of the work to enableparallel section initialization, I introduced a classstruct Ctx to lld/ELF/Config.h. Here is myplan:
Global variables will be migrated into Ctx.
Functions will be modified to accept a new Ctx &ctxparameter.
The previously global variable lld::elf::ctx will be transformedinto a local variable within lld::elf::link.
Encapsulating globalvariables into Ctx
Over the past two years and a half, I have migrated global variablesinto the Ctx class, e.g..
diff --git a/lld/ELF/Config.h b/lld/ELF/Config.h index 590c19e6d88d..915c4d94e870 100644 --- a/lld/ELF/Config.h +++ b/lld/ELF/Config.h @@ -382,2 +382,10 @@ struct Ctx { std::atomic<bool> hasSympart{false}; + // A tuple of (reference, extractedFile, sym). Used by --why-extract=. + SmallVector<std::tuple<std::string, const InputFile *, const Symbol &>, 0> + whyExtractRecords; + // A mapping from a symbol to an InputFile referencing it backward. Used by + // --warn-backrefs. + llvm::DenseMap<const Symbol *, + std::pair<const InputFile *, const InputFile *>> + backwardReferences; }; diff --git a/lld/ELF/Driver.cpp b/lld/ELF/Driver.cpp index 8315d43c776e..2ab698c91b01 100644 --- a/lld/ELF/Driver.cpp +++ b/lld/ELF/Driver.cpp @@ -1776,3 +1776,3 @@ static void handleUndefined(Symbol *sym, const char *option) { if (!config->whyExtract.empty()) - driver->whyExtract.emplace_back(option, sym->file, *sym); + ctx->whyExtractRecords.emplace_back(option, sym->file, *sym); } @@ -1812,3 +1812,3 @@ static void handleLibcall(StringRef name) {
-void LinkerDriver::writeArchiveStats() const { +static void writeArchiveStats() { if (config->printArchiveStats.empty()) @@ -1834,3 +1834,3 @@ void LinkerDriver::writeArchiveStats() const { ++extracted[CachedHashStringRef(file->archiveName)]; - for (std::pair<StringRef, unsigned> f : archiveFiles) { + for (std::pair<StringRef, unsigned> f : driver->archiveFiles) { unsigned &v = extracted[CachedHashString(f.first)];
I did not do anything thing with the global variables in 2024. Thework was resumed in July 2024. I moved TarWriter,SymbolAux, Out, ElfSym,outputSections, etc into Ctx.
The config variable, used to store command-line options,was pervasive throughout lld/ELF. To enhance code clarity andmaintainability, I renamed it to ctx.arg (mold naming).
I've removed other instances of static storage variables throughtlld/ELF, e.g.
staticmember LinkerDriver::nextGroupId
staticmember SharedFile::vernauxNum
sectionMapin lld/ELF/Arch/ARM.cpp
Passing Ctx &ctxas parameters
The subsequent phase involved adding Ctx &ctx as aparameter to numerous functions and classes, gradually eliminatingreferences to the global ctx.
I incorporated Ctx &ctx as a member variable to afew classes (e.g. SyntheticSection,OutputSection) to minimize the modifications to memberfunctions. This approach was not suitable for Symbol andInputSection, since even a single word could increasememory consumption significantly.
-LLVM_LIBRARY_VISIBILITY extern Ctx ctx; - // The first two elements of versionDefinitions represent VER_NDX_LOCAL and diff --git a/lld/ELF/Driver.cpp b/lld/ELF/Driver.cpp index 334dfc0e3ba1..631051c27381 100644 --- a/lld/ELF/Driver.cpp +++ b/lld/ELF/Driver.cpp @@ -81,4 +81,2 @@ using namespace lld::elf;
Prior to this modification, the cleanupCallback function wasessential for resetting the global ctx when lld::elf::link was calledmultiple times.
Previously, cleanupCallback was essential for resettingthe global ctx when lld::elf::link was invokedmultiple times. With the removal of the global variable, this callbackis no longer necessary. We can now rely on the constructor to initializeCtx and avoid the need for a resetfunction.
Removing global state fromlld/Common
While significant progress has been made to lld/ELF,lld/Common needs a lot of work as well. A lot of sharedutility code (diagnostics, bump allocator) utilizes the globallld::context().
1 2 3 4 5 6 7 8 9 10
/// Returns the default error handler. ErrorHandler &errorHandler();
Although thread-local variables are an option, worker threads spawnedby llvm/lib/Support/Parallel.cpp don't inherit their valuesfrom the main thread. Given our direct access toCtx &ctx, we can leverage context-aware APIs asreplacements.
errorOrWarn(toString(f) + "xxx") =>Err(ctx) << f << "xxx"
error(toString(f) + "xxx") =>ErrAlways(ctx) << f << "xxx"
fatal("xxx") =>Fatal(ctx) << "xxx"
As of Nov 16, 2024, I have eliminatedlog/warn/error/fatal from lld/ELF.
The underlying functions lld::ErrorHandler::fatal, andlld::ErrorHandler::error when the error limit is hit andexitEarly is true, call exitLld(1).
This transformation eliminates a lot of code size overhead due tollvm::Twine. Even in the simplest Twine(123)case, the generated code needs a stack object to hold the value and aTwine kind.
lld::make from lld/include/lld/Common/Memory.his an allocation function that uses the global context. When theownership is clear, std::make_unique might be a betterchoice.
Avoid lld::make from lld/include/lld/Common/Memory.h
Avoid fatal error in a half-initialized object, e.g. fatal error ina base class constructor (ELFFileBase::init) ([LLD][COFF] When usingLLD-as-a-library, always prevent re-entrance on failures)
Global state in LLVM
LTO link jobs utilize LLVM. Understanding its global state iscrucial.
While LLVM allows for multiple LLVMContext instances tobe allocated and used concurrently, it's important to note that theseinstances share certain global states, such as cl::opt andManagedStatic. Specifically, it's not possible to run twoconcurrent LLVM compilations (including LTO link jobs) with distinctsets of cl::opt option values. To link with distinctcl::opt values, even after removing LLD's global state,you'll need to spawn a new LLD process.
Any proposal that moves away from global state seems to complicatecl::opt usage, making it impractical.
LLD also utilizes functions from llvm/Support/Parallel.hfor parallelism. These functions rely on global state likegetDefaultExecutor andllvm::parallel::strategy. Ongoing work by Alexandre Ganeaaims to make these functions context-aware. (It's nice to meet you inperson in LLVM Developers' Meeting last month)
Supported library usagescenarios
You can repeatedly call lld::lldMain from lld/Common/Driver.h.If fatal has been invoked, it will not be safe to calllld::lldMain again in certain rare scenarios. Runninglld::lldMain concurrently in two threads is notsupported.
The command LLD_IN_TEST=3 lld-link ... runs the linkprocess three times, but only the final invocation outputs diagnosticsto stdout/stderr. lld/test/lit.cfg.py has configured theCOFF port to run tests twice ([lld] Add test suite mode forrunning LLD main twice). Other ports need work to make this modework.
LLVM's C++ API doesn't offer a stability guarantee. This meansfunction signatures can change or be removed between versions, forcingprojects to adapt.
On the other hand, LLVM has an extensive API surface. When a librarylike llvm/lib/Y relies functionality from another library,the API is often exported in header files underllvm/include/llvm/X/, even if it is not intended to beuser-facing.
To be compatible with multiple LLVM versions, many projects rely on#if directives based on the LLVM_VERSION_MAJORmacro. This post explores the specific techniques used by ccls to ensurecompatibility with LLVM versions 7 to 19. For the latest release (ccls0.20241108), support for LLVM versions 7 to 9 has beendiscontinued.
Given the tight coupling between LLVM and Clang, theLLVM_VERSION_MAJOR macro can be used for both versiondetection. There's no need to checkCLANG_VERSION_MAJOR.
Changed namespaces
In Oct 2018, https://reviews.llvm.org/D52783 moved the namespaceclang::vfs to llvm::vfs. To remaincompatibility, I renamed clang::vfs uses and added aconditional namespace alias:
In March 2019, https://reviews.llvm.org/D59377 removed the membervariable VirtualFileSystem and removedsetVirtualFileSystem. To adapt to this change, ccls employsan #if.
In April 2020, the LLVM monorepo integrated a new subproject: flang.flang developers made many changes to clangDriver to reuse it for flang.https://reviews.llvm.org/D86089 changed the constructorclang::driver::Driver. I added
In November 2020, https://reviews.llvm.org/D90890 changed an argument ofComputePreambleBounds fromconst llvm::MemoryBuffer *Buffer toconst llvm::MemoryBufferRef &Buffer.
In April 2024, https://github.com/llvm/llvm-project/pull/89548/ removedllvm::StringRef::startswith in favor ofstarts_with. starts_with has been available since Oct 2022 andstartswith had been deprecated. I added the followingsnippet:
could break code that callsstd::string_view::starts_with.
Changed enumerators
In November 2023, https://github.com/llvm/llvm-project/pull/71160 changedan unnamed enumeration to a scoped enumeration. To keep the followingsnippet compiling,
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
switch (tag_d->getTagKind()) { case TTK_Struct: tag = "struct"; break; case TTK_Interface: tag = "__interface"; break; case TTK_Union: tag = "union"; break; case TTK_Class: tag = "class"; break; case TTK_Enum: tag = "enum"; break; }
The above examples illustrate how to adapt to changes in the LLVM andClang APIs. It's important to remember that API changes are a naturalpart of software development, and testing with different releases iscrucial for maintaining compatibility with a wide range of LLVMversions.
When introducing new interfaces, we should pay a lot of attention toreduce the chance that the interface will be changed in a way thatcauses disruption to the downstream. That said, changes are normal. Whenan API change is justified, do it.
Downstream projects should be mindful of the stability guarantees ofdifferent LLVM APIs. Some API may be more prone to change than others.It's essential to write code in a way that can easily adapt to changesin the LLVM API.
LLVM C API
While LLVM offers a C API with an effort made towards compatibility,its capabilities often fall short.
Clang provides a C API called libclang. Whilehighly stable, libclang's limited functionality makes it unsuitable formany tasks.
In 2018, when creating ccls (a fork of cquery), I encounteredmultiple limitations in libclang's ability to handle code completion andindexing. This led to rewriting the relevant code to leverage the ClangC++ API for a more comprehensive solution. The following commits offerinsights into how the C API and the mostly equivalent but better C++ APIworks:
Firstdraft: replace libclang indexer with clangIndex
After migrating fromVim to Emacs as my primary C++ editor in 2015, I switched from Vimto Neovim for miscellaneous non-C++ tasks as it is more convenient in aterminal. Customizing the editor with a language you are comfortablewith is important. I found myself increasingly drawn to Neovim'sterminal-based simplicity for various tasks. Recently, I've refined myNeovim setup to the point where I can confidently migrate my entire C++workflow away from Emacs.
This post explores the key improvements I've made to achieve thistransition. My focus is on code navigation.
Key mapping
I've implemented custom functions that simplify key mappings.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
localfunctionmap(mode, lhs, rhs, opts) local options = {} if opts then iftype(opts) == 'string'then opts = {desc = opts} end options = vim.tbl_extend('force', options, opts) end vim.keymap.set(mode, lhs, rhs, options) end localfunctionnmap(lhs, rhs, opts) map('n', lhs, rhs, opts) end localfunctiontmap(lhs, rhs, opts) map('t', lhs, rhs, opts) end
I've swapped ; and : for easier access toEx commands, especially since leap.nvim renders ; lessuseful for repeating ftFT.
Like many developers, I spend significantly more time reading codethan writing it. Efficiently navigating definitions and references iscrucial for productivity.
While the built-in LSP client's C-] is functional (see:h lsp-defaultstagfunc), I found it lessconvenient. Many Emacs and Neovim configurations advocate forgd. However, both G and D are placed on the left half ofthe QWERTY keyboard, making it slow to press them using the lefthand.
For years, I relied on M-j to quickly jump todefinitions.
To avoid a conflict with my recent zellij change (I adoptedM-hjkl for pane navigation), I've reassigned Jto trigger definition jumps. Although I've lost the originalJ (join lines) functionality, vJ provides asuitable workaround.
After making a LSP-based jump, the jump list can quickly fill withirrelevant entries as I navigate the codebase. Thankfully, Telescope'sLSP functionality sets push_tagstack_on_edit to push anentry to the tag stack (see :h tag-stack). To efficientlyreturn to my previous position, I've mapped H to:pop and L to :tag.
I utilize xn and xp to find the next orprevious reference. The implementation, copied from from LazyVim, onlyworks with references within the current file. I want to enable thexn map to automatically transition to the next file whenreaching the last reference in the current file.
While using Emacs, I created a hydra with x as the prefix key tocycle through next references. Unfortunately, I haven't been able toreplicate this behavior in Neovim.
1 2 3 4 5 6 7 8 9 10 11 12
;; This does not work. local Hydra = require('hydra') Hydra({ name = 'lsp xref', mode = 'n', body = 'x', heads = { {'n', function() M.lsp.words.jump(1) end}, {'p', function() M.lsp.words.jump(-1) end}, { "q", nil, { exit = true, nowait = true } }, }, })
Movement
I use leap.nvim to quickly jump to specific identifiers(s{char1}{char2}), followed by telescope.nvim to exploredefinitions and references. Somtimes, I use the following binding:
I've implemented rainbow semantic highlighting using ccls. Pleaserefer to cclsand LSP Semantic Tokens for my setup.
Other LSP features
I have configured the CursorHold event to triggertextDocument/documentHighlight. When using Emacs,lsp-ui-doc automatically requests textDocument/hover, whichI now lose.
Additionally, the LspAttach and BufEnterevents trigger textDocument/codeLens.
Window navigation
While I've been content with the traditional C-w + hjklmapping for years, I've recently opted for the more efficientC-hjkl approach.
To accommodate this change, I've shifted my tmux prefix key fromC-l to C-Space. Consequently, I've alsoadjusted my input method toggling from C-Space toC-S-Space.
Debugging
For C++ debugging, I primarily rely on cgdb. I find it superior toGDB's single-key mode and significantly more user-friendly than LLDB'sgui command.
1 2 3 4
cgdb --args ./a.out args
rr record ./a.out args rr replay -d cgdb
I typically arrange Neovim and cgdb side-by-side in tmux or zellij.During single-stepping, when encountering interesting code snippets, Ioften need to manually input filenames into Neovim. While Telescope aidsin this process, automatic file and line updates would be ideal.
Given these considerations, nvim-dap appears to be a promisingsolution. However, I haven't yet determined the configuration forintegrating rr with nvim-dap.
Live grep
Telescope's extension telescope-fzf-native is useful.
I've defined mappings to streamline directory and project-widesearches using Telescope's live grep functionality:
Additionally, I've mapped M-n to insert the word underthe cursor, mimicking Emacs Ivy'sM-n (ivy-next-history-element) behavior.
Task runner
I use overseer.nvim torun build commands like ninja -C /tmp/Debug llc llvm-mc.This plugin allows me to view build errors directly in Neovim's quickfixwindow.
Following LazyVim, I use <leader>oo to run buildsand <leader>ow to toggle the overseer window. Tonavigate errors, I use trouble.nvim with the ]q and[q keys.
nmap('<leader>oo', '<cmd>OverseerRun<cr>') nmap('<leader>ow', '<cmd>OverseerToggle<cr>') nmap('[q', function() ifrequire('trouble').is_open() then require('trouble').prev({ skip_groups = true, jump = true }) else local ok, err = pcall(vim.cmd.cprev) ifnot ok then vim.notify(err, vim.log.levels.ERROR) end end end) nmap(']q', function() ifrequire('trouble').is_open() then require('trouble').next({ skip_groups = true, jump = true }) else local ok, err = pcall(vim.cmd.cnext) ifnot ok then vim.notify(err, vim.log.levels.ERROR) end end end)
Reducing reliance onterminal multiplexer
As https://rutar.org/writing/from-vim-and-tmux-to-neovim/nicely summarizes, running Neovim under tmux has some annoyance. I'vebeen experimenting with reducing my reliance on zellij. Instead, I'llutilize more Neovim's terminal functionality.
toggleterm.nvim is a particularly useful plugin that allows me toeasily split windows, open terminals, and hide them when not in use.
The default command <C-\><C-n> (switch tothe Normal mode) is clumsy. I've mapped it to <C-s>(useless feature pausetransmission, fwd-i-search in zsh).
tmap('<C-s>', '<C-\\><C-n>') -- Binding C-/ doesn't work in tmux/zellij map({'n', 't'}, '<C-/>', '<cmd>ToggleTerm<cr>') -- This actually binds C-/ in tmux/zellij map({'n', 't'}, '<C-_>', '<cmd>ToggleTerm<cr>')
neovim-remoteallows me to open files without starting a nested Neovim process.
I use mini.sessions tomanage sessions.
Config switcher
Neovim's NVIM_APPNAMEfeature is fantastic for exploring pre-configured distributions to getinspiration.
Lua
Neovim embraces Lua 5.1 as a preferred scripting language. WhileLua's syntax is lightweight and easy to learn, it doesn't shy away fromconvenience features like func 'arg' andfunc {a=42}.
LuaJIT offers exceptional performance.
LuaJIT with the JIT enabled is much faster than all of the otherlanguages benchmarked, including Wren, because Mike Pall is a robot fromthe future. -- wren.io
This translates into noticeably smoother editing with LSP, especiallyfor hefty C++ files – a significant advantage over Emacs. With Emacs,I've always felt that editing a large C++ file is slow.
The non-default local variables and 1-based indexing(shared with languages like Awk and Julia) are annoyances that I canlive with when using a configuration language. So far, I've only neededindex-sensitive looping in one specific location.
1 2 3 4 5 6
-- For LSP semantic tokens fortype, colors inpairs(all_colors) do for i = 1,#colors do vim.api.nvim_set_hl(0, string.format('@lsp.typemod.%s.id%s.cpp', type, i-1), {fg=colors[i]}) end end
Dual-role keys
I utilize the software keyboard remapper kanata to make some keys bothas normals keys and as a modifier. I have followed the guide https://shom.dev/start/using-kanata-to-remap-any-keyboard/as the official configuration guide is intimidating.
~/.config/kanatta/config.kbdis my current configuration. A simplified version is provided below:
(defsrc tab q w e r t y u i o p [ caps a s d f g h j k l ; ' lsft z x c v b n m , . / rsft ) (deflayer default @tab _ _ _ _ _ _ _ _ _ _ _ @cap @a @s @d @f _ _ @j @k @l @; _ _ _ _ _ _ _ _ _ _ _ _ _ ) (deflayer extend _ _ _ _ lrld _ _ C-S-tab C-tab _ _ _ _ _ _ _ _ _ left down up rght _ _ _ _ _ _ _ _ home pgdn pgup end _ _ )
(defchordsv2 (j k ) esc 100 all-released () ( k l ) = 100 all-released () (j l ) S-= 100 all-released () ( l ;) - 100 all-released () )
I've spent countless hours writing and reading C++ code. For manyyears, Emacs has been my primary editor, and I leverage ccls' (my C++ languageserver) rainbow semantic highlighting feature.
The feature relies on two custom notification messages$ccls/publishSemanticHighlight and$ccls/publishSkippedRanges.$ccls/publishSemanticHighlight provides a list of symbols,each with kind information (function, type, or variable) of itself andits semantic parent (e.g. a member function's parent is a class),storage duration, and a list of ranges.
1 2 3 4 5 6 7 8 9 10 11 12 13 14
structCclsSemanticHighlightSymbol { int id = 0; SymbolKind parentKind; SymbolKind kind; uint8_t storage; std::vector<std::pair<int, int>> ranges;
std::vector<lsRange> lsRanges; // Only used by vscode-ccls };
An editor can use consistent colors to highlight differentoccurrences of a symbol. Different colors can be assigned to differentsymbols.
Tobias Pisani created emacs-cquery (the predecessor to emacs-ccls) inNov 2017. Despite not being a fan of Emacs Lisp, I added the rainbowsemantic highlighting feature for my own use in early 2018. My setupalso relied heavily on these two settings:
Bolding and underlining variables of static duration storage
Key symbol properties (member, static) were visually prominent in myEmacs environment.
My Emacs hacking days are a distant memory – beyond basicconfiguration tweaks, I haven't touched elisp code since 2018. As myElisp skills faded, I increasingly turned to Neovim for various editingtasks. Naturally, I wanted to migrate my C++ development workflow toNeovim as well. However, a major hurdle emerged: Neovim lacked thebeloved rainbow highlighting I enjoyed in Emacs.
Thankfully, Neovim supports "semantic tokens" from LSP 3.16, astandardized approach adopted by many editors.
I've made changes to ccls (available on abranch; PR)to support semantic tokens. This involves adapting the$ccls/publishSemanticHighlight code to additionally supporttextDocument/semanticTokens/full andtextDocument/semanticTokens/range.
I utilize a few token modifiers (static,classScope, functionScope,namespaceScope) for highlighting:
1 2 3 4 5
vim.cmd([[ hi @lsp.mod.classScope.cpp gui=italic hi @lsp.mod.static.cpp gui=bold hi @lsp.typemod.variable.namespaceScope.cpp gui=bold,underline ]])
treesitter, tokyonight-moon
While this approach is a significant improvement over relying solelyon nvim-treesitter, I'm still eager to implement rainbow semantictokens. Although LSP semantic tokens don't directly distinguish symbols,we can create custom modifiers to achieve similar results.
In the user-provided initialization options, I sethighlight.rainbow to 10.
ccls assigns the same modifier ID to tokens belonging to the samesymbol, aiming for unique IDs for different symbols. While we only havea few predefined IDs (each linked to a specific color), there's a slightpossibility of collisions. However, this is uncommon and generallyacceptable.
For a token with type variable, Neovim's built-in LSPplugin assigns a highlight group@lsp.typemod.variable.id$i.cpp where $i is aninteger between 0 and 9. This allows us to customize a unique foregroundcolor for each modifier ID.
local func_colors = { '#e5b124', '#927754', '#eb992c', '#e2bf8f', '#d67c17', '#88651e', '#e4b953', '#a36526', '#b28927', '#d69855', } local type_colors = { '#e1afc3', '#d533bb', '#9b677f', '#e350b6', '#a04360', '#dd82bc', '#de3864', '#ad3f87', '#dd7a90', '#e0438a', } local param_colors = { '#e5b124', '#927754', '#eb992c', '#e2bf8f', '#d67c17', '#88651e', '#e4b953', '#a36526', '#b28927', '#d69855', } local var_colors = { '#429921', '#58c1a4', '#5ec648', '#36815b', '#83c65d', '#419b2f', '#43cc71', '#7eb769', '#58bf89', '#3e9f4a', } local all_colors = { class = type_colors, constructor = func_colors, enum = type_colors, enumMember = var_colors, field = var_colors, ['function'] = func_colors, method = func_colors, parameter = param_colors, struct = type_colors, typeAlias = type_colors, typeParameter = type_colors, variable = var_colors } fortype, colors inpairs(all_colors) do for i = 1,#colors do for _, lang inpairs({'c', 'cpp'}) do vim.api.nvim_set_hl(0, string.format('@lsp.typemod.%s.id%s.%s', type, i-1, lang), {fg=colors[i]}) end end end
vim.cmd([[ hi @lsp.mod.classScope.cpp gui=italic hi @lsp.mod.static.cpp gui=bold hi @lsp.typemod.variable.namespaceScope.cpp gui=bold,underline ]])
Now, let's analyze the C++ code above using this configuration.
tokyonight-moon
While the results are visually pleasing, I need help implementingcode lens functionality.
Inactive code highlighting
Inactive code regions (skipped ranges in Clang) are typicallydisplayed in grey. While this can be helpful for identifying unusedcode, it can sometimes hinder understanding the details. I simplydisabled the inactive code feature.
1 2 3 4 5
#ifdef X ... // colorful #else ... // normal instead of grey #endif
Refresh
When opening a large project, the initial indexing or cache loadingprocess can be time-consuming, often leading to empty lists of semantictokens for the initially opened files. While ccls prioritizes indexingthese files, it's unclear how to notify the client to refresh the files.The existing workspace/semanticTokens/refresh request,unfortunately, doesn't accept text document parameters.
In contrast, with $ccls/publishSemanticHighlight, cclsproactively sends the notification after an index update (seemain_OnIndexed).
// Update indexed content, skipped ranges, and semantic highlighting. if (update->files_def_update) { auto &def_u = *update->files_def_update; if (WorkingFile *wfile = wfiles->getFile(def_u.first.path)) { wfile->setIndexContent(g_config->index.onChange ? wfile->buffer_content : def_u.second); QueryFile &file = db->files[update->file_id]; // Publish notifications to the file. emitSkippedRanges(wfile, file); emitSemanticHighlight(db, wfile, file); // But how do we send a workspace/semanticTokens/refresh request????? } } }
While the semantic token request supports partial results in thespecification, Neovim lacks this implementation. Even if it were, Ibelieve a notification message with a text document parameter would be amore efficient and direct approach.
1 2 3 4 5 6 7
exportinterfaceSemanticTokensParamsextendsWorkDoneProgressParams, PartialResultParams { /** * The text document. */ textDocument: TextDocumentIdentifier; }
Other clients
emacs-ccls
Once this feature branch is merged, Emacs users can simply remove thefollowing lines:
(setq lsp-semantic-tokens-enable t) (defface lsp-face-semhl-namespace-scope '((t :weight bold)) "highlight for namespace scope symbols":group 'lsp-semantic-tokens) (cl-loop for color in '("#429921""#58c1a4""#5ec648""#36815b""#83c65d" "#417b2f""#43cc71""#7eb769""#58bf89""#3e9f4a") for i = 0 then (1+ i) do (custom-declare-face (intern (format"lsp-face-semhl-id%d" i)) `((t :foreground ,color)) "":group 'lsp-semantic-tokens)) (setq lsp-semantic-token-modifier-faces `(("declaration" . lsp-face-semhl-interface) ("definition" . lsp-face-semhl-definition) ("implementation" . lsp-face-semhl-implementation) ("readonly" . lsp-face-semhl-constant) ("static" . lsp-face-semhl-static) ("deprecated" . lsp-face-semhl-deprecated) ("abstract" . lsp-face-semhl-keyword) ("async" . lsp-face-semhl-macro) ("modification" . lsp-face-semhl-operator) ("documentation" . lsp-face-semhl-comment) ("defaultLibrary" . lsp-face-semhl-default-library) ("classScope" . lsp-face-semhl-member) ("namespaceScope" . lsp-face-semhl-namespace-scope) ,@(cl-loop for i from 0 to 10 collect (cons (format "id%d" i) (intern (format "lsp-face-semhl-id%d" i)))) ))
vscode-ccls
We require assistance to eliminate the$ccls/publishSemanticHighlight feature and adopt built-insemantic tokens support. Due to the lack of active maintenance forvscode-ccls, I'm unable to maintain this plugin for an editor I don'tfrequently use.
Misc
I use a trick to switch ccls builds without changing editorconfigurations.
[sanitizer]Reject unsupported -static at link time
__asan_register_elf_globals:properly check the "no instrumented global variable" case
[asan,test]Disable _FORTIFY_SOURCE test incompatible with glibc 2.40
LLVM binary utilities
[llvm-readobj,ELF]Support --decompress/-z
[llvm-objcopy]Improve help messages
[llvm-readelf]Print a blank line for the first hex/string dump
[llvm-objcopy]Add --compress-sections
[llvm-readelf]Print more information for RELR
Hashing
I optimized the bit mixer used byllvm::DenseMap<std::pair<X, Y>> andllvm::DenseMap<std::tuple<X...>>.llvm/ADT/Hashing.h, used by StringRef hashingand DenseMap, was supposed to be non-deterministic. Despitethis, a lot of code relied on a specific iteration order. I mademultiple fixes across the code base and landed [Hashing] Use anon-deterministic seed if LLVM_ENABLE_ABI_BREAKING_CHECKS to improvetest coverage (e.g. assertion builds) and ensure future flexibility toreplace the algorithm.
I optimizedDenseMap::{find,erase}, yielding compile timeimprovement.
Optimizations to the bit mixer in Hashing.h and theDenseMap code have yielded significant benefits, reducingboth compile time and code size. This suggests there's further potentialfor improvement in this area.
However, the reduced code size also highlights potential significantcode size increase when considering faster unordered map implementationslike boost::unordered_flat_map,Abseil's SwissTable, and Folly'sF14. While these libraries may offer better performance, they oftencome with a significant increase in code complexity and size.
Introducing a new container alongside DenseMap toselectively replace performance-critical instances could lead tosubstantial code modifications. This approach requires carefulconsideration to balance potential performance gains with the additionalcomplexity.
NumericalStabilitySanitizer
NumericalStabilitySanitizer is a new feature for the 19.x releases. Ihave made many changes on the compiler-rt part.
Options used by the LLVM integrated assembler are currently handledin an ad-hoc way. There is deduplication with and without LTO.Eventually we might want to adopt TableGen for these -Wa,options.
I reviewed a wide range of patches, including areas like ADT/Support,binary utilities, MC, lld, clangDriver, LTO, sanitizers, LoongArch,RISC-V, and new features like NumericalStabilitySanitizer andRealTimeSanitizer.
To quantify my involvement, a search for patches I commented on(repo:llvm/llvm-project is:pr -author:MaskRay commenter:MaskRay created:>2024-01-23)yields 780 results.
LLVM 19 will be released. As usual, I maintain lld/ELF and have addedsome notes to https://github.com/llvm/llvm-project/blob/release/19.x/lld/docs/ReleaseNotes.rst.I've meticulously reviewed nearly all the patches that are not authoredby me. I'll delve into some of the key changes.
Experimental CREL relocations with explicit addends are nowsupported using the temporary section type code 0x40000020(clang -c -Wa,--crel,--allow-experimental-crel). LLVM willchange the code and break compatibility (Clang and lld of differentversions are not guaranteed to cooperate, unlike other features). CRELwith implicit addends are not supported. (#98115)
EI_OSABI in the output is now inferred from inputobject files. (#97144)
--compress-sections <section-glib>={none,zlib,zstd}[:level]is added to compress matched output sections without theSHF_ALLOC flag. (#84855) (#90567)
The default compression level for zlib is now independent of linkeroptimization level (Z_BEST_SPEED).
zstd compression parallelism no longer requiresZSTD_MULITHREAD build.
GNU_PROPERTY_AARCH64_FEATURE_PAUTH notes,R_AARCH64_AUTH_ABS64 andR_AARCH64_AUTH_RELATIVE relocations are now supported. (#72714)
--no-allow-shlib-undefined now rejects non-exporteddefinitions in the def-hidden.so ref.so case. (#86777)
--debug-names is added to create a merged.debug_names index from input .debug_namessections. Type units are not handled yet. (#86508)
--enable-non-contiguous-regions option allowsautomatically packing input sections into memory regions byautomatically spilling to later matches if a region would overflow. Thisreduces the toil of manually packing regions (typical for embedded). Italso makes full LTO feasible in such cases, since IR merging currentlyprevents the linker script from referring to input files. (#90007)
--default-script/-dT is implemented tospecify a default script that is processed if--script/-T is not specified. (#89327)
--force-group-allocation is implemented to discardSHT_GROUP sections and combine relocation sections if theirrelocated section group members are placed to the same output section.(#94704)
--build-id now defaults to generating a 20-byte digest("sha1") instead of 8-byte ("fast"). This improves compatibility withRPM packaging tools. (#93943)
-z lrodata-after-bss is implemented to place.lrodata after .bss. (#81224)
--export-dynamic no longer creates dynamic sections for-no-pie static linking.
--lto-emit-asm is now added as the canonical spellingof --plugin-opt=emit-llvm.
--lto-emit-llvm now uses the pre-codegen module. (#97480)
When AArch64 PAuth is enabled, -z pack-relative-relocsnow encodes R_AARCH64_AUTH_RELATIVE relocations in.rela.auth.dyn. (#96496)
-z gcs and -z gcs-report are now supportedfor AArch64 Guarded Control Stack extension.
-r now forces -Bstatic.
Thumb2 PLT is now supported for Cortex-M processors. (#93644)
DW_EH_sdata4 of addresses larger than 0x80000000 is nowsupported for MIPS32. (#92438)
Certain unknown section types are rejected. (#85173)
PROVIDE(lhs = rhs) PROVIDE(rhs = ...), lhsis now defined only if rhs is needed. (#74771) (#87530)
OUTPUT_FORMAT(binary) is now supported. (#98837)
NOCROSSREFS and NOCRFOSSREFS_TO commandsnow supported to prohibit cross references between certain outputsections. (#98773)
Orphan placement is refined to prefer the last similar section whenits rank <= orphan's rank. (#94099)Non-alloc orphan sections are now placed at the end. (#94519)
R_X86_64_REX_GOTPCRELX of the addq form is no longerincorrectly optimized when the address is larger than 0x80000000.
CREL
I've developed CREL (compact relocations) to reduce relocatable filetremendously for LLVM 19. LLD now supports CREL with explicit addends.Clang and lld of different versions are not guaranteed to cooperate,unlike other features.
See Integratedassembler improvements in LLVM 19 for details.
--compress-sections
The --compress-sections option has been enhanced. Youcan choose between zlib and zstd for compression, along with specifyingthe desired compression level. Looking ahead, zlib is deprecated infavor of zstd. While zstd offers additional tuning options, we onlyprovide the compression level.
My Compressedarbitrary sections has analyzed potential use cases.
Orphan sections
My Understandingorphan sections explains the changes in detail.
Linker scripts
There are quite a few enhancements to the linker script support.NOCROSSREFS and--enable-non-contiguous-regions are noteworthy newfeatures. There is now an increasing demand of features for embeddedprogramming.
The world of embedded programming is a fascinating mix of open andclosed ecosystems. Developers of proprietary hardware and closed-sourcesoftware are increasingly interested in migrating their toolchains tothe LLVM Linker (LLD). The allure of faster link speeds, a cleancodebase, and seamless LTO integration is undeniable. However, as LLD'smaintainer, I must tread carefully. While accommodating these users isnice for LLD's growth, incorporating custom linker extensions riskscompromising the project's code quality and maintainability. Strikingthe right balance between flexibility and code integrity is essential toensure LLD remains a robust and efficient linker for a wide range ofusers.
GNU ld also supports extensions for embedded programming. Icategorize these extensions into two groups: mature and experimental.Many of the established extensions exhibit well-defined semantics andhave been incorporated into LLD. However, some newer extensions in GNUld appear less thoughtfully designed and inflexible.
When considering a specific extension, we should prioritize practicalneeds over arbitrary adherence to GNU ld's implementation. If compellingreasons justify a particular feature and GNU ld's approach provesrestrictive, we should feel empowered to innovate within LLD.
Conversely, when developing new extensions, it's essential to engagewith the broader community. I often submit feature requests to GNU ld toinform decisions we are going to make. I believe this collaborativeapproach fosters knowledge sharing.
There is no performance-specific change.
In the future, we should refactorRelocationScanner::scanOne to make Arch/*.cppdrive the relocation process, removing the virtual functionoverhead.
In object files, certain code patterns embed data within instructionsor transitions occur between instruction sets. This can create hurdlesfor disassemblers, which might misinterpret data as code, resulting ininaccurate output. Furthermore, code written for one instruction setcould be incorrectly disassembled as another. To address these issues,some architectures (Arm, C-SKY, NDS32, RISC-V, etc) define mappingsymbols to explicitly denote state transition. Let's explore thisconcept using an AArch32 code example:
.thumb .type thumb_callee, %function thumb_callee: bx lr
Jump tables (.LJTI0_0): Jump tables canreside in either data or text sections, each with its trade-offs. Herewe see a jump table in the text section(MachineJumpTableInfo::EK_Inline in LLVM), allowing asingle instruction to take its address. Other architectures generallyprefer to place jump tables in data sections. While avoiding data incode, RISC architectures typically require two instructions tomaterialize the address, since text/data distance can be prettylarge.
Constant pool (.LCPI0_0): Thevldr instruction loads a 16-byte floating-point literal tothe SIMD&FP register.
ISA transition: This code blends A32 and T32instructions (the latter used in thumb_callee).
In these cases, a dumb disassembler might treat data as code and trydisassembling them as instructions. Assemblers create mapping symbols toassist disassemblers. For this example, the assembled object file lookslike the following:
Now, let's delve into how mapping symbols are managed within thetoolchain.
Disassemblers
llvm-objdump sorts symbols, including mapping symbols, relative tothe current section, presenting interleaved labels and instructions.Mapping symbols act as signals for the disassembler to switchstates.
00000028 <$t.4>: 00000028 <thumb_callee>: 28: 4770 bx lr
I changed llvm-objdump18 to not display mapping symbols as labels unless--show-all-symbols is specified.
nm
Both llvm-nm and GNU nm typically conceal mapping symbols alongsideSTT_FILE and STT_SECTION symbols. However, youcan reveal these special symbols using the --special-symsoption.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
% cat a.s foo: bl thumb_callee .long 42 .thumb thumb_callee: bx lr % clang --target=arm-linux-gnueabi -c a.s % llvm-nm a.o 00000000 t foo 00000008 t thumb_callee % llvm-nm --special-syms a.o 00000000 t $a.0 00000004 t $d.1 00000008 t $t.2 00000000 t foo 00000008 t thumb_callee
GNU nm behaves similarly, but with a slight quirk. If the default BFDtarget isn't AArch32, mapping symbols are displayed even without--special-syms.
1 2 3 4 5 6 7 8 9
% arm-linux-gnueabi-nm a.o 00000000 t foo 00000008 t thumb_callee % nm a.o 00000000 t $a.0 00000004 t $d.1 00000008 t $t.2 00000000 t foo 00000008 t thumb_callee
Symbolizers
Mapping symbols, being non-unique and lacking descriptive names, areintentionally omitted by symbolizers like addr2line and llvm-symbolizer.Their primary role lies in guiding the disassembly process rather thanproviding human-readable context.
Size problem: symbol tablebloat
While mapping symbols are useful, they can significantly inflate thesymbol table, particularly in 64-bit architectures(sizeof(Elf64_Sym) == 24) with larger programs. This issuebecomes more pronounced when using-ffunction-sections -fdata-sections, which generatesnumerous small sections.
Symbol table '.symtab' contains 16 entries: Num: Value Size Type Bind Vis+Other Ndx(SecName) Name [+ Version Info] 0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND 1: 0000000000000000 0 FILE LOCAL DEFAULT ABS a.c 2: 0000000000000000 0 SECTION LOCAL DEFAULT 3 (.text.f0) .text.f0 3: 0000000000000000 0 NOTYPE LOCAL DEFAULT 3 (.text.f0) $x 4: 0000000000000000 0 SECTION LOCAL DEFAULT 4 (.text.f1) .text.f1 5: 0000000000000000 0 NOTYPE LOCAL DEFAULT 4 (.text.f1) $x 6: 0000000000000000 0 SECTION LOCAL DEFAULT 5 (.text.f2) .text.f2 7: 0000000000000000 0 NOTYPE LOCAL DEFAULT 5 (.text.f2) $x 8: 0000000000000000 0 NOTYPE LOCAL DEFAULT 6 (.data) $d 9: 0000000000000000 0 NOTYPE LOCAL DEFAULT 7 (.comment) $d 10: 0000000000000000 0 NOTYPE LOCAL DEFAULT 9 (.eh_frame) $d 11: 0000000000000000 4 FUNC GLOBAL DEFAULT 3 (.text.f0) f0 12: 0000000000000000 4 FUNC GLOBAL DEFAULT 4 (.text.f1) f1 13: 0000000000000000 4 FUNC GLOBAL DEFAULT 5 (.text.f2) f2 14: 0000000000000000 4 OBJECT GLOBAL DEFAULT 6 (.data) d1 15: 0000000000000004 4 OBJECT GLOBAL DEFAULT 6 (.data) d2
Except the trivial cases (e.g. empty section), in both GNU assemblerand LLVM integrated assemble's AArch64 ports:
A non-text section (data, debug, etc) almost always starts with aninitial $d.
A text section almost always starts with an initial $x.ABI requires a mapping symbol at offset 0.
The behaviors ensure that each function or data symbol has acorresponding mapping symbol, while extra mapping symbols might occur inrare cases. Thereore, the number of mapping symbols in the output symboltable usually exceeds 50%.
Most text sections have 2 or 3 symbols:
A STT_FUNC symbol.
A STT_SECTION symbol due to a referenced from.eh_frame. This symbol is absent if-fno-asynchronous-unwind-tables.
A $x mapping symbol.
During the linking process, the linker combines input sections andeliminates STT_SECTION symbols.
Note: LLVM integrated assemblers used to create unique$x.<digit> due to an assembler limitation. I haveupdated LLVM 19 to drop.<digit> suffixes.
In LLVM's ARM port, data sections do not have mapping symbols, unlessthere are A32 or T32 instructions (D30724).
Alternative mapping symbolscheme
I have proposed an alternaive scheme to address the size concern.
Text sections: Assume an implicit $x at offset 0. Addan ending $x if the final data isn't instructions.
Non-text sections: Assume an implicit $d at offset 0.Add an ending $d only if the final data isn't datadirectives.
This approach eliminates most mapping symbols while ensuring correctdisassembly. Here is an illustrated assembler example:
1 2 3 4 5 6 7 8
.section .text.f0,"ax" ret // emit $d .long 42 // emit $x. Without this, .text.f1 might be interpreted as data.
.section .text.f1,"ax" ret
The ending mapping symbol is to ensure the subsequent section in thelinker output starts with the desired state. The data in code case isextremely rare for AArch64 as jump tables are placed in.rodata.
Impressive results
I have developed a LLVM patches to add an opt-in optionclang -Wa,-mmapsyms=implicit. Experiments with a Clangbuild using this alternative scheme have shown impressive results,eliminating over 50% of symbol table entries.
However, omitting a mapping symbol at offset 0 for sections withinstructions is currently non-conformant. An ABI update has been requestedto address this, though it unlikely has an update in the near term dueto lack of GNU toolchain support and interoperability concern.
I'll elaborate interoperability concerns below. Note, they'reunlikely to impact a majority of users.
For instance, if a text section with trailing data is assembled usingthe traditional behavior, the last mapping symbol will be$d. When linked with another text section assembled usingthe new behavior (lacking an initial $x), disassemblersmight misinterpret the start of the latter section as data.
Similarly, linker scripts that combine non-text and text sectionscould lead to text sections appearing in a data state.
However, many developers would classify these scenarios as errorconditions.
A text section may rarely start with data directives (e.g.,-fsanitize=function, LLVM prefixdata). When the linker combines two such sections, the ending$x of the first section and the initial $d ofthe second might have the same address.
1 2 3 4 5 6 7 8 9
.section .text.0, "ax" // $d .word 0 // $x this
.section .text.1, "ax" // $d may have the same address .word 0 // $x
In a straightforward implementation, symbols are stable-sorted byaddress and the last symbol at an address wins. Ideally we want$d $x $d $x. If the sections are in different files, alinker that respects input order will naturally achieves this. Ifthey're in the same file, the assembler should output$d $x $d $x instead of $d $d $x $x. This worksif .text.0 precedes .text.1 in the linkeroutput, but the other section order might be unexpected. In the worstcase where the linker's section order mismatches the assembler's sectionorder (--symbol-ordering-file=,--call-graph-profile-sort, linker scripts), the initialdata directives could be mistakenly identified as code. But thefollowing code won't, making this an acceptable risk for certainusers.
Teaching linkers to scan and insert missing mapping symbols istechnically possible but inelegant and impactsperformance. There's a strong emphasis on the philosophy of "smartformat, dumb linker," which favors keeping the format itself intelligentand minimizing the complexity of the linker.
Ultimately, the proposed alternative scheme effectively addressessymbol table bloat, but requires careful consideration for complianceand interoperability. With this optimization enabled, the remainingsymbols would primarily stem from range extension thunks, prebuiltlibraries, or highly specialized assembly code.
Mapping symbols forrange extension thunks
When lld creates an AArch64range extension thunk, it defines a $x symbol tosignify the A64 state. This symbol is only relevant when the precedingsection ends with the data state, a scenario that's only possible withthe traditional assembler behavior.
Given the infrequency of range extension thunks, the $xsymbol overhead is generally tolerable.
Peculiar alignmentbehavior in GNU assembler
In contrast to LLVM's integrated assembler, which restricts statetransitions to instructions and data directives, GNU assemblerintroduces additional state transitions for alignments. These alignmentscan be either implicit (arising from alignment requirements) or explicit(specified through directives). This behavior has led to someinteresting edge cases and bug fixes over time. (See related code beside[PATCH][GAS][AARCH64]Fix"align directive causes MAP_DATA symbol to be lost"https://sourceware.org/bugzilla/show_bug.cgi?id=20364)
1 2 3 4 5 6 7 8 9 10 11 12 13 14
.section .foo1,"a" // no $d .word 0
.section .foo2,"a" // $d .balign 4 .word 0
.section .foo3,"a" // $d .word 0 // $a nop
In the example, .foo1 only contains data directives andthere is no $d. However, .foo2 includes analignment directive, triggering the creation of a $dsymbol. Interestingly, .foo3 starts with data but ends withan instruction, necessitating both a $d and an$a mapping symbol.
It's worth noting that DWARF sections, typically generated by thecompiler, don't include explicit alignment directives. They behavesimilarly to the .foo1 example and lack an associated$d mapping symbol.
AArch32 ld --be8
The BE-8 mode (byte-invariant addressing big-endian mode) requiresthe linker to convert big-endian code to little-endian. This is implemented by scanningmapping symbols. See Linker notes onAArch32#--be8 for context.
RISC-V ISA extension
RISC-V mapping symbols are similar to AArch64, but with a notableextension:
1 2
$x<ISA> | Start of a sequence of instructions with <ISA> extension. $x<ISA>.<any>
The alternative scheme for optimizing symbol table size can beadapted to accommodate RISC-V's $x<ISA> symbols. Theapproach remains the same: add an ending $x<ISA> onlyif the final data in a text section doesn't belong to the desiredISA.
The alternative scheme can be adapted to work with$x<ISA>: Add an ending $x<ISA> ifthe final data isn't of the desired ISA.
This adaptation works seamlessly as long as all relocatable filesprovided to the linker share the same baseline ISA. However, inscenarios where the relocatable files are more heterogeneous, a crucialquestion arises: which state should be restored at section end? Wouldthe subsequent section in the linker output be compiled with differentISA extensions?
Technically, we could teach linkers to insert $xsymbols, but scanning each input text section isn't elegant.
Mach-OLC_DATA_IN_CODE load command
In contrast to ELF's symbol pair approach, Mach-O employs theLC_DATA_IN_CODE load command to store non-instructionranges within code sections. This method is remarkably compact, witheach entry requiring only 8 bytes. ELF, on the other hand, needs twosymbols ($d and $x) per data region, consuming48 bytes (in ELFCLASS64) in the symbol table.
1 2 3 4 5
structdata_in_code_entry { uint32_t offset; /* from mach_header to start of data range*/ uint16_t length; /* number of bytes in data range */ uint16_t kind; /* a DICE_KIND_* value */ };
In llvm-project, the possible kind values are defined inllvm/include/llvm/BinaryFormat/MachO.h. I recentlyrefactored the generic MCAssembler to place this Mach-Ospecific thing, alongside others, to MachObjectWriter.
1 2 3 4 5 6 7 8
enumDataRegionType { // Constants for the "kind" field in a data_in_code_entry structure DICE_KIND_DATA = 1u, DICE_KIND_JUMP_TABLE8 = 2u, DICE_KIND_JUMP_TABLE16 = 3u, DICE_KIND_JUMP_TABLE32 = 4u, DICE_KIND_ABS_JUMP_TABLE32 = 5u };
Achieving Mach-O'sefficiency in ELF
Given ELF's symbol table bloat due to the st_size member(myprevious analysis), how can it attain Mach-O's level of efficiency?Instead of introducing a new format, we can leverage the standard ELFfeature: SHF_COMPRESSED.
Both .symtab and .strtab lack theSHF_ALLOC flag, making them eligible for compressionwithout requiring any changes to the ELF specification.
LLVMdiscussion
A featurerequest has already been submitted to binutils to explore thispossibility.
The implementation within LLVM shouldn't be overly complex, and I'mmore than willing to contribute if there's interest from thecommunity.
The output of ld.lld -v includes a message "compatiblewith GNU linkers" to address detectionmechanism used by GNU Libtool. This problem is described by Softwarecompatibility and our own "User-Agent" problem.
The latest m4/libtool.m4 continues to rely on aGNU check.
1 2 3 4 5 6 7 8 9 10
[AC_CACHE_CHECK([if the linker ($LD) is GNU ld], lt_cv_prog_gnu_ld, [# I'd rather use --version here, but apparently some GNU lds only accept -v. case `$LD -v 2>&1 </dev/null` in *GNU* | *'with BFD'*) lt_cv_prog_gnu_ld=yes ;; *) lt_cv_prog_gnu_ld=no ;; esac])
Check-basedconfiguration can be a valuable tool, ensuring software remainsfunctional in the future. However, this example highlights how overlyspecific checks can lead to unintended consequences.
If Libtool needs to check whether certain options are available, itcan utilize -v.
1 2 3 4 5 6 7
% ld.bfd -v --whole-archive GNU ld (GNU Binutils) 2.42.0 % ld.bfd -v --whole-archivex; echo $? GNU ld (GNU Binutils) 2.42.0 ld.bfd: unrecognized option '--whole-archivex' ld.bfd: use the --help option for usage information 1
This blog post explores more forms of the "User-Agent" problemexposed by an LLD patch changing the version message format.
LLD supports many object file formats. It largely emulates thebehavior of GNU ld for ELF, while emulating the behavior of MSVClink.exe for PE/COFF. Previously, LLD's ELF port displays the versioninformation like this:
1 2
% /tmp/out/custom2/bin/ld.lld --version LLD 19.0.0 (compatible with GNU linkers)
A recent patch (llvm-project#97323)changed it to one of the following formats, depending on the build-timevariable LLVM_APPEND_VC_REV:
With LLVM_APPEND_VC_REV=on:
1 2
% /tmp/out/custom2/bin/ld.lld --version LLD 19.0.0 (git@github.com:llvm/llvm-project.git 0f9fbbb63cfcd2069441aa2ebef622c9716f8dbb), compatible with GNU linkers
With LLVM_APPEND_VC_REV=off:
1 2
% /tmp/out/custom2/bin/ld.lld --version LLD 19.0.0, compatible with GNU linkers
Meson
In Meson, mesonbuild/linkers/detect.py:guess_win_linkerchecks the --version output to determine whether the LLDinvocation is for ELF or PE/COFF. It performed an overly strict check"(compatible with GNU linkers)", which failed when the parentheses werestripped by #97323.
The latest Meson has loosened the check (meson#13383).
It seems that the linker detection has a larger problem that--target= is not taken into account with Clang (#6662).
Linux kernel
The Linux kernel's scripts/ld-version.sh script detectslinker versions. Introduced in 2014, it initially checked for GNU ldcompatibility with GCC LTO (though LTO support remains unmerged). It waslater revamped to handle LLD versions as well. While it can handlesuffixes like 2.34-4.fc32, it struggles with versionscontaining with comma suffix (19.0.0,).
The script extracts the version string from the--version output and parses it as major.minor.patch.
1 2 3 4 5 6 7 8 9 10 11 12 13 14
# Get the first line of the --version output. IFS=' ' set -- $(LC_ALL=C "$@" --version)
# Split the line on spaces. IFS=' ' set -- $1
...
# Some distributions append a package release number, as in 2.34-4.fc32 # Trim the hyphen and any characters that follow. version=${version%-*}
To support suffixes starting with either - or,, the script willemploy a POSIX shell trick utilizing the "Remove Largest SuffixPattern" feature:
1
version=${version%%[!0-9.]*}
More fun with versions
llvm-nm and llvm-objcopy also claim GNU compatibility.
1 2 3 4 5 6 7 8 9 10
% /tmp/Rel/bin/llvm-nm --version llvm-nm, compatible with GNU nm LLVM (http://llvm.org/): LLVM version 19.0.0git Optimized build with assertions. % /tmp/Rel/bin/llvm-objcopy --version llvm-objcopy, compatible with GNU objcopy LLVM (http://llvm.org/): LLVM version 19.0.0git Optimized build with assertions.
Ever wondered what the subtle differences are between-v, -V, and --version when usingGNU ld? Let's break it down:
--version skips linker input processing and displaysbrief copyright information.
-v and -V keep processing command linearguments and perfoming a linking step. This behavior gives an easy wayto check whether an option is supported.
-V goes a step further than -v byincluding a list of supported BFD emulations alongside the versioninformation.
Prior to September 2022, -V in ld.lld used to an aliasfor --version. This caused issues when usinggcc -v -fuse-ld=lld on certain targets like*-freebsd and powerpc-*: gcc passes -V to thelinker, expecting it to process the input files and complete the linkingstep. However, ld.lld's behavior with -V skipped thisprocess.
I made an adjustment by making-V an alias for -v instead. This ensuresthat gcc -v -fuse-ld=lld performs the linking step.
GCC has a similar -v and --versionbehavior, but -V does not exist.
Clang's GNU driver emulates GCC 4.2.1, but you can change the versionwith -fgnuc-version=.
Within the LLVM project, MC is a library responsible for handlingassembly, disassembly, and object file formats. Introto the LLVM MC Project, which was written back in 2010, remains agood source to understand the high-level structures.
In the latest release cycle, substantial effort has been dedicated torefining MC's internal representation for improved performance andreadability. These changes have decreased compiletime significantly. This blog post will delve into the details,providing insights into the specific changes.
MergedMCAsmLayout into MCAssembler
MCAssembler manages assembler states (includingsections, symbols) and implements post-parsing passes (computing alayout and writing an object file). MCAsmLayout, tightlycoupled with MCAssembler, was in charge of symbol andfragment offsets during MCAssembler::Finish.MCAsmLayout was a wrapper of MCAssembler and asection order vector (actually Mach-O specific). ManyMCAssembler and MCExpr member functions have aconst MCAsmLayout & parameter, contributing to slightoverhead. Here are some functions that are called frequently:
MCAssembler::computeFragmentSize is called a lot in thelayout process.
MCAsmBackend::handleFixup andMCAsmBackend::applyFixup evaluate each fixup and producerelocations.
MCAssembler::fixupNeedsRelaxation determines whether aMCRelaxableFragment needs relaxation due to aMCFixup.
MCAssembler::relaxFragment andMCAssembler::relaxInstruction relax a fragment.
I startedto merge MCAsmLayout into MCAssembler andsimplify MC code, and eventually removed llvm/include/llvm/MC/MCAsmLayout.h.
Fragments
Fragments, representing sequences of non-relaxable instructions,relaxable instruction, alignment directives, and other elements.MCDataFragment and MCRelaxableFragment, whosesizes are crucial for memory consumption, have undergone severaloptimizations:
The fragment management system has also been streamlined bytransitioning from a doubly-linked list (llvm::iplist) to asingly-linkedlist, eliminating unnecessary overhead. A few prerequisite commitsremoved backward iterator requirements.
Furthermore, I introducedthe "current fragment" concept (MCSteamer::CurFrag)allowing for faster appending of new fragments.
I have also simplified and optimized fragment offset computation:
[MC] Relax fragments eagerly
Previously, calculating fragment offsets happened lazily in thegetFragmentOffset function. All sections were iterativelyrelaxed until they all converged. This process was inefficient as theslowest section determined the number of iterations for all others,resulting in extra calculations.
Previously, fragment offset computation was lazily performed bygetFragmentOffset. The section that converged the slowestdetermined other sections' iteration steps, leading to some unneededcomputation.
The new layout algorithm assigns fragment offsets and iterativelyrefines them for each section until it's optimized. Then, it moves on tothe next section. If relaxation doesn't change anything, fragment offsetassignment will be skipped. This way, sections that converge quicklydon't have to wait for the slowest ones, resulting in asignificant decrease in compile time for full LTO.
In MCObjectStreamer, newly defined labels were put intoa "pending label" list and initially assigned to aMCDummyFragment associated with the current section. Thesymbols will be reassigned to a new fragment when the next instructionor directive is parsed. This pending label system, while necessary foraligned bundling, introduced complexity and potential for subtlebugs.
To streamline this, I revampedthe implementation by directly adjusting offsets of existingfragments, eliminating over 100 lines of code and reducing the potentialfor errors.
Details: In 2014, [MC]Attach labels to existing fragments instead of using a separatefragment introduced flushPendingLabels aligned bundlingassembler extension for Native Client. [MC] Match labels to existingfragments even when switching sections., built on top offlushPendingLabels, added further complication.
In MCObjectStreamer, a newly defined label wastemporarily assigned to a MCDummyFragment. The symbol wouldbe reassigned to a new fragment when the next instruction or directivewas parsed. The MCDummyFragment was not in the section'sfragment list. However, during expression evaluation, it should beconsidered as the temporary end of the section.
For the following code, aligned bundling requires that.Ltmp is defined at addl.
Worse, a lot of directive handling code had to addflushPendingLabels and a missingflushPendingLabels could lead to subtle bugs related toincorrect symbol values.
( MCAsmStreamer doesn't callflushPendingLabels in its handlers. This is the reason thatwe cannot change MCAsmStreamer::getAssemblerPtr to use aMCAssembler and changeAsmParser::parseExpression. )
Sections
Section handling was also refined. MCStreamer maintains a a sectionstack for features like.push_section/.pop_section/.previousdirectives. Many functions relied on the section stack for loading thecurrent section, which introduced overhead due to the additionalindirection and nullable return values.
By leveraging the "current fragment" concept, the need for thesection stack was eliminated in most cases, simplifying the codebase andimproving efficiency.
I have eliminated nullable getCurrentSectionOnly usesand changedgetCurrentSectionOnly to leverage the "current fragment"concept. This change also revealedan interesting quirk in NVPTX assembly related to DWARFsections.
Section symbols
Many section creation functions (MCContext::get*Section)had a const char *BeginSymNameparameter to support the section symbol concept. This led to issueswhen we want to treat the section name as a symbol. In 2017, theparameter was removedfor ELF, streamlining section symbol handling.
I changed the way MC handles section symbols for COFF andremoved the unused parameters for WebAssembly. The work planned forXCOFF is outlined in https://github.com/llvm/llvm-project/issues/96810.
Expression evaluation
Expression evaluation in MCAssembler::layout previouslyemployed a complex lazy evaluation algorithm, which aimed to minize thenumber of fragment relaxation. It proved difficult to understand andresulted in complex recursiondetection.
To address this, I removed lazy evaluation in favor of eagerfragment relaxation. This simplification improved the reliability ofthe layout process, eliminating the need for intricate workarounds likethe MCFragment::IsBeingLaidOut flag introduced earlier.
Note: the benefit of lazy evaluation largely diminished when https://reviews.llvm.org/D76114 invalidated all sectionsto fix the correctness issue for the following assembly:
1 2 3 4 5 6 7 8
.section .text1,"ax" .skip after-before,0x0 .L0:
.section .text2 before: jmp .L0 after:
In addition, I removed anoverload of isSymbolRefDifferenceFullyResolvedImpl, enablingconstant folding for variable differences in Mach-O.
Target-specificfeatures misplaced in the generic implementation
I have made efforts to relocate target-specific functionalities totheir respective target implementations:
Note: This feature is unstable. While relocatable files created withClang version A will work with lld version A, they might not becompatible with newer versions of lld (where A is older than B).
As thefuture of the generic ABI remains uncertain, CREL might not get"standardized". In that case, I will just get the section code agreedwith the GNU community to ensure wider compatibility.
Assembly parser
\+, the per-macro invocation count, is nowavailable for .irp/.irpc/.rept.
[MCParser].altmacro: Support argument expansion not preceded by \
I've been contributing to MC for several years. Back then, while manycontributed, most focused on adding specific features. Rafael Ávila deEspíndola was the last to systematically review and improve the MClayer. Unfortunately, simplification efforts stalled after Rafael'sdeparture in 2018.
Picking up where Rafael left off, I'm diving into the MC layer tostreamline its design. A big thanks to @aengelke for his invaluable performancecentric contributions in this release cycle. LLVM 19 introducessignificant enhancements to the integrated assembler, resulting innotable performance gains, reduced memory usage, and a more streamlinedcodebase. These optimizations pave the way for future improvements.
I compiled the preprocessed SQLite Amalgamation (from llvm-test-suite)using a Release build of clang:
build
2024-05-14
2024-07-02
-O0
0.5304
0.4930
-O0 -g
0.8818
0.7967
-O2
6.249
6.098
-O2 -g
7.931
7.682
clang -c -w sqlite3.i
The AsmPrinter pass, which is coupled with the assembler, consumes asignificant portion of the -O0 compile time. I havemodified the -ftime-report mechanism to decrease theper-instruction overhead. The decrease in compile time matches thedecrease in the spent in AsmPrinter. Coupled with a recent observationthat BOLT, which heavily utilizes MC, is ~8% faster, it's clear that MCmodifications have yielded substantial improvements.
Noticeableoptimizations in previous releases
[MC] Always encodeinstruction into SmallVector optimizedMCCodeEmitter::encodeInstruction for x86 by avoidingraw_ostream::write overhead. I have migrated other targetsand removedthe extra overload.
1
raw_ostream::write =(inlinable)=> flush_tied_then_write (unneeded TiedStream check) =(virtual function call)=> raw_svector_ostream::write_impl ==> SmallVector append(ItTy in_start, ItTy in_end) (range; less efficient then push_back).
llvm-mc:Diagnose misuse (mix) of defined symbols and labels. addedredefinition error. This was refined many times. I hope to fix this inthe future.
Addressing Mach-O weakness
The Mach-O assembler lacks the robustness of its ELF counterpart.Notably, certain aspects of the Mach-O implementation, such as theconditions for constant folding inMachObjectWriter::isSymbolRefDifferenceFullyResolvedImpl(different for x86-64 and AArch64), warrant revisiting.
Additionally, the Mach-O has a hack to maintaincompatibility with Apple cctools assembler, when the relocationaddend is non-zero.
1 2 3 4 5 6 7 8 9 10
.data a = b + 4 .long a # ARM64_RELOC_UNSIGNED(a) instead of b; This might work around the linker bug(?) when the referenced symbol is b and the addend is 4.
c = d .long c # ARM64_RELOC_UNSIGNED(d)
y: x = y + 4 .long x # ARM64_RELOC_UNSIGNED(x) instead of y
This leads to another workaround inMCFragment.cpp:getSymbolOffsetImpl ([MC] Recursively calculatesymbol offset), which is to support the following assembly:
1 2 3 4
l_a: l_b = l_a + 1 l_c = l_b .long l_c
Misc
emitLabel at switchSection was for DWARFsections, which might be no longer useful
GNU ld's output section layout is determined by a linker script,which can be either internal (default) or external (specified with-T or -dT). Within the linker script,SECTIONS commands define how input sections are mapped intooutput sections.
Input sections not explicitly placed by SECTIONScommands are termed "orphansections".
Orphan sections are sections present in the input files which are notexplicitly placed into the output file by the linker script. The linkerwill still copy these sections into the output file by either finding,or creating a suitable output section in which to place the orphanedinput section.
GNU ld's default behavior is to create output sections to hold theseorphan sections and insert these output sections into appropriateplaces.
Orphan section placement is crucial because GNU ld's built-in linkerscripts, while understanding common sections like.text/.rodata/.data, are unawareof custom sections. These custom sections should still be included inthe final output file.
Grouping: Orphan input sections are grouped into orphan outputsections that share the same name.
Placement: These grouped orphan output sections are then insertedinto the output sections defined in the linker script. They are placednear similar sections to minimize the number of PT_LOADsegments needed.
GNU ld's algorithm
GNU ld's orphan section placement algorithm is primarily specifiedwithin ld/ldlang.c:lang_place_orphans andld/ldelf.c:ldelf_place_orphan.lang_place_orphans is a linker pass that is betweenINSERT processing and SHF_MERGE sectionmerging.
The algorithm utilizes a structure (orphan_save) toassociate desired BFD flags (e.g., SEC_ALLOC, SEC_LOAD)with special section names (e.g., .text, .rodata) and areference to the associated output section. The associated outputsection is initialized to the special section names (e.g.,.text, .rodata), if present.
For each orphan section:
If an output section of the same name is present and--unique is not specified, the orphan section is placed init.
Otherwise, GNU ld identifies the matching orphan_saveelement based on the section's flags.
If an associated output section exists related to theorphan_save element, the orphan section is placed afterit.
Otherwise, heuristics are applied to place the orphan section aftera similar existing section. For example:
.data-like sections can follow .rodata-like sections.
The associated output section is replaced with the new outputsection. The next orphan output section of similar flags will be placedafter the current output section.
As a special case, if an orphan section is placed after the lastoutput section(else if (as != snew && as->prev != snew)), itwill be adjusted to be placed after all trailing commands(sym = expr, . = expr, etc).
For example, custom code section mytext (withSHF_ALLOC | SHF_EXECINSTR) would typically be placed after.text, and custom data section mydata (withSHF_ALLOC | SHF_WRITE) after .data.
.interp and .rodata have the same BFDflags, but they are anchors for different sections.SHT_NOTE sections go after .interp, whileother read-only sections go after .rodata.
Consider a scenario where a linker script defines .dataand .rw1 sections with identical BFD flags. If we haveorphan sections that share the same flags, GNU ld would insert theseorphans after .data, even if it might seem more logical toplace them after .rw1.
1 2 3
.data : { *(.data .data.*) } // .rw2 .rw3 orphans are inserted here .rw1 : { *(.rw1) }
Renaming the output section .data will achieve thedesired placement:
1 2 3
.mydata : { *(.data .data.*) } .rw1 : { *(.rw1) } // .rw2 .rw3 orphans are inserted here
lld's algorithm
The LLVM linker lld implements a large subset of the GNU ld linkerscript. However, due to the complexity of GNU ld and lack of an officialspecification, there can be subtle differences in behavior.
While lld strives to provide a similar linker script behavior, itoccasionally makes informed decisions to deviate where deemedbeneficial. We balance compatibility with practicality andinterpretability.
Users should be aware of these potential discrepancies whentransitioning from GNU ld to lld, especially when dealing with intricatelinker script features.
lld does not have built-in linker scripts. When noSECTIONS is specified, all input sections are orphansections.
Rank-based sorting
lld assigns a rank to each output section, calculated using variousflags like RF_NOT_ALLOC, RF_EXEC, RF_RODATA, etc. Orphanoutput sections are then sorted by these ranks.
For each orphan section, lld identifies the output section with themost similar rank. The similarity is determined by counting the numberof leading zeros in the XOR of the two ranks.
1 2 3 4 5 6 7 8 9 10
// We want to find how similar two ranks are. // The more branches in getSectionRank that match, the more similar they are. // Since each branch corresponds to a bit flag, we can just use // countLeadingZeros. staticintgetRankProximity(OutputSection *a, SectionCommand *b){ auto *osd = dyn_cast<OutputDesc>(b); return (osd && osd->osec.hasInputSections) ? llvm::countl_zero(a->sortRank ^ osd->osec.sortRank) : -1; }
When multiple output sections share the maximum similarity with anorphan section, resolving the ambiguity is crucial. I refined thebehavior for lld 19: if the orphan section's rank is not lower thanthe similar sections, the last similar section is chosen forplacement.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
// Find the most similar output section as the anchor. Rank Proximity is a // value in the range [-1, 32] where [0, 32] indicates potential anchors (0: // least similar; 32: identical). -1 means not an anchor. // // In the event of proximity ties, we select the first or last section // depending on whether the orphan's rank is smaller. int maxP = 0; auto i = e; for (auto j = b; j != e; ++j) { int p = getRankProximity(sec, *j); if (p > maxP || (p == maxP && cast<OutputDesc>(*j)->osec.sortRank <= sec->sortRank)) { maxP = p; i = j; } } if (i == e) return e;
For example, when inserting .bss orphan sections(SHF_ALLOC|SHF_WRITE, SHT_NOBITS), lld shouldfind the last output section that carries the flags/typeSHF_ALLOC|SHF_WRITESHT_PROGBITS.
1 2 3 4 5 6
WA PROGBITS (not here) A WA PROGBITS AX WA PROGBITS (here) <== WA NOBITS
Placement decision
The orphan section is placed either before or after the most similarsection, based on a complex rule involving:
The relative ranks of the orphan and similar section.
The presence of PHDRSor MEMORYcommands in the linker script.
Scanning backward or forward through the script for a suitableinsertion point.
In essence:
If the orphan section's rank is lower than the similar section'srank, and no PHDRS or MEMORY commands exist,it's placed before the similar section.
Otherwise, it's placed after the similar section,potentially skipping symbol assignments or output sections without inputsections in the process.
auto isOutputSecWithInputSections = [](SectionCommand *cmd) { auto *osd = dyn_cast<OutputDesc>(cmd); return osd && osd->osec.hasInputSections; };
// If i's rank is larger, the orphan section can be placed before i. // // However, don't do this if custom program headers are defined. Otherwise, // adding the orphan to a previous segment can change its flags, for example, // making a read-only segment writable. If memory regions are defined, an // orphan section should continue the same region as the found section to // better resemble the behavior of GNU ld. bool mustAfter = script->hasPhdrsCommands() || !script->memoryRegions.empty(); if (cast<OutputDesc>(*i)->osec.sortRank <= sec->sortRank || mustAfter) { for (auto j = ++i; j != e; ++j) { if (!isOutputSecWithInputSections(*j)) continue; if (getRankProximity(sec, *j) != proximity) break; i = j + 1; } } else { for (; i != b; --i) if (isOutputSecWithInputSections(i[-1])) break; }
// As a special case, if the orphan section is the last section, put // it at the very end, past any other commands. // This matches bfd's behavior and is convenient when the linker script fully // specifies the start of the file, but doesn't care about the end (the non // alloc sections for example). if (std::find_if(i, e, isOutputSecWithInputSections) == e) return e;
while (i != e && shouldSkip(*i)) ++i; return i;
Special case: last section
If the orphan section happens to be the last one, it's placed at thevery end of the output, mimicking GNU ld's behavior for cases where thelinker script fully specifies the beginning but not the end of thefile.
Special case: skipping symbol assignments
It is common to surround an output section description withencapsulation symbols. lld has a special case to not place orphansbetween foo and a following symbol assignment.
Backward scan example:
1 2 3 4 5
previous_start = .; previous : { *(previous) } // Found output section with a backward scan previous_end = .; // The orphan should be after here
similar : { *(similar) } // The most similar section found by the first step
Forward scan example:
1 2 3 4
similar0 : { *(similar0) } similar1_start = .; similar1 : { *(similar1) } // The most similar section found by the first step similar1_end = .; // The orphan should be after here
However, an assignment to the location counter serves as a barrier tostop the forward scan.
1 2 3 4 5 6 7
previous_start = .; previous : { *(previous) } // Found output section with a backward scan previous_end = .; // The orphan should be after here symbol = .; // We conservatively assume any symbol as a probable "end" symbol. . = ALIGN(CONSTANT(MAXPAGESIZE)); // Barrier
similar : { *(similar) } // The most similar section found by the first step
Special case: initial location counter
In addition, if there is a location counter assignment before thefirst output section, orphan sections cannot be inserted before theinitial location counter assignment. This is to recognize the commonpattern that the initial location counter assignments specifies the loadaddress.
1 2 3
sym0 = .; . = initial_location; // Initial assignment to DOT. Orphan sections cannot be inserted before here. .text : {} // First output section
Presence of PHDRS orMEMORY
The presence of PHDRS or MEMORY commandsdisallows lld to place the orphan section before the anchor. Thiscondition is introduced in two patches:
When a linker script defines PHDRS, it typicallyspecifies the initial section within each PT_LOAD segment.These sections often have address requirements, indicated by a preceding. = expr statement. If an orphan section is associated withsuch a section as its anchor, lld avoids inserting the orphan before theanchor to maintain the intended segment structure and addressalignment.
For instance, consider this linker script excerpt:
Here, .rodata is the first section in aPT_LOAD segment, and it's aligned toMAXPAGESIZE. If an orphan section is inserted before.rodata, it would inherit the previous segment's flags andbreak the intended address requirement.
Program headers propagation
After orphan section placement, if the PHDRS command isspecified, lld will propagate program headers to output sections that donot specify :phdr.
Case study
By employing this rank-based approach, lld provides an elegantimplementation that does not hard code specific section names (e.g.,.text/.rodata/.data). In GNU ld,if you rename special section names.text/.rodata/.data in the linkerscript, the output could become subtle different.
Orphan sections matching an output section name
The following output section description does not match.foo input sections, but .foo orphan sectionswill still be placed inside .foo.
1
.foo : { *(.bar) }
Read-only sections
Among read-only sections (e.g., .dynsym,.dynstr, .gnu.hash, .rela.dyn,.rodata, .eh_frame_hdr,.eh_frame), lld prioritizes the placement ofSHT_PROGBITS sections (.rodata,.eh_frame_hdr, and .eh_frame) closer to codesections. This is achieved by assigning them a higher rank.The rationale behind this design is to mitigate the risk of relocationoverflow in the absence of an explicit linker script.
These non-SHT_PROGBITS sections do not containrelocations to code sections and can be placed away from codesections.
1 2 3 4 5 6 7 8 9 10 11
.dynsym .gnu.version .gnu.version_r .gnu.hash .dynstr .rela.dyn .rela.plt .rodata // closer to .text .eh_frame_hdr // closer to .text .eh_frame // closer to .text .text
If a linker script explicitly includes a SECTIONScommand specifying .rodata without mentioning otherread-only sections, orphan sections like .dynsym might beplaced before .rodata.
1
.rodata : { *(.rodata .rodata.*) }
This behavior can be further influenced by the presence ofPHDRS commands. If an outputsection phdr is specified with .rodata, orphan sectionslike .dynsym would not be placed before.rodata, ensuring that the orphans would not affect theflags of the preceding program header.
1 2 3 4 5 6 7 8 9
PHDRS { text PT_LOAD; }
SECTIONS { ... .rodata : { *(.rodata .rodata.*) } : text // .dynsym cannot be placed before .rodata with specified program headers }
Symbol assignment between two output sections
A symbol assignment placed between output sections can be interpretedin two ways: as marking the end of the preceding section or the start ofthe following section. lld doesn't attempt to guess the intendedmeaning, leading to potential ambiguity in scenarios like this:
In both cases, lld might place SHF_ALLOC|SHF_WRITESHT_PROGBITS orphan sections before .bss,potentially disrupting the intended behavior if the goal was to mark thestart of the .bss section with bss_start = ..
To avoid this ambiguity and ensure consistent behavior, therecommended practice is to place symbol assignments within the outputsection descriptions (FreeBSDexample):
To maximize portability of linker scripts across different linkers,it's essential to establish clear boundaries for PT_LOAD segments. Thiscan be achieved by:
Explicit alignment: Utilizing MAXPAGESIZE alignment todistinctly separate sections within the linker script.
Anchoring sections: Ensuring that the first section in eachPT_LOAD segment includes at least one input section,preventing ambiguous placement decisions by the linker. When thePHDRS command is present, ensure that the first sectionshave :phdr.
By adhering to these guidelines, you can reduce reliance onlinker-specific orphan section placement algorithms, promotingconsistency across GNU ld and lld.
When linking a regular position-dependent executable, you may alsosupply a minimal linker script like the following for a-no-pie link:
For projects that require absolute control over section placement,GNU ld version 2.26 and later provides--orphan-handling=[place|warn|error|discard]. This allowsyou to choose how orphan sections are handled:
place (default): The linker places orphan sections according to itsinternal algorithm.
warn: The linker places orphan sections but also issues warnings foreach instance.
error: The linker treats orphan sections as errors, preventing thelinking process from completing.
discard: The linker discards orphan sections entirely.
The ELF object file format is adopted by many UNIX-like operatingsystems. While I've previously delvedinto the control structures of ELF and its predecessors, tracing thehistorical evolution of ELF and its relationship with the System V ABIcan be interesting in itself.
The format consists of the generic specification, processor-specificspecifications, and OS-specific specifications. Three key documentsoften surface when searching for the generic specification:
The TIS specification breaks ELF into the generic specification, aprocessor-specific specification (x86), and an OS-specific specification(System V Release 4). However, it has not been updated since 1995. TheSolaris guide, though well-written, includes Solaris-specific extensionsnot applicable to Linux and *BSD. This leaves us primarily with theSystem V ABI hosted on www.sco.com, which dedicates Chapters 4 and 5 tothe ELF format.
Let's trace the ELF history to understand its relationship with theSystem V ABI.
History
UnixSystem Laboratories (USL) created ELF for their System V Release 4in late 1980s. USL also maintained the System V Application BinaryInterface, of which ELF was a core component. The dynamic shared librarysystem was contributed by Sun Microsystems from their SunOS 4.x (in 1988, SunOS4.0 got an extended a.out format with dynamic shared librarysupport).
USL intended ELF to be an open standard and published documents aboutthe format, e.g.
In Proceedings of the Summer 1990 USENIX Conference, ELF: AnObject File to Mitigate Mischievous Misoneism by James Q.Arnold
UNIX System V Release 4 Programmer's Guide: ANSI C andProgramming Support Tools (ISBN 0-13-933706-7) published in1990
System V Application Binary Interface (Standards) (ISBN0-13-104670-5) published in 1993
In 1993, the Tool Interface Standard (TIS) Committee, a consortium ofindustry leaders, adopted ELF and developed the "Tool Interface Standard(TIS) Portable Formats Specification". Version 1.2 was released in May1995.
ELF has been very influential. In the 1990s, many Unix and Unix-likeoperating systems, including Solaris, IRIX, HP-UX, Linux, and FreeBSD,switched to ELF. The 86open Project's FAQ specified:
Q18: How can you get a single binary to work identically across allthese diverse systems?
Most Unix-on-Intel binary packages are already largely similar.Almost all such operating systems use the "ELF" binary 'packaging'; thevarious operating systems have small but significant differences,though, that make each system's ELF binary unusable on others'.
The evolvingstewardship of the System V ABI
The Tool Interface Standard (TIS) Committee essentially dissolvedafter 1995. The stewardship of the System V ABI, and consequently thegeneric ELF specification, has followed a complex path mirroring thetransfer of Unix software assets.
Between 1993 and 2011, Unix assets underwent a few transfers.
In 1993, Novell acquiredUnix assets including all copyrights, trademarks, and licensingcontracts.
In September 1995, Novell sold the "develop and sell licenses toUnix binaries" plus "handle source licencees" business to The Santa CruzOperation (sometimes referred to as "old SCO"). Novell still owned thecopyrights (SCOvs Novell verdict).
In 2001, The Santa Cruz Operation sold its Unix software asserts toCaldera Systems (later renamed The SCO Group, Inc; sometimes referred toas "new SCO" or "SCOX").
In 2011, The SCO Group's Unix software assets were sold off to UnXis(later renamed Xinuos).
The task of maintaining and updating the generic ABI fell tothese successive owners of Unix software assets. The Santa CruzOperation, and later The SCO Group and Xinuos, managed updates andextensions to the ABI, including the ELF specification.
In this binutilscommit in November 2000, it was said that e_machinevalues should eventually ask registry@sco.com for blessing(now registry@xinuos.com).
Dave Prosser had maintainedthe System V ABI at USL, then The Santa Cruz Operation, and then The SCOGroup. The last maintainer at The SCO Group and UnXis/Xinuous was JohnWolfe, who oversaw updates until his departurefrom Xinuos in 2015. The generic ABI (including the ELFspecification) then became unmaintained.
The final functional update on https://www.sco.com/developers/gabi/latest/contents.htmlwas made in June 2013 forSHF_COMPRESSED. Since then, the specification hasremained frozen.
"All rights reserved"?
The copyright notices on the SCO website's documentation for theSystem V ABI seem potentially misleading.
The footnotes of https://www.sco.com/developers/gabi/1998-04-29/contents.htmlpages today (and in 2003 per web.archive.org) specify:
The repeated phrase "All rights reserved" could be interpreted asimplying exclusive ownership over the ELF format itself. This isinaccurate, as ELF is an open standard developed through thecollaboration of many organizations and individuals. The Santa CruzOperation's role in the evolution of the System V ABI seems to have beenmore of an editor than an innovator. After The Santa Cruz Operation soldits Unix assets in 2001, the specification has largely stayed unchangedwith occasional constant updates.
The earliest available snapshot on the Wayback Machine dates back to2003, a time when The SCO Group had assumed ownership and initiated alawsuit against IBM, alleging that the success of Linux was due to themisappropriation of SCO's technology. Regrettably, earlier snapshots areunavailable to provide a more complete historical context.
Tool Interface Standard (TIS) Portable Formats Specification,version 1.2 effectively putthe specification in the public domain:
The TIS Committee grants you a non-exclusive, worldwide, royalty-freelicense to use the information disclosed in this Specification to makeyour software TIS-compliant; no other license, express or implied, isgranted or intended hereby.
A neutral GoogleGroup not affliated with The SCO Group/Xinuous exists for discussingthe generic ABI. Hongjiu Lu might be the owner. The group served as aplatform for OS and toolchain vendors to collaborate. In recent years,participation has dwindled to primarily representatives from OracleSolaris (just Ali Bahrami) and the GNU toolchain.
The reduced activity might not seem critical, as significantnon-OS-specific changes to the ELF format are infrequent.
However, discussions regarding these specific features seemunavailable. Please let me know if you have any information onthem.
There were merely constant updates from April 2001 to June 2013.SHF_COMPRESSED was added in June 2013.
The generic-abi Google Group reached consensus onproposals that haven't been reflected on the www.sco.comwebsite:
2018: RELRrelative relocation format
2022: ELFCOMPRESS_ZSTD
A future in flux
In April 2020, Cary Coutant reached a preliminaryagreement with Xinuos, but the future remainsuncertain. While some constants (e.g., e_machineand EI_OSABI values, ELFCOMPRESS_ZSTD) havebeen defined, no functional updates to the ABI have materialized.
The absence of a centralized, up-to-date repository for thespecification complicates matters. While some clarificationsand consensus have been reached within the generic-abi group, accessingthe latest, definitive text remains a challenge.
A potential solution could be to decouple the ELFspecification from the broader System V ABI, as was done in thepast with the TIS specification. This would create a dedicated andaccessible reference for ELF, independent of the broader System Vspecificities that are of less general interest.
Despite this uncertainty, innovation within the ELF ecosystem shouldcontinue. Efforts like my own to replace ELF control structures toreduce object file sizes (e.g., compactrelocations.) can still move forward. In practice, achievingconsensus among major toolchain vendors (GNU and LLVM) may besufficient, even without formal approval from the generic ABI. Whilealigning with Solaris would be ideal and I will try doing so, this mightnot always be feasible due to varying priorities.
FreeBSD, which Xinuos's OpenServer is based on, utilizes the LLVMtoolchain. Xinuos might indirectly benefit from my heavy involvementinto the LLVM toolchain.
System V ABI ProcessorSupplement (psABI)
Processor-specific details for the System V ABI are found in thepsABI documents. Actively maintained psABIs exist for variousarchitectures including AArch32, AArch64, LoongArch,PPC64, RISC-V,s390x,i386, and x86-64.(These links refer to my notes.)
Many architectures have older or unavailable psABIs. Forinstance:
ppc32: Power Architecture® 32-bit Application Binary InterfaceSupplement 1.0 - Linux & Embedded was published in 2011. My notes
MIPS: The most recent o32 ABI dates back to February 1996, and then64 ABI is unavailable. The n32 ABI accompanies the discontinuedcompiler MIPSpro: MIPSpro N32 ABI Handbook. My notes
Noteworthy details
Architectures like Motorola 6800, which have 16-bit address spaces,use the ELFCLASS32 format.
Many architectures have never been ported to any System V derivativeOS, but their psABI documents still use the "System V" name.
Some behaviors are not formally documented and can only be found inthe binutils project's source code.
Operating System SpecificABIs
In the System V ABI, Operating System Specific ABIs (OSABI) areextensions that provide operating system-specific details to supplementthe generic ABI.
For example, Oracle Solaris Linkers and Libraries Guidedefines the OSABI for Solaris.
The term OSABI is vague and might not be one single document. ForLinux, we need the following two documents:
The Linux Standard Base (LSB) is a related document that aims tostandardize the Linux system interface.
Until the recent introduction of gABI supplement for programloading and dynamic linking on GNU, SHT_GNU_HASH,while widely adopted, was absent from any official documentation.
Interestingly, many Linux ABI extensions are generic enough to beadopted by other operating systems like FreeBSD, suggesting that adedicated FreeBSD OSABI document may not be necessary.
The Linux kernel is written in C, but it also leverages extensionsprovided by GCC. In 2022, it moved from GCC/Clang-std=gnu89 to -std=gnu11. This articleexplores my notes on how these GNU extensions are utilized within thekernel.
Statement expressions
Statementexpressions are commonly used in macros.
Locallabels
Some macros use this extension to restart a for loop in a macro'sreplacement list.
Bytecode interpreters often leverage this extension. BPF, forinstance, utilizes this extension. We can also see this extension indrm_exec_retry_on_contention for the Direct RenderingManager.
When it comes to Position-Dependent Code (PDC), a switch statementwith a default label marked as __builtin_unreachable can bejust as efficient as using labels as values. However, forPosition-Independent Code (PIC), absolute addresses offer a slightperformance edge, although at the cost of requiring more dynamicrelocations.
typeof is used in numerous code.__auto_type has a few occurrences.
C23 standardizes typeof and auto.
Conditionalswith omitted operands
x ?: y is known as the "Elvis operator". This is used innumerous code.
Emptystructures
The C standard (at least C11 and C23) specifies that:
If the member declaration list does not contain any named members,either directly or via an anonymous structure or anonymous union, thebehavior is undefined.
The empty structure extension enables a dummy structure when aconfiguration option disables the functionality.
This extension (case low ... high:), frequently used inthe kernel, allows us to specify a range of consecutive values in asingle case label within a switch statement.
Objectsize checking
glibc 2.3.4 introduced _FORTIFY_SOURCE in 2004 to catchsecurity errors due to misuse of some C library functions (primarilybuffer overflow). The implementation leverages inline functions and__builtin_object_size.
Linux kernel introduced CONFIG_FORTIFY_SOURCE in 2017-07.Like the userspace, the implementation relies on optimizations likeinlining and constant folding.
For example, include/linux/string.h combines thisfeature with BUILD_BUG_ON(__attribute__((error(...)))) to provide compile-timeerrors:
The Clang implementation utilizes__attribute__((pass_object_size(type))) and__attribute__((overloadable(...))).
Clang introduced__builtin_dynamic_object_size in 2019 to possiblyevaluate the size dynamically. The feature was ported to GCC in 2021 andis used by the kernel.
Pragmas
#pragma GCC diagnosticis occasionally used to disable a local diagnostic.
include/linux/hidden.h uses #pragma GCC visibility("hidden")to force direct access (instead of GOT indirection) for external symbolsfor -fpie/-fpic.
bpf uses #pragma GCC poisonto forbid some undesired identifiers.
Structure-layoutpragmas are commonly used.
Inline assembly
The Linux kernel uses inline assembly for a few reasons
Hardware interaction: Certain operations require direct interactionwith the underlying hardware capabilities, which might not beexpressible in standard C.
Performance optimization: In critical code paths, inline assemblycan be used to squeeze out the last bit of performance by utilizingprocessor-specific instructions or bypassing certain C languageconstructs that might introduce overhead.
Special facility: Some features, such as static keys, alternativesruntime patching, cannot be implemented with standard C.
Built-in functions
Special machine codeinstructions
GCC provides built-in functions to generate machine code instructionsdesigned for certain tasks like popcount/clz/ctz. The compiler has moreinformation about the function's purpose and leverages internaloptimizations tailored for these tasks.
__builtin_choose_expr
This is analogous to ? : but the condition is a constantexpression and the return type is not altered by promotion rules. InGCC, __builtin_choose_expr is not available in C++.
#define __abs_choose_expr(x, type, other) __builtin_choose_expr( \ __builtin_types_compatible_p(typeof(x), signed type) || \ __builtin_types_compatible_p(typeof(x), unsigned type), \ ({ signed type __x = (x); __x < 0 ? -__x : __x; }), other)
__builtin_constant_p
__builtin_constant_p identifies whether an expressioncan be evaluated to a constant. This capability unlocks several codepatterns.
Conditional static assertions:
1
BUILD_BUG_ON(__builtin_constant_p(nr) && nr != 1);
__builtin_constant_p decides whetherMAYBE_BUILD_BUG_ON expands to a compile-time static assertmechanism (__attribute__((error(...)))) or a runtimemechanism.
1 2 3 4 5 6 7
#define MAYBE_BUILD_BUG_ON(cond) \ do { \ if (__builtin_constant_p((cond))) \ BUILD_BUG_ON(cond); \ else \ BUG_ON(cond); \ } while (0)
Alternative code paths:
Sometimes, the constant expression case might enable a more efficientcode path.
By knowing expressions are constant, the compiler can performconstant folding optimizations. This means pre-calculating theexpression's value during compilation and replacing it with the actualconstant in the code. This leads to potentially smaller and faster codeas the calculation doesn't need to be done at runtime. However, I noticethat __builtin_constant_p is often abused. This is one ofthe source why the kernel cannot be built with -O0.
__builtin_expect
Used by likely and unlikely macros to givethe compiler hints for optimization.
include/linux/jump_label.h combines__builtin_types_compatible_p and dead code elimination toemulate C++17 if constexpr.____wrong_branch_error is not defined. If the unexpectedcode path is taken, there will be a linker error.
tools/include/linux/compiler.h defines macros for manycommonly used attributes. Many of them are commonly used by otherprojects and probably not worth describing.
tools/testing/selftests has a few__attribute__((constructor)) for initialization.
__attribute__((error(...))) is used with inlining anddead code elimination to provide static assertion.
1 2 3 4 5 6 7 8 9 10 11 12
// include/linux/compiler_types.h #ifdef __OPTIMIZE__ # define __compiletime_assert(condition, msg, prefix, suffix) \ do { \ __noreturn extern void prefix ## suffix(void) \ __compiletime_error(msg); \ if (!(condition)) \ prefix ## suffix(); \ } while (0) #else # define __compiletime_assert(condition, msg, prefix, suffix) do { } while (0) #endif
__attribute__((naked)) (__naked) is used byarm and bpf code.
__attribute__((section(...))) is used to place functionsin the specified section, e.g. .init.text.
__attribute__((weak)) is used to allow a genericfunction to be replaced by an arch-specific implementation. The weakreference feature is used by some__start_<sectionname>/__stop_<sectionname>encapsulation symbols.
#define __noendbr __attribute__((nocf_check)) is usedto disable Intel CET for some special functions.
Variable attributes
__attribute__((cleanup(...))) runs a function when thevariable goes out of scope. include/linux/cleanup.h definesDEFINE_FREE and __free based on thisfeature.
__attribute__((aligned(...))) and__attribute__((packed)) are often to control structurelayouts.
Language dialects
-fno-delete-null-pointer-checks
Assume that programs can safely dereference null pointers, and codeor data element may reside at address zero.
-fno-strict-aliasing
The C11 standard (section 6.5) defines strict aliasing rules.
An object shall have its stored value accessed only by an lvalueexpression that has one of the following types:
— a type compatible with the effective type of the object, — aqualified version of a type compatible with the effective type of theobject, — a type that is the signed or unsigned type corresponding tothe effective type of the object, — a type that is the signed orunsigned type corresponding to a qualified version of the effective typeof the object, — an aggregate or union type that includes one of theaforementioned types among its members (including, recursively, a memberof a subaggregate or contained union), or — a character type.
Compilers leverage these aliasing rules for optimizations, which canbe disabled with -fno-strict-aliasing.
Linus Torvalds has expressed reservations about strict aliasing. Hereis a discussionfrom 2018.
-fno-strict-overflow
In GCC, this option is identical to-fwrapv -fwrapv-pointer: make signed integer overflowdefined using wraparound semantics. While avoiding undefined behaviors,unexpected arithmetic overflow bugs might be lurking.
[RFC]Mitigating unexpected arithmetic overflow is a 2024-05 thread aboutdetecting and mitigating unsigned integer overflow.
登录
