Skip to content

infra: use LLD for all configurations #13203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/advanced-topics/fuzz_introspector.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,16 +101,16 @@ There are some differences in build environment for Fuzz Introspector builds
in comparison to e.g. ASAN or code coverage builds. The reason is that
Fuzz Introspector relies on certain compile-time tools to do its analysis.
This compile time tooling differs between languages, namely:
- For C/C++, Fuzz Introspector relies on [LLVM LTO](https://llvm.org/docs/LinkTimeOptimization.html) and [LLVM Gold](https://llvm.org/docs/GoldPlugin.html)
- For C/C++, Fuzz Introspector relies on [LLVM LTO](https://llvm.org/docs/LinkTimeOptimization.html)
- For Python, Fuzz Introspector relies on a modified [PyCG](https://github.com/vitsalis/PyCG)
- For Java, Fuzz Introspector relies on [Soot](https://soot-oss.github.io/soot/)

The consequence of this is your project must be compatible with these projects.
PyCG and Soot have not shown to be a blocker for many projects, however, experience
has shown that sometimes a project's build needs modification in order to compile
with LLVM LTO. The easiest way to test if your project works with LLVM is checking
whether your project can compile with the flags `-flto -fuse-ld=gold` and using
the gold linker. OSS-Fuzz automatically sets these flags and linker options when
whether your project can compile with the flags `-flto -fuse-ld=lld` and using
the LLD linker. OSS-Fuzz automatically sets these flags and linker options when
using `infra/helper.py` to build your project with `--sanitizer=introspector`, e.g.

```bash
Expand Down
25 changes: 13 additions & 12 deletions infra/base-images/base-builder/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -68,21 +68,25 @@ ENV BAZELISK_VERSION 1.9.0
RUN curl -L https://github.com/bazelbuild/bazelisk/releases/download/v$BAZELISK_VERSION/bazelisk-linux-amd64 -o /usr/local/bin/bazel && \
chmod +x /usr/local/bin/bazel

# Use '-Wno-unused-command-line-argument' to suppress "argument unused during compilation"
# warnings which are treated as errors by some projects.
ARG COMMON_SANITIZER_FLAGS="-fuse-ld=lld -Wno-unused-command-line-argument"

# Default build flags for various sanitizers.
ENV SANITIZER_FLAGS_address "-fsanitize=address -fsanitize-address-use-after-scope"
ENV SANITIZER_FLAGS_hwaddress "-fsanitize=hwaddress -fuse-ld=lld -Wno-unused-command-line-argument"
ENV SANITIZER_FLAGS_address "${COMMON_SANITIZER_FLAGS} -fsanitize=address -fsanitize-address-use-after-scope"
ENV SANITIZER_FLAGS_hwaddress "${COMMON_SANITIZER_FLAGS} -fsanitize=hwaddress"

# Set of '-fsanitize' flags matches '-fno-sanitize-recover' + 'unsigned-integer-overflow'.
ENV SANITIZER_FLAGS_undefined "-fsanitize=array-bounds,bool,builtin,enum,function,integer-divide-by-zero,null,object-size,return,returns-nonnull-attribute,shift,signed-integer-overflow,unsigned-integer-overflow,unreachable,vla-bound,vptr -fno-sanitize-recover=array-bounds,bool,builtin,enum,function,integer-divide-by-zero,null,object-size,return,returns-nonnull-attribute,shift,signed-integer-overflow,unreachable,vla-bound,vptr"
ENV SANITIZER_FLAGS_undefined "${COMMON_SANITIZER_FLAGS} -fsanitize=array-bounds,bool,builtin,enum,function,integer-divide-by-zero,null,object-size,return,returns-nonnull-attribute,shift,signed-integer-overflow,unsigned-integer-overflow,unreachable,vla-bound,vptr -fno-sanitize-recover=array-bounds,bool,builtin,enum,function,integer-divide-by-zero,null,object-size,return,returns-nonnull-attribute,shift,signed-integer-overflow,unreachable,vla-bound,vptr"

# Don't include "function" since it is unsupported on aarch64.
ENV SANITIZER_FLAGS_undefined_aarch64 "-fsanitize=array-bounds,bool,builtin,enum,integer-divide-by-zero,null,object-size,return,returns-nonnull-attribute,shift,signed-integer-overflow,unsigned-integer-overflow,unreachable,vla-bound,vptr -fno-sanitize-recover=array-bounds,bool,builtin,enum,integer-divide-by-zero,null,object-size,return,returns-nonnull-attribute,shift,signed-integer-overflow,unreachable,vla-bound,vptr"
ENV SANITIZER_FLAGS_undefined_aarch64 "${COMMON_SANITIZER_FLAGS} -fsanitize=array-bounds,bool,builtin,enum,integer-divide-by-zero,null,object-size,return,returns-nonnull-attribute,shift,signed-integer-overflow,unsigned-integer-overflow,unreachable,vla-bound,vptr -fno-sanitize-recover=array-bounds,bool,builtin,enum,integer-divide-by-zero,null,object-size,return,returns-nonnull-attribute,shift,signed-integer-overflow,unreachable,vla-bound,vptr"

ENV SANITIZER_FLAGS_memory "-fsanitize=memory -fsanitize-memory-track-origins"
ENV SANITIZER_FLAGS_memory "${COMMON_SANITIZER_FLAGS} -fsanitize=memory -fsanitize-memory-track-origins"

ENV SANITIZER_FLAGS_thread "-fsanitize=thread"
ENV SANITIZER_FLAGS_thread "${COMMON_SANITIZER_FLAGS} -fsanitize=thread"

ENV SANITIZER_FLAGS_introspector "-O0 -flto -fno-inline-functions -fuse-ld=gold -Wno-unused-command-line-argument"
ENV SANITIZER_FLAGS_introspector "${COMMON_SANITIZER_FLAGS} -O0 -flto -fno-inline-functions"

# Do not use any sanitizers in the coverage build.
ENV SANITIZER_FLAGS_coverage ""
Expand All @@ -95,11 +99,8 @@ ENV UBSAN_OPTIONS="silence_unsigned_overflow=1"
ENV DFSAN_OPTIONS='warn_unimplemented=0'

# Default build flags for coverage feedback.
ENV COVERAGE_FLAGS="-fsanitize=fuzzer-no-link"

# Use '-Wno-unused-command-line-argument' to suppress "warning: -ldl: 'linker' input unused"
# messages which are treated as errors by some projects.
ENV COVERAGE_FLAGS_coverage "-fprofile-instr-generate -fcoverage-mapping -pthread -Wl,--no-as-needed -Wl,-ldl -Wl,-lm -Wno-unused-command-line-argument"
ENV COVERAGE_FLAGS="${COMMON_SANITIZER_FLAGS} -fsanitize=fuzzer-no-link"
ENV COVERAGE_FLAGS_coverage "${COMMON_SANITIZER_FLAGS} -fprofile-instr-generate -fcoverage-mapping -pthread -Wl,--no-as-needed -Wl,-ldl -Wl,-lm"

# Default sanitizer, fuzzing engine and architecture to use.
ENV SANITIZER="address"
Expand Down
10 changes: 1 addition & 9 deletions projects/connectedhomeip/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,6 @@
################################################################################


# workaround to get Fuzz Introspector to build; making it link with lld instead of the environment's gold linker which gives an error
if [ "$SANITIZER" == "introspector" ]; then
export CFLAGS=$(echo "$CFLAGS" | sed 's/gold/lld/g')
export CXXFLAGS=$(echo "$CXXFLAGS" | sed 's/gold/lld/g')
fi

cd $SRC/connectedhomeip

# Activate Pigweed environment
Expand All @@ -42,15 +36,13 @@ export PATH="/src/connectedhomeip/.environment/cipd/packages/zap/:$PATH"
# error on GenericConnectivityManagerImpl_Thread.ipp and current fuzzing
# does not differentiate between thread/Wifi/TCP/UDP/BLE connectivity
# implementations.
# - `target_ldflags` forces compiler to use LLVM's linker
gn gen out/fuzz_targets \
--args="
oss_fuzz=true \
is_clang=true \
enable_rtti=true \
chip_enable_thread_safety_checks=false \
chip_enable_openthread=false \
target_ldflags=[\"-fuse-ld=lld\"]"
chip_enable_openthread=false

# Deactivate Pigweed environment to use OSS-Fuzz toolchains
deactivate
Expand Down
10 changes: 5 additions & 5 deletions projects/dbus-broker/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,12 @@ MESON_CXXFLAGS=${CXXFLAGS:-}
MESON_LDFLAGS=${LDFLAGS:-}

if [[ "$SANITIZER" == introspector ]]; then
MESON_CFLAGS="${MESON_CFLAGS//-fuse-ld=gold/ }"
MESON_CXXFLAGS="${MESON_CXXFLAGS//-fuse-ld=gold/ }"
MESON_LDFLAGS="${MESON_LDFLAGS//-fuse-ld=gold/ }"
MESON_CFLAGS="${MESON_CFLAGS//-fuse-ld=lld/ }"
MESON_CXXFLAGS="${MESON_CXXFLAGS//-fuse-ld=lld/ }"
MESON_LDFLAGS="${MESON_LDFLAGS//-fuse-ld=lld/ }"
MESON_LDFLAGS+=" -flto"
export CC_LD=gold
export CXX_LD=gold
export CC_LD=lld
export CXX_LD=lld
fi

apt-get update -y
Expand Down
3 changes: 0 additions & 3 deletions projects/ecc-diff-fuzzer/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,6 @@
#
################################################################################

# use a linker that supports Dwarf v5
export LDFLAGS="-fuse-ld=lld"

# build projects
#nettle
(
Expand Down
14 changes: 7 additions & 7 deletions projects/ffmpeg/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -45,19 +45,19 @@ else
export PKG_CONFIG_PATH="$FFMPEG_DEPS_PATH/$LIBDIR/pkgconfig:$FFMPEG_DEPS_PATH/lib/pkgconfig"
fi

# The option `-fuse-ld=gold` can't be passed via `CFLAGS` or `CXXFLAGS` because
# The option `-fuse-ld=` can't be passed via `CFLAGS` or `CXXFLAGS` because
# Meson injects `-Werror=ignored-optimization-argument` during compile tests.
# Remove the `-fuse-ld=` and let Meson handle it.
# https://github.com/mesonbuild/meson/issues/6377#issuecomment-575977919
export MESON_CFLAGS="$CFLAGS"
if [[ "$CFLAGS" == *"-fuse-ld=gold"* ]]; then
export MESON_CFLAGS="${CFLAGS//-fuse-ld=gold/}"
export CC_LD=gold
if [[ "$CFLAGS" == *"-fuse-ld=lld"* ]]; then
export MESON_CFLAGS="${CFLAGS//-fuse-ld=lld/}"
export CC_LD=lld
fi
export MESON_CXXFLAGS="$CXXFLAGS"
if [[ "$CXXFLAGS" == *"-fuse-ld=gold"* ]]; then
export MESON_CXXFLAGS="${CXXFLAGS//-fuse-ld=gold/}"
export CXX_LD=gold
if [[ "$CXXFLAGS" == *"-fuse-ld=lld"* ]]; then
export MESON_CXXFLAGS="${CXXFLAGS//-fuse-ld=lld/}"
export CXX_LD=lld
fi

meson_install() {
Expand Down
2 changes: 0 additions & 2 deletions projects/halide/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,6 @@
################################################################################
set euox pipefail

export LDFLAGS="-fuse-ld=lld"

declare -A LLVM_SANITIZER=( ["address"]="Address" ["undefined"]="Undefined" ["memory"]="Memory" )

if [[ -v LLVM_SANITIZER[$SANITIZER] ]]; then
Expand Down
2 changes: 1 addition & 1 deletion projects/libiec61850/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ $CC $CFLAGS $LIB_FUZZING_ENGINE ../fuzz/fuzz_mms_decode.c -c \
-I../hal/inc -I../src/logging


$CXX $CXXFLAGS -fuse-ld=lld $LIB_FUZZING_ENGINE fuzz_mms_decode.o -o $OUT/fuzz_mms_decode ./src/libiec61850.a ./hal/libhal.a
$CXX $CXXFLAGS $LIB_FUZZING_ENGINE fuzz_mms_decode.o -o $OUT/fuzz_mms_decode ./src/libiec61850.a ./hal/libhal.a

# Copy over the options file
cp $SRC/fuzz_decode.options $OUT/fuzz_decode.options
2 changes: 1 addition & 1 deletion projects/llhttp/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,4 @@ npm ci
yes | make build/libllhttp.a

$CC $CFLAGS -c ./test/fuzzers/fuzz_parser.c -I./build/ ./build/libllhttp.a -o $WORK/fuzz_parser.o
$CXX $CXXFLAGS $LIB_FUZZING_ENGINE -fuse-ld=lld -I./build/ ./build/libllhttp.a $WORK/fuzz_parser.o -o $OUT/fuzz_parser
$CXX $CXXFLAGS $LIB_FUZZING_ENGINE -I./build/ ./build/libllhttp.a $WORK/fuzz_parser.o -o $OUT/fuzz_parser
14 changes: 7 additions & 7 deletions projects/mpv/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -47,17 +47,17 @@ make -j`nproc`
make install
popd

# The option `-fuse-ld=gold` can't be passed via `CFLAGS` or `CXXFLAGS` because
# The option `-fuse-ld=` can't be passed via `CFLAGS` or `CXXFLAGS` because
# Meson injects `-Werror=ignored-optimization-argument` during compile tests.
# Remove the `-fuse-ld=` and let Meson handle it.
# https://github.com/mesonbuild/meson/issues/6377#issuecomment-575977919
if [[ "$CFLAGS" == *"-fuse-ld=gold"* ]]; then
export CFLAGS="${CFLAGS//-fuse-ld=gold/}"
export CC_LD=gold
if [[ "$CFLAGS" == *"-fuse-ld=lld"* ]]; then
export CFLAGS="${CFLAGS//-fuse-ld=lld/}"
export CC_LD=lld
fi
if [[ "$CXXFLAGS" == *"-fuse-ld=gold"* ]]; then
export CXXFLAGS="${CXXFLAGS//-fuse-ld=gold/}"
export CXX_LD=gold
if [[ "$CXXFLAGS" == *"-fuse-ld=lld"* ]]; then
export CXXFLAGS="${CXXFLAGS//-fuse-ld=lld/}"
export CXX_LD=lld
fi

pushd $SRC/mpv
Expand Down
2 changes: 0 additions & 2 deletions projects/ntp/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,6 @@
################################################################################

cd ntp-dev
#avoids https://bugs.llvm.org/show_bug.cgi?id=34636
cp /usr/bin/ld.gold /usr/bin/ld
./bootstrap
./configure --enable-fuzztargets
make
Expand Down
4 changes: 0 additions & 4 deletions projects/tdengine/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,6 @@ sed -i 's/-Werror//g' ./cmake/define.inc
mkdir debug && cd debug
export LDFLAGS="${CXXFLAGS}"

if [[ $SANITIZER = *coverage* ]]; then
ln -f -s /usr/bin/gold /usr/bin/ld
fi

cmake -DBUILD_HTTP=true ..
cmake --build .

Expand Down
4 changes: 0 additions & 4 deletions projects/thrift/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,6 @@
# build project
export ASAN_OPTIONS=detect_leaks=0

if [ "$SANITIZER" = "coverage" ]
then
cp /usr/bin/ld.gold /usr/bin/ld
fi
./bootstrap.sh
# rust fails compilation with clippy warnings
./configure --with-rs=no
Expand Down
Loading