Some experience for GroupTuner Paper Reproduction on Ubuntu 24.04

published: 10/27/2025 and updated: 9/7/2026

FYP - UL FYP - UL , Compiler Optimisation , Ubuntu

Overview#

Update, September 2026: The FYP is finished, so I have finally filled in the results and added what happened after the reproduction. The setup notes below describe my October 2025 environment; the final Clang experiments used the separate setup described in the update.

This document records the process and my experience of successfully deploying and running GroupTuner tests on Ubuntu 24.04.3, including all issues encountered and their solutions. The original paper’s experiments were based on Ubuntu 20.04, so some compatibility issues need to be addressed.

Environment Information#

• Operating System: Ubuntu 24.04.3 LTS (Noble)
• Kernel: 6.14.0-33-generic
• System GCC: 13.3.0
• Python: 3.9 (conda environment)
• Processor used: AMD Ryzen 7 6800H. This was my test machine, not a requirement to avoid Intel CPUs.
• Repository: PKU-ASAL/GroupTuner: This is an implementation of GroupTuner described in LCTES 2025 paper.

Reproduction Process#

Step 1: System Preparation#

1.1 Check CPU boost#

Correction, September 2026: I originally skipped this because my CPU was AMD. That was wrong: AMD processors also have boost behaviour, usually called Core Performance Boost. The controls depend on the active scaling driver; the Linux CPU frequency documentation explains the available interfaces.

For repeatable timing, check and record the boost setting as part of the experiment setup. The final FYP report records boost as disabled; that does not retrospectively verify the setting in this earlier reproduction.

1.2 Isolate CPU Core 0#

# Edit grub configuration
sudo vim /etc/default/grub

# Add isolcpus=0 to GRUB_CMDLINE_LINUX line
# Example: GRUB_CMDLINE_LINUX="isolcpus=0"

# Update grub and reboot
sudo update-grub
sudo reboot

# Verify after reboot
cat /proc/cmdline | grep isolcpus

Step 2: Install Perf Tool#

2.1 Issue: Kernel Version Mismatch#

# Install perf
sudo apt update
sudo apt install -y linux-tools-common linux-tools-generic linux-tools-$(uname -r)

# Issue: Shows "perf not found for kernel 6.14.0-33"
# This was the package state encountered in my October 2025 environment.

Issue: I could not get a matching perf package in this environment. Workaround used at the time: Run the available 6.8 perf binary. This is a record of my setup, not a guarantee that every perf feature works across kernel versions.

2.2 Historical workaround: Create Perf Symbolic Link#

I would now try the matching distribution package first, or invoke an alternative binary by its full path while checking the required events. The commands below record the workaround I used; replacing the system wrapper is not necessary for that check.

# Check available perf versions
ls -la /usr/lib/linux-tools/

# Create symbolic link to perf from 6.8.0 version
sudo ln -sf /usr/lib/linux-tools-6.8.0-86/perf /usr/bin/perf

# Verify
perf --version
# Output: perf version 6.8.12

# Test perf functionality
taskset -c 0 perf stat -- ls

2.3 Configure Perf Permissions#

# Set perf permissions (allow non-root users)
sudo sh -c 'echo -1 > /proc/sys/kernel/perf_event_paranoid'

# Verify setting
cat /proc/sys/kernel/perf_event_paranoid
# Should output: -1

Note: This setting resets after reboot. To make it persistent, add to /etc/sysctl.conf:

echo "kernel.perf_event_paranoid = -1" | sudo tee -a /etc/sysctl.conf

Step 3: Install GCC 9.2.0#

3.1 Issue: GCC Compilation Failure#

In my Ubuntu 24.04 environment, gcc_install.sh failed while building libsanitizer. Disabling that library let me continue. I did not isolate whether the failure came from the kernel, system headers or another toolchain difference, so my original explanation blaming the new kernel was too definite. This workaround also leaves the corresponding sanitizer runtime unavailable in that GCC build.

Error Message Example:

sanitizer_platform_limits_posix.cc: error: ...
make[4]: *** [Makefile:xxx: sanitizer_platform_limits_posix.o] Error 1

3.2 Solution: Use Script to Download, Then Compile Manually#

cd /path/to/GroupTuner

# Step 1: Use script to download and extract GCC (will fail automatically, but downloads files)
chmod +x gcc_install.sh
./gcc_install.sh
# Expected: Fails at libsanitizer, but gcc-9.2.0 source code downloaded and patched

# Step 2: Clean failed build, recompile with correct configuration
cd gcc_install
rm -rf gcc-build  # Remove failed build directory
mkdir gcc-build && cd gcc-build

# Step 3: Reconfigure (KEY: add --disable-libsanitizer)
../gcc-9.2.0/configure \
  --prefix="$PWD/build" \
  --enable-languages=c,c++ \
  --disable-multilib \
  --disable-bootstrap \
  --disable-libsanitizer

# Step 4: Compile and install (it took about 30mins on my machine)
make -j"$(nproc)"
make install

If Starting from zero:

cd /path/to/GroupTuner

# Install dependencies
sudo apt-get update
sudo apt-get install -y build-essential gcc-multilib

# Create directory and download
mkdir -p gcc_install && cd gcc_install
wget http://mirrors.ustc.edu.cn/gnu/gcc/gcc-9.2.0/gcc-9.2.0.tar.gz
tar -xvf gcc-9.2.0.tar.gz
cd gcc-9.2.0

# Change download script to http (faster for China mirrors)
sed -i 's/ftp/http/g' contrib/download_prerequisites

# Fix libsanitizer compatibility (comment out line in sed script)
sed -e '1161 s|^|//|' -i libsanitizer/sanitizer_common/sanitizer_platform_limits_posix.cc

# Download GCC dependencies (GMP, MPFR, MPC, ISL)
./contrib/download_prerequisites

# Create build directory
cd ..
mkdir gcc-build && cd gcc-build

# Configure (disable libsanitizer)
../gcc-9.2.0/configure \
  --prefix="$PWD/build" \
  --enable-languages=c,c++ \
  --disable-multilib \
  --disable-bootstrap \
  --disable-libsanitizer

# Compile and install
make -j"$(nproc)"
make install

cd ../..

Important Notes:

• --disable-libsanitizer: the workaround used for the libsanitizer build failure in this environment
• --disable-bootstrap: Skips 3-stage bootstrap, speeds up compilation (may work)
• --disable-multilib: Only compile 64-bit version
• Both methods work, first one recommended (uses script’s download and patch logic)

3.3 Verify GCC Installation#

# Verify GCC version
./gcc_install/gcc-build/build/bin/gcc --version
# Should output: gcc (GCC) 9.2.0

# Test compilation
echo 'int main(){return 0;}' > test.c
./gcc_install/gcc-build/build/bin/gcc -O3 test.c -o test
./test && echo "GCC works!"
rm test test.c

Issue: Must use GCC 9.2.0 to ensure consistency with paper results. Different GCC versions will generate different binary code, affecting performance measurements

Step 4: Configure Python Environment#

4.1 Create Conda Environment#

# Create Python 3.9 environment
conda create -n grouptuner python=3.9
conda activate grouptuner

4.2 Install Python Dependencies#

cd /path/to/GroupTuner

# Install basic dependencies
pip install -r requirements.txt

# Manually install possibly missing packages (requirements.txt incomplete)
pip install sqlalchemy psutil

Issue: Original requirements.txt missing sqlalchemy Solution: Manually install missing dependencies

4.3 Configure PYTHONPATH#

# Set PYTHONPATH (needed before each run)
export PYTHONPATH=/home/mio/projects:$PYTHONPATH

# Or add to ~/.zshrc or ~/.bashrc
echo 'export PYTHONPATH=/home/mio/projects:$PYTHONPATH' >> ~/.zshrc

Issue: Code uses from GroupTuner.Dataset... imports Reason: Project structure requires parent directory in PYTHONPATH

Step 5: Generate Baseline Files#

This is just one of the methods, I don’t know it is correct or not, but it works. The original paper does not specify the code implementation, but I think my process is generally fine.

5.1 Issue: Dataset Number Mismatch#

Finding: Code uses .__run 1 (dataset 1), but some programs may not have dataset 1

5.2 Generate Baseline for Test Programs#

cd /path/to/GroupTuner

# GCC path
GCC_PATH="$PWD/gcc_install/gcc-build/build/bin/gcc"

# Generate baseline for automotive_qsort1
cd Dataset/dataset/cBench_V1.1/automotive_qsort1/src
make -f Makefile.gcc clean
make -f Makefile.gcc CC="$GCC_PATH" CFLAGS="-O3 -w"
taskset -c 0 ./__run 1
cp -f ftmp_out sorted_output.dat
echo "automotive_qsort1 baseline created"

# Generate baseline for bzip2d
cd ../../../bzip2d/src
make -f Makefile.gcc clean
make -f Makefile.gcc CC="$GCC_PATH" CFLAGS="-O3 -w"
taskset -c 0 ./__run 1
cp -f ftmp_out sorted_output.dat
echo "bzip2d baseline created"

# Generate baseline for network_dijkstra
cd ../../../network_dijkstra/src
make -f Makefile.gcc clean
make -f Makefile.gcc CC="$GCC_PATH" CFLAGS="-O3 -w"
taskset -c 0 ./__run 1

# network_dijkstra outputs to output_large.dat
cp -f output_large.dat sorted_output.dat
echo "network_dijkstra baseline created"

5.3 Verify Baseline Files#

cd /path/to/GroupTuner

for prog in automotive_qsort1 bzip2d network_dijkstra; do
    echo "=== $prog ==="
    ls -lh Dataset/dataset/cBench_V1.1/$prog/src/sorted_output.dat
    md5sum Dataset/dataset/cBench_V1.1/$prog/src/sorted_output.dat
done

Output recorded in the original run (paths, binary sizes and hashes are specific to that build):

=== automotive_qsort1 ===
-rw-rw-r-- 1 user user 59 Oct 27 17:23 ...sorted_output.dat
e81aa2dde0051befae4b8a7e7ebbd930  ...sorted_output.dat

=== bzip2d ===
-rw-rw-r-- 1 user user 52M Oct 27 17:23 ...sorted_output.dat
f4776bab7561a6a4a3e8260b5031872d  ...sorted_output.dat

=== network_dijkstra ===
-rw-rw-r-- 1 user user 264 Oct 27 17:24 ...sorted_output.dat
0ee2aafafbcb64a9f3a402be9f22d377  ...sorted_output.dat

Baseline files must be generated with dataset 1, and must use GCC 9.2.0

Step 6: Run Experiments#

6.1 Test Run (10-50 rounds)#

cd /path/to/GroupTuner

# Activate conda environment
source ~/miniconda3/etc/profile.d/conda.sh
conda activate grouptuner

# Set PYTHONPATH
export PYTHONPATH=/home/mio/projects:$PYTHONPATH

# Run cBench test (10 rounds for quick verification)
python main_cbench.py --gcc-path $PWD/gcc_install/gcc-build/build/bin/gcc --round 10

Output recorded in the original run (paths, binary sizes and hashes are specific to that build):

automotive_qsort1 GroupTuner:
[-1] O3 time:1.396s
[0] default time: 1.402s
[1] current trial: 1.388s, best performance so far: 1.388s
[2] current trial: 1.401s, best performance so far: 1.388s
...

6.2 Full Experiment (500 rounds)#

# cBench experiment
python main_cbench.py --gcc-path $PWD/gcc_install/gcc-build/build/bin/gcc --round 500

# PolyBench experiment (need to generate polybench baseline first)
python main_poly.py --gcc-path $PWD/gcc_install/gcc-build/build/bin/gcc --round 500

Note:

• Experiment cannot be interrupted, if interrupted must restart from beginning
• Full 500-round experiment may takes several hours to several days (I just ran it for 500 rounds, took around 1.5 hrs)
• Recommend using nohup or screen to run in background

6.3 Background Execution Example#

# Use nohup for background execution
nohup python main_cbench.py --gcc-path $PWD/gcc_install/gcc-build/build/bin/gcc --round 500 > cbench.log 2>&1 &

# Or use screen
screen -S grouptuner
python main_cbench.py --gcc-path $PWD/gcc_install/gcc-build/build/bin/gcc --round 500
# Ctrl+A, D to detach session

Step 7: Results#

I can finally replace the TBA here. The numbers below come from Section 4.2.6 of my final FYP report, rather than the short test run shown above.

Under the original -O3 wall-clock setting, GroupTuner ranked first on four of the six benchmarks. With -O1 and cycle counts, it ranked first on five. The improvements were smaller than those in the original paper under the comparable setting, so this was not an exact reproduction of its numbers.

Also, the jump from 4.50% to 22.30% does not mean that changing the baseline made the algorithm five times better. -O1 leaves more optimisation work to the tuner. These are different reference points and should not be compared as though they were the same experiment.

The cycle-count measurements had lower variation in my GCC experiments. This was useful for the next part of the project, where I used perf to evaluate Clang flag combinations.

Main Differences from Ubuntu 20.04#

After GroupTuner: Finishing the FYP#

When I wrote this post, I was still getting the reproduction environment to work. The plan was to modify GroupTuner and move the approach to Clang. The final project ended up being a bit different.

The migration did not work as planned#

GroupTuner builds its groups from GCC source analysis. I could not directly carry that grouping method over to Clang: the relationship between command-line flags and the LLVM optimisation pipeline did not give me the same structure to work with.

So I kept the GCC reproduction as a reference and built a separate Clang autotuning framework. Calling the final implementation a successful port of GroupTuner would be inaccurate.

The pipeline became:

1. Extract Boolean flags from clang-18 --help-hidden, including positive and negative forms.
2. Compile the benchmarks with individual flags and compare binary hashes with the baseline.
3. Combine the effective flags and manually remove options whose purpose was instrumentation or diagnostics rather than optimisation.
4. Search combinations of the remaining flags with a genetic algorithm.

This left 39 flags for -O1 and 41 for -O3. The manual filtering step is worth mentioning: the pipeline was not completely automatic. A different binary also does not necessarily mean a faster program, and testing flags individually can miss effects that only appear in combination.

What I actually measured#

The final Clang experiments used Clang 18 and PolyBench/C 4.2.1 on an AMD Ryzen 7 6800H machine running Ubuntu 24.04. The report records CPU isolation, pinning to core 0 and disabled boost for this setup. This is separate from the early GCC installation notes above.

The GA used a population of 30 for 50 generations, with adaptive crossover and mutation and an elitism strategy with a 2% noise tolerance. Each candidate was measured 30 times using perf stat -e cycles, taking the median. For each benchmark, I ran ten independent GA searches with different seeds.

These are the results from Tables 5.3 and 5.4 of the final report:

Here the percentage is (baseline median cycles / candidate median cycles - 1) × 100, averaged over the best fitness from each of the ten searches. It is not the percentage reduction in execution time. For example, +101.6% corresponds to about 2.016 times the baseline-to-candidate cycle ratio, not a runtime reduction of more than 100%.

The average also hides quite different behaviour. 2mm accounts for a large part of the -O3 improvement, while jacobi-2d only improves by 8.1%. I would not turn the average into a claim that an arbitrary application will get 26% faster.

Was the improved GA actually better?#

Not in the clear-cut way the name might suggest.

PyGAD reached similar results on the same search space. I also adapted BOCA and SATuner and compared them with my GA using 1,500 evaluations and seed 42. BOCA led on four benchmarks, my GA on gemm, and SATuner on jacobi-2d. That comparison used one seed, unlike the ten-run GA results above.

The more interesting observation was that fitness settled down early while the chromosomes remained quite different. Many flag combinations produced similar performance. In the report’s analysis, only nine of the 39 flags showed a noticeable association with fitness, and some of that association came from compilation failures rather than runtime performance.

This suggests that, for this particular flag set, choosing a more elaborate search algorithm was not necessarily the main issue. I also tried a larger population and a longer run, but did not see a consistent improvement across the benchmarks. That is useful evidence for this experiment, not proof that larger searches are generally unnecessary.

Things left unfinished#

There are still limits to the results. I only evaluated six numerical kernels on one machine. atax had noticeably higher measurement noise, so the fixed 2% tolerance was not equally suitable for every benchmark. The BOCA/SATuner comparison needs multiple independent runs, and I did not complete formal significance tests for the performance comparisons before submission.

Some flags also change floating-point behaviour or security-related code generation. A lower cycle count alone is not enough to decide that such a configuration is suitable for another application; its correctness and other requirements would need checking separately.

If I continue this work, I would first extend the evaluation and check those comparisons more carefully. LLVM pass-level tuning is another possible direction, since it would let me explore beyond the command-line flag space used here.

For now, that is where the FYP ended. The GCC reproduction worked, the direct migration did not, and the final implementation became a different Clang framework. This post started with getting the tools to run; these results finish the record of what I used them for.

A shorter overview is on the projects page.