LLVM Compiler/Printable version

LLVM Compiler

The current, editable version of this book is available in Wikibooks, the open-content textbooks collection, at
https://en.wikibooks.org/wiki/LLVM_Compiler

Permission is granted to copy, distribute, and/or modify this document under the terms of the Creative Commons Attribution-ShareAlike 3.0 License.

Installation

Installing Pre-build Binaries

on Ubuntu

• sudo apt-get install clang llvm

Installing from github master

Hardware/Operating system

• Azure VM | Standard NC6_Promo (6 vcpus, 56 GiB memory)
• Tesla K80, which has compute capability 3.7. You can find out what card you got via lshw -C display; you can find out the compute capability of your card https://developer.nvidia.com/cuda-gpus .
• Linux (ubuntu 18.04)

Prerequisites

sudo apt update
sudo apt install build-essential
sudo apt install cmake
sudo apt install -y libelf-dev libffi-dev
sudo apt install -y pkg-config

Install CUDA 10.2 , using instructions https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=debnetwork

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
sudo add-apt-repository "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"
sudo apt-get update
sudo apt-get -y install cuda

After installation, export two environment variables, vim ~/.bashrc to add two lines

• export PATH=$PATH:/usr/local/cuda-10.2/bin/
• export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.2/lib/

Get source packages

Assuming you are in your home directory

cd

git clone https://github.com/llvm/llvm-project.git

Build using GCC

mkdir build

cd build/

cmake  -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;libcxx;libcxxabi;lld;openmp"   \ 
    -DCMAKE_BUILD_TYPE=Release   \
    -DLLVM_TARGETS_TO_BUILD="X86;NVPTX"     \
    -DCMAKE_INSTALL_PREFIX=$(pwd)/../llvm   \
    -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_37    \
    -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=35,37,50,52,60,61,70,75  \
    -DCMAKE_C_COMPILER=gcc    \
    -DCMAKE_CXX_COMPILER=g++   \
    -G "Unix Makefiles" ../llvm-project/llvm

time make -j
time make -j install

Explanation for configuration options

• -DCMAKE_C_COMPILER=gcc // the C compiler used to compile clang/llvm, GCC
• -DCMAKE_CXX_COMPILER=g++ // the C++ compiler used to compile clang/llvm: G++
• -DLLVM_TARGETS_TO_BUILD=X86;PowerPC;NVPTX;AMDGPU // explicitly specify target devices to support, Intel ,Nvidia, IBM, and AMD CPUs or GPUs
• -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_70 // default GPU computing capability version to support, https://developer.nvidia.com/cuda-gpus lists such information.
• -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=37,60,70 // all GPU computing capability versions to build in libomptarget

Other optional options

• -DGCC_INSTALL_PREFIX=${GCC_PATH}
• -DCMAKE_C_COMPILER=${GCC_PATH}/bin/gcc
• -DCMAKE_CXX_COMPILER=${GCC_PATH}/bin/g++
• -DCMAKE_Fortran_COMPILER=${GCC_PATH}/bin/gfortran
• -DCUDA_PATH= // this option should be automatically set if cuda is in your search path
• -DCUDA_TOOLKIT_ROOT_DIR= // this option should be automatically set if cuda is in your search path
• -DOPENMP_ENABLE_LIBOMPTARGET=ON // this should be on by default
• -DLIBOMP_FORTRAN_MODULES=ON
• -DBUILD_SHARED_LIBS=OFF // turn off shared libs

More examples

• cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;libcxx;libcxxabi;lld;openmp" -DCMAKE_INSTALL_PREFIX=/Users/abc/llvm-research/inst-10.0.1 -DCMAKE_BUILD_TYPE=Debug ../llvm/
• ninja -j8
• ninja install

Rebuild using Clang

add the path to the installed clang, vim ~/.bashrc

export PATH=~/llvm/bin:$PATH
export LD_LIBRARY_PATH=~/llvm/lib:$LD_LIBRARY_PATH

cd 
build-openmp
cd build-openmp/

cmake -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;libcxx;libcxxabi;lld;openmp"  \
 -DCMAKE_BUILD_TYPE=Release \
 -DLLVM_TARGETS_TO_BUILD="X86;NVPTX"  \
 -DCMAKE_INSTALL_PREFIX=$(pwd)/../llvm  \
 -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_37   \
 -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=35,37,50,52,60,61,70,75    \
 -DCMAKE_C_COMPILER=clang  \
 -DCMAKE_CXX_COMPILER=clang++   \
 -G "Unix Makefiles" ../llvm-project/llvm

 make -j
 make -j install

Optional options, explictly turn on bitcode lib, and the compiler/linker to build it

• -DLIBOMPTARGET_NVPTX_ENABLE_BCLIB=true
• -DLIBOMPTARGET_NVPTX_CUDA_COMPILER=${PREFIX}/bin/clang
• -DLIBOMPTARGET_NVPTX_BC_LINKER=${PREFIX}/bin/llvm-link

Installing from releases

Hardware/Operating system

• Azure VM | Standard NC6_Promo (6 vcpus, 56 GiB memory)
• Tesla K80, which has compute capability 3.7. You can find out what card you got via lshw -C display; you can find out the compute capability of your card here.
• Linux (ubuntu 18.04)

Prerequisites

sudo apt update
sudo apt install build-essential
sudo apt install cmake
sudo apt install -y libelf-dev libffi-dev
sudo apt install -y pkg-config

Install CUDA 10.2 , using instructions https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=debnetwork

Assuming you current path is

• /home/ubuntu/omp5-gpu-llvm

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
sudo add-apt-repository "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"
sudo apt-get update
sudo apt-get -y install cuda

After installation, export two environment variables, vim ~/.bashrc to add two lines

• export PATH=$PATH:/usr/local/cuda-10.2/bin/
• export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.2/lib/

Get source packages

Three steps to download, untar, and put them into the right locations.

wget https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/llvm-10.0.0.src.tar.xz
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/clang-10.0.0.src.tar.xz
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/openmp-10.0.0.src.tar.xz
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/compiler-rt-10.0.0.src.tar.xz


tar xf llvm-10.0.0.src.tar.xz
tar xf clang-10.0.0.src.tar.xz
tar xf openmp-10.0.0.src.tar.xz
tar xf compiler-rt-10.0.0.src.tar.xz

mv clang-10.0.0.src llvm-10.0.0.src/tools/clang
mv openmp-10.0.0.src llvm-10.0.0.src/projects/openmp
mv compiler-rt-10.0.0.src llvm-10.0.0.src/projects/compiler-rt

In the end, the directory layout should look like

• llvm-10.0.0.src
  • tools/clang
  • projects/openmp
  • projects/compiler-rt

Build the Compiler with OpenMP offloading support

You need to know the Compute Capability version of your GPU. https://developer.nvidia.com/cuda-gpus lists such information. For example, some typical GPUs and their CC versions are:

• Tesla K80 3.7, sm_37
• Tesla P100 6.0, sm_37
• Tesla V100 7.0, sm_37

mkdir build
cd build

# this step is to generate a make file using cmake. picking gcc/g++ as the compiler
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$(pwd)/../install \
	-DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_37 \
        -DCMAKE_C_COMPILER=gcc    \
        -DCMAKE_CXX_COMPILER=g++   \
	-DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=37,60,70 ../llvm-10.0.0.src

# the screen output of the step above should show the following info: 
# -- Found LIBOMPTARGET_DEP_CUDA_DRIVER: /usr/lib/x86_64-linux-gnu/libcuda.so
# -- LIBOMPTARGET: Building offloading runtime library libomptarget.
# -- LIBOMPTARGET: Building CUDA offloading plugin.
# -- LIBOMPTARGET: Building x86_64 offloading plugin.

make -j6

make install -j6

After the installation, you should expand your PATH and LD_LIBRARY_PATH again

• export PATH=$PATH:/home/ubuntu/omp5-gpu-llvm/install/bin
• export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/ubuntu/omp5-gpu-llvm/install/lib

Rebuild the OpenMP runtime libraries

You should use the freshly installed clang to rebuild the OpenMP runtime library

cd /home/ubuntu/omp5-gpu-llvm
mkdir build-openmp
cd build-openmp

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$(pwd)/../install \
	-DCMAKE_C_COMPILER=$(pwd)/../install/bin/clang \
	-DCMAKE_CXX_COMPILER=$(pwd)/../install/bin/clang++ \
	-DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=37,60,70 \
	../llvm-10.0.0.src/projects/openmp

make -j6
make install -j6

Test the installation

save the following code into a file named ongpu.c

#include <stdio.h>
#include <omp.h>

int main()
{
  int runningOnGPU = 0;
  /* Test if GPU is available using OpenMP4.5 */
#pragma omp target map(from:runningOnGPU)
  {
    if (omp_is_initial_device() == 0)
      runningOnGPU = 1;
  }
  /* If still running on CPU, GPU must not be available */
  if (runningOnGPU)
    printf("### Able to use the GPU! ### \n");
  else
    printf("### Unable to use the GPU, using CPU! ###\n");

  return 0;
}

Compile and run it

clang -fopenmp -fopenmp-targets=nvptx64-nvidia-cuda ongpu.c

 ./a.out

### Able to use the GPU! ###

Troubleshooting

error while loading libomp.so

./a.out: error while loading shared libraries: libomp.so: cannot open shared object file: No such file or directory solution

• export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/pathTo/installed-llvm/lib

External Links

• OpenMP 5 GPU offloading instructions [Fabio]: https://github.com/devitocodes/devito/wiki/OpenMP-5-GPU-offloading-instructions--%5BFabio%5D
  • Clang_with_OpenMP_Offloading_to_NVIDIA_GPUs: https://hpc-wiki.info/hpc/Building_LLVM/Clang_with_OpenMP_Offloading_to_NVIDIA_GPUs
• OpenMP 5 GPU offloading instructions with Clang 11 [Italo] https://github.com/devitocodes/devito/wiki/OpenMP-5-GPU-offloading-instructions-with-Clang-11-%5BItalo%5D
• a docker file showing LLVM installation: https://github.com/freeCompilerCamp/play-with-compiler/blob/master/dockerfiles/dind/Dockerfile.llvm10

Clang Plugins

How to Debug a Plugin

There are two ways to run a Clang plugin:

• call the clang frontend directly: clang -cc1 ...
• call the compiler driver, clang -v, to find out real command lines

OpenMP Support

With the release of Clang 3.8.0, OpenMP 3.1 support is enabled in Clang by default, and the OpenMP runtime is therefore built as a normal part of the Clang build, and distributed with the binary distributions. You do not, therefore, need explicitly to check out this code, or build it out of tree; a normal Clang check out and build will automatically include building these runtime libraries.

https://openmp.llvm.org/

installation

cmake -G Ninja -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=$LLVM_PATH -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;compiler-rt;openmp" -DCLANG_BUILD_EXAMPLES=1 $LLVM_SRC/llvm

ninja install -j8 -l8

OpenMP lowering or Code Geneneration

Hello example

cat omp_hello.c 
#include <omp.h>
#include <stdio.h>

int main (int argc, char *argv[]) 
{
  int nthreads, tid;

#pragma omp parallel private(nthreads, tid)
  {

    tid = omp_get_thread_num();
    printf("Hello World from thread = %d\n", tid);
  } 

}

clang -fopenmp -S -O3 -emit-llvm omp_hello.c

cat omp_hello.ll

 1 ; ModuleID = 'omp_hello.c'
  2 source_filename = "omp_hello.c"
  3 target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
  4 target triple = "x86_64-unknown-linux-gnu"
  5 
  6 %struct.ident_t = type { i32, i32, i32, i32, i8* }
  7 
  8 @.str = private unnamed_addr constant [30 x i8] c"Hello World from thread = %d\0A\00", align 1
  9 @.str.1 = private unnamed_addr constant [23 x i8] c";unknown;unknown;0;0;;\00", align 1
 10 @0 = private unnamed_addr global %struct.ident_t { i32 0, i32 2, i32 0, i32 0, i8* getelementptr inbounds ([23 x i8], [23 x i8]* @.str.1, i32 0, i32 0) }, align 8
 11 
 12 ; Function Attrs: nounwind uwtable
 13 define dso_local i32 @main(i32 %argc, i8** nocapture readnone %argv) local_unnamed_addr #0 {
 14 entry:
 15   tail call void (%struct.ident_t*, i32, void (i32*, i32*, ...)*, ...) @__kmpc_fork_call(%struct.ident_t* nonnull @0, i32 0, void (i32*, i32*, ...)* bitcast (void (i32*,     i32*)* @.omp_outlined. to void (i32*, i32*, ...)*)) #4
 16   ret i32 0
 17 }
 18 
 19 ; Function Attrs: norecurse nounwind uwtable
 20 define internal void @.omp_outlined.(i32* noalias nocapture readnone %.global_tid., i32* noalias nocapture readnone %.bound_tid.) #1 {
 21 entry:
 22   %call = tail call i32 @omp_get_thread_num() #4
 23   %call1 = tail call i32 (i8*, ...) @printf(i8* nonnull dereferenceable(1) getelementptr inbounds ([30 x i8], [30 x i8]* @.str, i64 0, i64 0), i32 %call)
 24   ret void
 25 }
 26 
 27 declare dso_local i32 @omp_get_thread_num() local_unnamed_addr #2
 28 
 29 ; Function Attrs: nofree nounwind
 30 declare dso_local i32 @printf(i8* nocapture readonly, ...) local_unnamed_addr #3
 31 
 32 declare !callback !2 dso_local void @__kmpc_fork_call(%struct.ident_t*, i32, void (i32*, i32*, ...)*, ...) local_unnamed_addr

offloading example

#include <stdio.h>
#include <omp.h>

int main(void) {
  int isHost = 0;

#pragma omp target map(from: isHost)
  { isHost = omp_is_initial_device(); }

  if (isHost < 0) {
    printf("Runtime error, isHost=%d\n", isHost);
  }

  // CHECK: Target region executed on the device
  printf("Target region executed on the %s\n", isHost ? "host" : "device");

  return isHost;
}

clang -fopenmp -S -O3 -emit-llvm -fopenmp-targets=nvptx64-nvidia-cuda offloading_success.cpp

relevant source files

List

• tests: clang/test/OpenMP
• Clang CodeGen for OpenMP IR:
  • clang/lib/CodeGen/CGOpenMPRuntime.cpp
  • clang/lib/CodeGen/CGStmtOpenMP.cpp //This contains code to emit OpenMP nodes as LLVM code.
• LLVM OMP IR Builder: llvm/lib/Frontend/OpenMP/OMPIRBuilder.cpp
• Runtime support: openmp/runtime/src/kmp_csupport.cpp

How to Debug

First, use -v to show the actual command line used by the clang driver:

clang -v -fopenmp omp_hello.c

clang version 10.0.1 (https://github.com/llvm/llvm-project d24d5c8e308e689dcd83cbafd2e8bd32aa845a15)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/ubuntu/install/llvm_install/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.5.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/8
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.5.0
Candidate multilib: .;@m64
Selected multilib: .;@m64

/home/ubuntu/install/llvm_install/bin/clang-10 -cc1 -triple x86_64-unknown-linux-gnu -emit-obj -mrelax-all -disable-free -main-file-name omp_hello.c -mrelocation-model static -mthread-model posix -mframe-pointer=all -fmath-errno -fno-rounding-math -masm-verbose -mconstructor-aliases -munwind-tables -target-cpu x86-64 -dwarf-column-info -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/ubuntu/install/llvm_install/lib/clang/10.0.1 -internal-isystem /usr/local/include -internal-isystem /home/ubuntu/install/llvm_install/lib/clang/10.0.1/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdebug-compilation-dir /home/ubuntu -ferror-limit 19 -fmessage-length 0 -fopenmp -fgnuc-version=4.2.1 -fobjc-runtime=gcc -fdiagnostics-show-option -fcolor-diagnostics -faddrsig -o /tmp/omp_hello-c73985.o -x c omp_hello.c


clang -cc1 version 10.0.1 based upon LLVM 10.0.1 default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /home/ubuntu/install/llvm_install/lib/clang/10.0.1/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
 "/usr/bin/ld" -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o a.out /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/7.5.0/crtbegin.o -L/usr/lib/gcc/x86_64-linux-gnu/7.5.0 -L/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../.. -L/home/ubuntu/install/llvm_install/bin/../lib -L/lib -L/usr/lib /tmp/omp_hello-c73985.o -lomp -lgcc --as-needed -lgcc_s --no-as-needed -lpthread -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-linux-gnu/7.5.0/crtend.o /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../x86_64-linux-gnu/crtn.o

Next, use gdb to debug the actual executable with the full command line options

• gdb -args /home/ubuntu/install/llvm_install/bin/clang-10 -cc1 -triple x86_64-unknown-linux-gnu -emit-obj -mrelax-all -disable-free -main-file-name omp_hello.c -mrelocation-model static -mthread-model posix -mframe-pointer=all -fmath-errno -fno-rounding-math -masm-verbose -mconstructor-aliases -munwind-tables -target-cpu x86-64 -dwarf-column-info -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/ubuntu/install/llvm_install/lib/clang/10.0.1 -internal-isystem /usr/local/include -internal-isystem /home/ubuntu/install/llvm_install/lib/clang/10.0.1/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdebug-compilation-dir /home/ubuntu -ferror-limit 19 -fmessage-length 0 -fopenmp -fgnuc-version=4.2.1 -fobjc-runtime=gcc -fdiagnostics-show-option -fcolor-diagnostics -faddrsig -o /tmp/omp_hello-c73985.o -x c omp_hello.c

Warmup run of the compiler within gdb

• (gdb) r

Set a breakpoint at 1829 llvm::FunctionCallee CGOpenMPRuntime::createRuntimeFunction(unsigned Function)

• b CGOpenMPRuntime.cpp:1829
• r # run it

Now can check the backtrace/call stack

Breakpoint 1, clang::CodeGen::CGOpenMPRuntime::createRuntimeFunction (this=0x555565d75fd0, Function=0)
    at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CGOpenMPRuntime.cpp:1829
1829	llvm::FunctionCallee CGOpenMPRuntime::createRuntimeFunction(unsigned Function) {
(gdb) bt
#0  clang::CodeGen::CGOpenMPRuntime::createRuntimeFunction (this=0x555565d75fd0, Function=0) at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CGOpenMPRuntime.cpp:1829
#1  0x000055555af54a62 in clang::CodeGen::CGOpenMPRuntime::<lambda(clang::CodeGen::CodeGenFunction&, clang::CodeGen::PrePostActionTy&)>::operator()(clang::CodeGen::CodeGenFunction &, clang::CodeGen::PrePostActionTy &) const (__closure=0x7fffffff80a0, CGF=...) at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CGOpenMPRuntime.cpp:3109
#2  0x000055555af89801 in clang::CodeGen::RegionCodeGenTy::CallbackFn<clang::CodeGen::CGOpenMPRuntime::emitParallelCall(clang::CodeGen::CodeGenFunction&, clang::SourceLocation, llvm::Function*, llvm::ArrayRef<llvm::Value*>, const clang::Expr*)::<lambda(clang::CodeGen::CodeGenFunction&, clang::CodeGen::PrePostActionTy&)> >(intptr_t, clang::CodeGen::CodeGenFunction &, clang::CodeGen::PrePostActionTy &) (CodeGen=140737488322720, CGF=..., Action=...) at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CGOpenMPRuntime.h:76
#3  0x000055555af45212 in clang::CodeGen::RegionCodeGenTy::operator() (this=0x7fffffff8080, CGF=...)
    at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CGOpenMPRuntime.cpp:787
#4  0x000055555af55072 in clang::CodeGen::CGOpenMPRuntime::emitParallelCall (this=0x555565d75fd0, CGF=..., Loc=..., OutlinedFn=0x555565d0bf78, CapturedVars=..., IfCond=0x0)
    at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CGOpenMPRuntime.cpp:3145
#5  0x000055555aabd108 in emitCommonOMPParallelDirective (CGF=..., S=..., InnermostKind=llvm::omp::Directive::OMPD_parallel, CodeGen=..., CodeGenBoundParameters=...)
    at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CGStmtOpenMP.cpp:1329
#6  0x000055555aabdbbf in clang::CodeGen::CodeGenFunction::EmitOMPParallelDirective (this=0x7fffffff8880, S=...)
    at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CGStmtOpenMP.cpp:1439
#7  0x000055555aa94671 in clang::CodeGen::CodeGenFunction::EmitStmt (this=0x7fffffff8880, S=0x555565e22980, Attrs=...)
    at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CGStmt.cpp:193
#8  0x000055555aa95392 in clang::CodeGen::CodeGenFunction::EmitCompoundStmtWithoutScope (this=0x7fffffff8880, S=..., GetLast=false, AggSlot=...)
    at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CGStmt.cpp:444
#9  0x000055555ab1ff79 in clang::CodeGen::CodeGenFunction::EmitFunctionBody (this=0x7fffffff8880, Body=0x555565e229b8)
    at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CodeGenFunction.cpp:1146
#10 0x000055555ab20be3 in clang::CodeGen::CodeGenFunction::GenerateCode (this=0x7fffffff8880, GD=..., Fn=0x555565d0bd38, FnInfo=...)
    at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CodeGenFunction.cpp:1311
#11 0x000055555ab498d0 in clang::CodeGen::CodeGenModule::EmitGlobalFunctionDefinition (this=0x555565d73210, GD=..., GV=0x555565d0bd38)
    at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CodeGenModule.cpp:4474
#12 0x000055555ab42709 in clang::CodeGen::CodeGenModule::EmitGlobalDefinition (this=0x555565d73210, GD=..., GV=0x0)
    at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CodeGenModule.cpp:2864
#13 0x000055555ab4123f in clang::CodeGen::CodeGenModule::EmitGlobal (this=0x555565d73210, GD=...) at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CodeGenModule.cpp:2567
#14 0x000055555ab4d9a0 in clang::CodeGen::CodeGenModule::EmitTopLevelDecl (this=0x555565d73210, D=0x555565e20f20)
    at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CodeGenModule.cpp:5279
#15 0x000055555baa4537 in (anonymous namespace)::CodeGeneratorImpl::HandleTopLevelDecl (this=0x555565d72870, DG=...)
    at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/ModuleBuilder.cpp:170
#16 0x000055555ba9de7a in clang::BackendConsumer::HandleTopLevelDecl (this=0x555565d72690, D=...) at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CodeGenAction.cpp:214
#17 0x000055555cf95464 in clang::ParseAST (S=..., PrintStats=false, SkipFunctionBodies=false) at /home/ubuntu/source/llvm_src/clang/lib/Parse/ParseAST.cpp:162
#18 0x000055555b1ba873 in clang::ASTFrontendAction::ExecuteAction (this=0x555565d52ac0) at /home/ubuntu/source/llvm_src/clang/lib/Frontend/FrontendAction.cpp:1043
#19 0x000055555ba9bd46 in clang::CodeGenAction::ExecuteAction (this=0x555565d52ac0) at /home/ubuntu/source/llvm_src/clang/lib/CodeGen/CodeGenAction.cpp:1179
#20 0x000055555b1ba1d4 in clang::FrontendAction::Execute (this=0x555565d52ac0) at /home/ubuntu/source/llvm_src/clang/lib/Frontend/FrontendAction.cpp:936
#21 0x000055555b14eb74 in clang::CompilerInstance::ExecuteAction (this=0x555565d4f490, Act=...) at /home/ubuntu/source/llvm_src/clang/lib/Frontend/CompilerInstance.cpp:965
#22 0x000055555b325e07 in clang::ExecuteCompilerInvocation (Clang=0x555565d4f490) at /home/ubuntu/source/llvm_src/clang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:290
#23 0x0000555557d60900 in cc1_main (Argv=..., Argv0=0x7fffffffe2fb "/home/ubuntu/install/llvm_install/bin/clang-10", 
    MainAddr=0x555557d541aa <GetExecutablePath[abi:cxx11](char const*, bool)>) at /home/ubuntu/source/llvm_src/clang/tools/driver/cc1_main.cpp:240
#24 0x0000555557d55a4c in ExecuteCC1Tool (ArgV=...) at /home/ubuntu/source/llvm_src/clang/tools/driver/driver.cpp:329
#25 0x0000555557d561ca in main (argc_=54, argv_=0x7fffffffdef8) at /home/ubuntu/source/llvm_src/clang/tools/driver/driver.cpp:403