Compiling

Cylon has C++ core, Java and Python bindings. You can compile these in three steps.

Prerequisites#

Here are the prerequisites for compiling Cylon.

  1. CMake 3.16.5
  2. OpenMPI 4.0.1 or higher
  3. Python 3.6 or higher
  4. C++ 14 or higher

Python Environment#

Because Cylon build Apache Arrow with it, we need to specify a Python environment to the build.

If you're using a virtual environment, make sure to set the virtual environment path. Or you can specify /usr as the path if you're installing in the system path.

Create a virtual environment#

cd /home/<username>/cylon
python3 -m venv ENV
source ENV/bin/activate

Here after we assume your Python ENV path is,

/home/<username>/cylon/ENV
Note: User must install Pyarrow with the Cylon build to use Cylon APIs.
Do not use a prior installed pyarrow in your python environment.
Uninstall it before running the setup.

OpenMPI#

From source#

  • We recommend using OpenMPI 4.0.1

  • Download OpenMPI 4.0.1 from https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.1.tar.gz

  • Extract the archive to a folder named openmpi-4.0.1

  • Also create a directory named build in some location. We will use this to install OpenMPI

  • Set the following environment variables

    BUILD=<path-to-build-directory>
    OMPI_401=<path-to-openmpi-4.0.1-directory>
    PATH=$BUILD/bin:$PATH
    LD_LIBRARY_PATH=$BUILD/lib:$LD_LIBRARY_PATH
    export BUILD OMPI_401 PATH LD_LIBRARY_PATH
  • The instructions to build OpenMPI depend on the platform. Therefore, we highly recommend looking into the $OMPI_401/INSTALL file. Platform specific build files are available in $OMPI_401/contrib/platform directory.

  • In general, please specify --prefix=$BUILD and --enable-mpi-java as arguments to configure script. If Infiniband is available (highly recommended) specify --with-verbs=<path-to-verbs-installation>. Usually, the path to verbs installation is /usr. In summary, the following commands will build OpenMPI for a Linux system.

    cd $OMPI_401
    ./configure --prefix=$BUILD --enable-mpi-java
    make -j 8;make install

Installing with package manager#

sudo apt install libopenmpi-dev

Building C++#

We have provided a build script to make the build process easier. It is found in Cylon source root directory.

Build C++, Python Cylon APIs#

./build.sh -pyenv <path to your environment> -bpath <path to the cmake build directory> [--release | --debug]

Example:

./build.sh -pyenv /home/<username>/cylon/ENV -bpath /home/<username>/cylon/build --cpp --release
Note: The default build mode is debug

Python Support#

Cylon provides Python APIs with Cython.

If you're building for the first time, you can use --all option in build. If you'have already built cpp and want to compile the your changes to the API, do the following,

./build.sh -pyenv /home/<username>/cylon/ENV -bpath /home/<username>/cylon/build --python

Note: You only need to do --python just once after the initial C++ build. If you develop the Cython or Python APIs, use --cython flag instead.

Example#

Before running the code in the base path of the cloned repo run the following command. Or add this to your bashrc.

export LD_LIBRARY_PATH=/home/<username>/cylon/build/arrow/install/lib:/home/<username>/twisterx
/build/lib:$LD_LIBRARY_PATH
  1. Test Python API
python3 python/test/test_pycylon.py