Installation Instructions

Note

These installation instructions are for the main Resiliparse module. For installing FastWARC, see Installing FastWARC.

Pre-built Resiliparse binaries can be installed from PyPi:

$ pip install resiliparse

To install the Resiliparse CLI and its dependencies, install the package with the cli flag (or alternatively the all flag):

$ pip install 'resiliparse[cli]'

Building Resiliparse From Source

You can compile Resiliparse either from the PyPi source package or directly from the Github repository, though in any case, you need to install all required build-time dependencies first. On Ubuntu, this is done as follows:

# Add Lexbor repository
curl -L https://lexbor.com/keys/lexbor_signing.key | sudo apt-key add -
echo "deb https://packages.lexbor.com/ubuntu/ $(lsb_release -sc) liblexbor" | \
    sudo tee /etc/apt/sources.list.d/lexbor.list

# Install build dependencies
sudo apt update
sudo apt install build-essential python3-dev libuchardet-dev liblexbor-dev libre2-dev

To build and install Resiliparse from PyPi, run

pip install --no-binary resiliparse resiliparse

That’s it. If you prefer to build directly from the GitHub repository instead, run:

# Clone repository
git clone https://github.com/chatnoir-eu/chatnoir-resiliparse.git
cd chatnoir-resiliparse

# Optional: Create a fresh venv
python3 -m venv venv && source venv/bin/activate

pip install -e resiliparse

To build the wheels without installing them, run:

pip wheel -e resiliparse

# Or:
pip install build && python -m build --wheel resiliparse