Server

Debugging Pritunl server

Getting Information For Bug Report

Use the commands below to get information needed for debugging an issue. Refer to Repair Database for information on repairing the database.

pritunl version
printf "dh:\n" && sudo df -h
printf "free:\n" && sudo free -l -m
printf "pritunl lsof: " && sudo lsof -p `pgrep -x pritunl` | wc -l
printf "pritunl-web lsof: " && sudo lsof -p `pgrep -x pritunl-web` | wc -l
printf "pritunl-dns lsof: " && sudo lsof -p `pgrep -x pritunl-dns` | wc -l
printf "pritunl limits:\n" && sudo cat /proc/`pgrep -x pritunl`/limits
printf "pritunl-web limits:\n" && sudo cat /proc/`pgrep -x pritunl-web`/limits
printf "pritunl-dns limits:\n" && sudo cat /proc/`pgrep -x pritunl-dns`/limits
printf "limits:\n" && sudo sh -c "ulimit -Hn; ulimit -Sn"
sudo netstat -tulpn | grep pritunl
printf "pritunl-http:\n" && curl -I http://localhost/check
printf "pritunl-internal:\n" && curl -I http://localhost:9756/check
printf "pritunl-https:\n" && curl --insecure https://localhost

sudo strace -p `pgrep -x pritunl`
sudo strace -p `pgrep -x pritunl-web`
sudo strace -p `pgrep -x pritunl-dns`

Server Synchronized Time

Several components of a multi-server Pritunl cluster rely on accurate synchronized time on all the servers. It is important to ensure all servers in a cluster have the same time.

Connection Reset Error

If you are seeing the macOS/Linux error connection reset by peer or the Windows error An existing connection was forcibly closed by the remote host in the client logs, this can indicate the Pritunl web server queue is overloaded. Use the commands below to increase this based on the resources available on the system and the requirements for the deployment. Generally, a server should have a thread for each concurrent connection, as the connection requests can take time with some authentication modes and will quickly overload if a server comes online and all clients attempt to reconnect at the same time.

# Low Capacity (1 CPU, 4GB RAM)
sudo pritunl set app.request_queue_size 200
sudo pritunl set app.request_accepted_queue_size 100
sudo pritunl set app.request_thread_count 20
sudo pritunl set app.request_max_thread_count 100

# Standard Capacity (2 CPU, 4GB RAM)
sudo pritunl set app.request_queue_size 600
sudo pritunl set app.request_accepted_queue_size 300
sudo pritunl set app.request_thread_count 50
sudo pritunl set app.request_max_thread_count 300

# High Capacity (4 CPU, 4GB RAM)
sudo pritunl set app.request_queue_size 1200
sudo pritunl set app.request_accepted_queue_size 600
sudo pritunl set app.request_thread_count 100
sudo pritunl set app.request_max_thread_count 600

# Extreme Capacity (8 CPU, 8GB RAM)
sudo pritunl set app.request_queue_size 4000
sudo pritunl set app.request_accepted_queue_size 2000
sudo pritunl set app.request_thread_count 100
sudo pritunl set app.request_max_thread_count 2000

# Restart Server After Changes
sudo systemctl restart pritunl

Process Inspection

To debug CPU usage of the pritunl process py-spy can be used. Run the command below to install py-spy from PyPI into the Pritunl packaged Python build. If sampling rate error is shown reduce the --rate 50 option. This will not work for the Golang based pritunl-web and pritunl-dns processes.

# Install pySpy from PyPI
sudo /usr/lib/pritunl/usr/bin/pip3 install py-spy

# Find pid for process "/usr/lib/pritunl/usr/bin/python3 /usr/lib/pritunl/usr/bin/pritunl start"
sudo ps ax --format user,pid,%cpu,%mem,tty,stat,start,time,command | grep pritunl

# Get live top view of function CPU usage press 4 to sort by total time spent on function
sudo /usr/lib/pritunl/usr/bin/py-spy top --rate 50 --pid <PID>

# Record CPU usage flamegraph to profile.svg press ctrl+c to stop collection and generate output
sudo /usr/lib/pritunl/usr/bin/py-spy record --rate 50 --output profile.svg --pid <PID>

# Dump list of active threads
sudo /usr/lib/pritunl/usr/bin/py-spy dump --pid <PID>