Architecture

Pritunl Cloud design architecture

Declarative State Based Designed

Pritunl Cloud uses a state based design similar to React. Every 3 seconds the node pulls the full state of all resources running on the node. This is then compared to the existing state and any changes are applied. This has a significant advantage in reliability to event based designs. No disruptive event will ever result in a loss of the expected configuration state. Even if the main process crashes and remains exited for several hours the virtual machines will continue to run and when the process is started again it will pull the latest state and apply any changes that occurred.

Each time the main process is started the current state is fully rebuilt by scanning the full system configuration. If the main process were stopped and some iptables rules were manually modified in an instances namespace it would detect this change when the main process starts and correct the configuration. The reliability of this is extensively tested in development where the main process is frequently closed and recompiled.

Nodes

Each node runs independent of other nodes. A Pritunl Cloud cluster has no single point of failure assuming a MongoDB replica set is also configured. In the case of scheduled tasks that should only run once a reservation system is used to allow any running node to reserve and run the task. No one node is designated a primary node.

Instance Networking

Instances run in network namespaces to provide full network isolation. Each instance network namespace contains the VPC default gateway, this address is assigned to the VPC bridge in the namespace. This prevents a single point of failure with with having one default gateway for the entire VPC. A VLAN interface is then attached to the bridge and bridged to the host internal interface. Each VPC is assigned a unique VLAN ID in the range of 1001-3999. This allows for about 3000 VPCs in a Pritunl Cloud cluster.

This provides a very similar network layout common on cloud providers. The instance will have one interface with the private VPC IP address and the public IP with accessible through a static NAT.

The primary difference is the utilization of a private VPC IPv6 range instead of directly applying the public IPv6 address to the instance interface. This has the benefit of supporting DHCP IPv6 environments where the public IPv6 changes. If the network provider reassigns the DHCP IPv6 range instances will regain IPv6 connectivity in about 1 minute without the instance seeing any changes to the network interface.

Firewall rules are applied using iptables and ipset inside the namespace.

┌──────────────────────────────┐
│         KVM Instance         │
│      ┌────────────────┐      │
│      │   IMDS Agent   │      │
│      └───────▼────────┘      │
│      ┌────────────────┐      │
│      │ Instance Iface │      │
│      └───────┬────────┘      │
└──────────────┼───────────────┘
               │
               │
┌──────────────┼─────────────────────────────────────┐
│      Network Namespace       ┌───────▲────────┐    │
│              │               │   IMDS Server  │    │
│     ┌────────▼─────────┐     └───────▼────────┘    │
│     │                  │                           │
│     │    VPC Bridge    │─────────Static NAT        │
│     │                  │             │             │
│     └────────┬─────────┘             │             │
│              │                       │             │
│     ┌────────▼─────────┐     ┌───────▲────────┐    │
│     │  Internal Iface  │     │ External Iface │    │
│     └────────┬─────────┘     └───────┬────────┘    │
└──────────────┼───────────────────────┼─────────────┘
               │                       │
               │                       │
      ┌────────▼──────────┐  ┌─────────▲─────────┐
      │Host Internal Iface│  │Host External Iface│
      └───────────────────┘  └───────────────────┘

Node Network Mode

The default configuration for instance networking will use a macvlan. This allows attaching instance network interfaces to a non-bridge interface. This has limitations including require the use of the VXLAN datacenter network mode and the host will be unable to reach the public IP address of instances running on the same host. To avoid these limitations a bridge interface can be configured for both the external and internal traffic on the host. If the same interface is used for external and internal traffic only one bridge will be needed. This should be done with STP off as shown below in the example configuration. After this is configured set the internal and external interfaces in the node settings to the bridge interface.

sudo nmcli general hostname cloud

sudo nmcli -g UUID connection show | while read connid; do sudo nmcli connection delete "$connid"; done

sudo nmcli connection show

sudo nmcli connection add type bridge con-name bridge0 ifname pritunlbr0
sudo nmcli connection modify bridge0 ipv4.method manual ipv4.addresses 123.123.123.123/16 ipv4.gateway 123.123.0.1 ipv4.dns "8.8.8.8,8.8.4.4" ipv6.method dhcp bridge.stp off 802-3-ethernet.mtu 9000
sudo nmcli connection add type bridge-slave ifname enp4s0 master pritunlbr0
sudo nmcli connection modify bridge-slave-enp4s0 802-3-ethernet.mtu 9000
sudo nmcli connection up bridge0

sudo nmcli connection show

Instance Virtual Machine

The virtual machines run in systemd units with sandboxing options. The QEMU process runs with sandboxing options and dropped permissions to a user that is created for each instance.

Instance IMDS Service

Each instance runs the IMDS service in a systemd unit with sandboxing options. The IMDS server binds to IP 169.254.169.254 in the namespace and to a unix socket. The unix socket allows the Pritunl host to communicate with the IMDS service without needing to create a network path. Both host and client requests are authenticated with different secrets. The client secret is stored in /etc/pritunl-imds.json with root read permissions. This will be read by the pci CLI tool used to make IMDS requests.

VPCs

Instance VPCs are always a VLAN, each VPC is given a random VLAN ID in the range 1001-3999. This would require layer 2 networking and unrestricted VLAN access between Pritunl Cloud nodes. To allow the VPC networking to function in layer 3 only environments the VXLAN network mode is used. This is enabled in the datacenter settings. When this is used a VXLAN with ID 9417 is created between all the Pritunl Cloud nodes and each node will have an IP address on that VXLAN. Then the VPC VLANs are routed through the VXLAN. This results in a 54 MTU overhead when using the VXLAN network mode and no additional overhead with the default or VLAN only networking mode.