Skip to main content

Cloudflare Tunnel Setup (Airbyte)

This document provides a complete guide to setting up Airbyte with a Cloudflare Tunnel for secure, public access.

Prerequisites​

  • AWS EC2 instance (Amazon Linux 2023)
  • Domain name (pixlr.to)
  • Cloudflare account with DNS management access
  • Kubernetes cluster with Airbyte installed via abctl

Step 1: Connect to Your EC2 Instance​

ssh -i /path/to/my/key.pem ec2-user@my-ec2-ip

Step 2: Install Cloudflared​

# Download the latest cloudflared binary
wget https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64

# Make it executable
chmod +x cloudflared-linux-amd64

# Move to PATH
sudo mv cloudflared-linux-amd64 /usr/local/bin/cloudflared

# Verify installation
cloudflared --version

Step 3: Authenticate with Cloudflare​

cloudflared tunnel login

Follow the prompts to authenticate with your Cloudflare account. This will save credentials to ~/.cloudflared/cert.pem.

Step 4: Create a New Tunnel​

cloudflared tunnel create airbyte-tunnel

Note the tunnel ID from the output (e.g., f3233a42-1058-47e5-83c7-1df90600cbd0).

Step 5: Configure the Tunnel​

Create a configuration file:

mkdir -p ~/.cloudflared
cat > ~/.cloudflared/config.yml <<EOL
tunnel: YOUR_TUNNEL_ID
credentials-file: /home/ec2-user/.cloudflared/YOUR_TUNNEL_ID.json
ingress:
  - hostname: airbyte.yourdomain.com
    service: http://localhost:8000
  - service: http_status:404
EOL

Replace YOUR_TUNNEL_ID with the actual tunnel ID from Step 4.

Step 6: Create DNS Record​

cloudflared tunnel route dns YOUR_TUNNEL_ID airbyte.yourdomain.com

Step 7: Set Up Port Forwarding​

In a new terminal, set up port forwarding to the Airbyte service:

kubectl port-forward -n airbyte-abctl svc/airbyte-abctl-airbyte-server-svc 8000:8001

Step 8: Run the Tunnel​

In a separate terminal, run:

cloudflared tunnel run

Step 9: Verify Access​

Visit https://airbyte.pixlr.to in your browser. You should see the Airbyte login page.

1. Create Port Forwarding Service​

cat > /etc/systemd/system/airbyte-port-forward.service <<EOL
[Unit]
Description=Airbyte Port Forwarding
After=network.target

[Service]
User=ec2-user
ExecStart=/usr/bin/kubectl port-forward -n airbyte-abctl svc/airbyte-abctl-airbyte-server-svc 8000:8001
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOL

2. Create Cloudflared Service​

cat > /etc/systemd/system/cloudflared.service <<EOL
[Unit]
Description=Cloudflare Tunnel
After=network.target

[Service]
User=ec2-user
ExecStart=/usr/local/bin/cloudflared tunnel run
WorkingDirectory=/home/ec2-user
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOL

3. Enable and Start Services​

sudo systemctl daemon-reload
sudo systemctl enable airbyte-port-forward
sudo systemctl enable cloudflared
sudo systemctl start airbyte-port-forward
sudo systemctl start cloudflared

4. Check Service Status​

sudo systemctl status airbyte-port-forward
sudo systemctl status cloudflared

Troubleshooting​

Check Logs​

journalctl -u airbyte-port-forward -f
journalctl -u cloudflared -f

Verify Port Forwarding​

curl -v http://localhost:8000

Check Tunnel Status​

cloudflared tunnel list
cloudflared tunnel info YOUR_TUNNEL_ID

Security Considerations​

  1. HTTPS: Cloudflare provides automatic HTTPS
  2. Access Control: Set up Cloudflare Access for additional security
  3. Firewall: Ensure your EC2 security group allows outbound traffic
  4. Updates: Regularly update cloudflared: cloudflared update

Maintenance​

Update Airbyte​

abctl upgrade

Update Cloudflared​

sudo cloudflared update

Backup Configuration​

# Backup Kubernetes resources
kubectl get all -n airbyte -o yaml > airbyte-backup-$(date +%Y%m%d).yaml

# Backup Cloudflare tunnel config
cp -r ~/.cloudflared/ ~/cloudflared-backup-$(date +%Y%m%d)

Verify Port Forwarding​

curl -v http://localhost:8000

Check Tunnel Status​

cloudflared tunnel list
cloudflared tunnel info YOUR_TUNNEL_ID

Check Service Status and Logs​

# Check service status
sudo systemctl status cloudflared

# View logs
journalctl -u cloudflared -f

Restart Services After Updates​

sudo systemctl restart airbyte-port-forward
sudo systemctl restart cloudflared