Skip to main content

Border0 Implementation - Complete Documentation

Pixlr Infrastructure Access Solution​

Table of Contents​

  1. Executive Summary
  2. Phase 1: Initial Discovery and Diagnosis
  3. Phase 2: Connector Recovery
  4. Phase 3: Socket Configuration - Bastion Host
  5. Phase 4: SSH Certificate Authentication Setup
  6. Phase 5: Target Server Configuration
  7. Complete Infrastructure Overview
  8. User Guide
  9. Troubleshooting Guide
  10. Security Considerations
  11. Maintenance Procedures
  12. Lessons Learned

Executive Summary​

This document chronicles the complete Border0 implementation journey at Pixlr, including:

  • Recovery of an offline Border0 connector
  • Configuration of SSH certificate-based authentication
  • Setup of multiple secure access sockets
  • Resolution of complex authentication and networking challenges

What is Border0?

Border0 is a zero-trust access platform that provides secure, identity-based access to infrastructure without traditional VPNs. It eliminates the need for:

  • VPN clients and configurations
  • Exposed inbound ports
  • Manual SSH key management
  • Bastion host public access

Why Border0 at Pixlr?

Before Border0:

  • ❌ Required VPN for server access
  • ❌ Manual SSH key distribution
  • ❌ Not ideal bastion host management

After Border0:

  • ✅ Web-based access (no VPN)
  • ✅ Automatic certificate-based authentication
  • ✅ Zero standing privileges
  • ✅ Complete session recording
  • ✅ Instant access provisioning

Implementation Results:

MetricResult
Connector StatusOnline and Stable
Sockets Configured3 (Bastion + 2 Node.js servers)
Authentication MethodSSH Certificate (passwordless)
Session RecordingEnabled
Access Time< 5 seconds
Setup Time per User< 5 minutes

Phase 1: Initial Discovery and Diagnosis​

Problem Statement​

Initial Issue: Border0 connector showing as "Offline" in the portal with the following details:

Connector: pixlr-prod
Status: Offline
Last Seen: October 9, 2025, 07:35:32 GMT+8
Hostname: ip-10-44-10-88
Public IP: 52.2.193.239 (Ashburn, VA)
Private IP: 100.124.0.8

Step 1: Instance Identification​

Challenge: Multiple EC2 instances existed, needed to identify which hosted the connector.

EC2 Infrastructure Inventory:

Total Instances: 32
Region: us-east-1 (N. Virginia)
VPC: Default VPC (10.44.0.0/16)

Key Instances:

Instance IDNameTypePrivate IPPublic IPRole
i-056f0f2e6a49d02f8pixlr-prod-bastion-border0t3.small10.44.10.8834.239.121.154Border0 Connector Host
i-0b405da8be80a8a2fpixlr-stag-nodejs-ubt3a.medium10.44.10.79NoneStaging Node.js Server
i-04907f267ca9683a6pixlr-prod-nodejs-paymentc6a.xlarge10.44.10.61NoneProduction Payment Server
i-0ee43fe8d759a7b4fpixlr-prod-bastion-developert3a.nano-3.233.65.175Legacy Bastion
i-036edef128b9d0306pixlr-prod-bastion-newt4g.medium-100.25.202.135Alternative Bastion

Discovery Process:

# Connected via AWS Systems Manager (SSM)
aws ssm start-session --target i-056f0f2e6a49d02f8

# Verified hostname matched Border0 connector
hostname
# Output: ip-10-44-10-88

# Confirmed private IP
ip addr show ens5
# Output: 10.44.10.88/27

# Retrieved instance metadata
curl -s http://169.254.169.254/latest/meta-data/instance-id
# Output: i-056f0f2e6a49d02f8

Confirmed: pixlr-prod-bastion-border0 (i-056f0f2e6a49d02f8) was the correct instance.

Step 2: Initial System Diagnostics​

Commands Executed:

# Check for Border0 service
sudo systemctl status border0-connector
# Result: Unit border0-connector.service could not be found.

sudo systemctl status border0
# Result: Unit border0.service could not be found.

# Search for any Border0-related services
systemctl list-units --all | grep -i border
# Result: No Border0 services found

# Check for running Border0 processes
ps aux | grep -i border
# Result: No processes found (only grep itself)

# Comprehensive system search for Border0
sudo find / -name "*border0*" -type f 2>/dev/null

Critical Finding:

/usr/local/bin/border0

✅ Border0 binary exists but no service configured to run it.

Step 3: Network and Connectivity Checks​

# Check if WireGuard port is listening (Border0 uses WireGuard)
sudo ss -tulpn | grep 32442
# Result: No service listening on port 32442

# Check all listening ports
sudo netstat -tlnp
# Result: Only SSH (22) and SSM agent ports open

# Check network interfaces
ip link show
# Output: Only ens5 (primary network) and lo (loopback)
# Missing: WireGuard tunnel interface (wg0 or utun)

# Test outbound connectivity to Border0 control plane
curl -I https://portal.border0.com
# Result: Connection successful

# Test DNS resolution
nslookup portal.border0.com
# Result: Resolved correctly

Diagnosis:

  • Border0 binary installed but not running
  • No active WireGuard tunnel
  • Network connectivity to Border0 cloud is functional
  • Issue is configuration/service-related, not network-related

Step 4: Error Log Discovery​

Searched system logs:

# Check syslog for Border0 entries
sudo grep -r "border0" /var/log/

Critical Error Logs Found:

{
"level": "fatal",
"ts": "2025-10-16T10:21:52Z",
"msg": "failed to get connector (v2) configuration",
"config_path": "",
"error": "no Border0 token in credentials chain"
}

{
"level": "error",
"ts": "2025-10-16T10:27:40Z",
"msg": "not running as root: only non-network sockets are supported, sockets will not have local DNS entries"
}

Root Causes Identified:

  1. Missing Authentication Token: Connector cannot authenticate with Border0 control plane
  2. Permission Issues: Connector not running with root privileges
  3. No Service Management: No systemd service to keep connector running

Phase 2: Connector Recovery​

Step 1: Authentication Investigation​

Attempted Login Methods:

Attempt 1: Basic User Login​

sudo /usr/local/bin/border0 login

Process:

  1. Command opened URL in browser
  2. Successfully authenticated via Google SSO
  3. Token supposedly saved

Verification:

sudo /usr/local/bin/border0 whoami

Error:

failed to get authenticated identity details: invalid token, please login

Analysis: Token not persisting between commands.

Attempt 2: Token Storage Investigation​

# Check user's Border0 directory
ls -la ~/.border0/
# Result: ls: cannot access '/home/ssm-user/.border0/': No such file or directory

# Check root's Border0 directory
sudo ls -la /root/.border0/
# Result: ls: cannot access '/root/.border0/': No such file or directory

# Search for any Border0 config files
sudo find /etc -name "*border0*" 2>/dev/null
# Result: No config files found

# Check for environment variables
env | grep -i border
# Result: No Border0 environment variables set

# Check for credential files
find ~ -name "*border0*" -type f 2>/dev/null
# Result: No credential files found

Problem Identified: No persistent token storage configured.

Attempt 3: Root User Authentication​

# Switch to root
sudo -i

# Attempt login as root
border0 login
# Result: Login successful

# Test authentication
border0 whoami
# Result: Still shows invalid token error

Conclusion: Login process works but token storage mechanism is broken.

Step 2: Connector Token Generation​

Understanding Border0 Authentication:

Border0 uses two types of tokens:

  1. User Tokens: For CLI management operations (login, create resources)
  2. Connector Tokens: For connector service authentication (long-lived)

Solution: Generate a connector-specific token.

List Available Connectors​

# First, successfully login and immediately run command
sudo border0 login && border0 connector list

Output:

┌──────────────────────────────────────┬────────────┬──────────────────────────────────┐
│ CONNECTOR ID │ NAME │ DESCRIPTION │
├──────────────────────────────────────┼────────────┼──────────────────────────────────┤
│ 7d5a5d82-d38d-4c0c-badd-842302c16a6b │ pixlr-prod │ Pixlr Staging / Prod Environment │
└──────────────────────────────────────┴────────────┴──────────────────────────────────┘

✅ Connector exists in Border0 organization.

Generate Connector Token​

border0 connector token create --connector-id 7d5a5d82-d38d-4c0c-badd-842302c16a6b

Success Output:

✓ Connector token created successfully

Token: eyJhbGciOiJFZERTQSIsImtpZCI6IjEzN....

⚠️ This token will only be displayed once. Store it securely.

Token Details:

  • Type: Connector Authentication Token
  • Connector ID: 7d5a5d82-d38d-4c0c-badd-842302c16a6b
  • Organization: bade2394-d87d-41ec-8cb6-7e9ea3e4d034
  • Issued: 1760672213 (Unix timestamp)
  • Validity: Long-lived (no expiration)

Step 3: Systemd Service Creation​

Challenge: Create a reliable service to run Border0 connector 24/7.

Attempt 1: Initial Service File (INCORRECT)​

sudo cat > /etc/systemd/system/border0-connector.service << 'EOF'
[Unit]
Description=Border0 Connector
After=network.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/border0 connector run
Restart=always
RestartSec=10
Environment=BORDER0_TOKEN=eyJhbGciOiJFZERTQSIs....

[Install]
WantedBy=multi-user.target
EOF

Attempt to Start:

sudo systemctl daemon-reload
sudo systemctl start border0-connector
sudo systemctl status border0-connector

Error:

● border0-connector.service - Border0 Connector
Loaded: loaded (/etc/systemd/system/border0-connector.service; disabled)
Active: failed (Result: exit-code)
Process: ExitCode=1

Error: unknown command "run" for "connector"

Problem: Border0 CLI uses connector start, not connector run.

Attempt 2: Service File Correction - Using sed​

# Attempt to fix with sed
sudo sed -i 's/connector run/connector start/g' /etc/systemd/system/border0-connector.service

# Verify change
sudo cat /etc/systemd/system/border0-connector.service | grep ExecStart

Result: File still showed old content due to permission/caching issues.

Attempt 3: Manual Edit with nano (SUCCESSFUL)​

# Open in text editor
sudo nano /etc/systemd/system/border0-connector.service

# Manually changed:
# FROM: ExecStart=/usr/local/bin/border0 connector run
# TO: ExecStart=/usr/local/bin/border0 connector start

# Save: Ctrl+O, Enter
# Exit: Ctrl+X

Final Working Service File:

[Unit]
Description=Border0 Connector
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/border0 connector start
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
Environment=BORDER0_TOKEN=eyJhbGciOiJFZERTQSIsImtpZ....

[Install]
WantedBy=multi-user.target

Step 4: Service Activation and Verification​

# Reload systemd to recognize changes
sudo systemctl daemon-reload

# Start the service
sudo systemctl start border0-connector

# Enable auto-start on boot
sudo systemctl enable border0-connector

# Check status
sudo systemctl status border0-connector

SUCCESS! Service Status:

● border0-connector.service - Border0 Connector
Loaded: loaded (/etc/systemd/system/border0-connector.service; enabled; preset: enabled)
Active: active (running) since Fri 2025-10-17 03:41:28 UTC; 8s ago
Main PID: 81944 (border0)
Tasks: 9 (limit: 2213)
Memory: 22.6M (peak: 23.6M)
CPU: 389ms
CGroup: /system.slice/border0-connector.service
└─81944 /usr/local/bin/border0 connector start

✅ Connector is now running!

Step 5: Connection Verification​

# Monitor logs in real-time
sudo journalctl -u border0-connector -f

Success Logs:

{
"level": "info",
"ts": "2025-10-17T03:41:29Z",
"msg": "starting Border0 connector",
"version": "v1.1-1755-g6b19208c",
"commit": "6b19208c"
}

{
"level": "info",
"ts": "2025-10-17T03:41:30Z",
"msg": "connected to relay",
"relay_pub": "BsMAvFScqUuMVfQFKxxDvQByaNNLngsfdHK7EMuDUAQ="
}

{
"level": "info",
"ts": "2025-10-17T03:41:30Z",
"msg": "received STUN response for udp4",
"old_address": "<nil>",
"new_address": "52.2.193.239:62310",
"changed": true
}

{
"level": "info",
"ts": "2025-10-17T03:41:30Z",
"msg": "sending discoverability message",
"discoverable": true,
"addrs": [
{
"from_stun": true,
"ip_address": "52.2.193.239",
"port": 62310
},
{
"iface_name": "ens5",
"iface_cidr": "10.44.10.64/27",
"ip_address": "10.44.10.88",
"port": 32442
}
]
}

{
"level": "info",
"ts": "2025-10-17T03:41:30Z",
"msg": "got network state",
"network_id": "e86a4b3e-965b-466a-a5e6-b42c80e34398",
"ipv4": "100.124.0.8/32",
"ipv6": "fd62:6f72:6465:7230::8/128",
"devices_cidr_v4": "100.124.0.0/16",
"devices_cidr_v6": "fd62:6f72:6465:7230::/64",
"resources_cidr_v4": "100.126.0.0/16",
"resources_cidr_v6": "fd62:6f72:6465:7231::/64"
}

{
"level": "info",
"ts": "2025-10-17T03:41:30Z",
"msg": "re-configured wireguard peers successfully"
}

Key Success Indicators:

  • ✅ Connected to Border0 relay server
  • ✅ STUN discovered public IP and port
  • ✅ Received network configuration
  • ✅ WireGuard tunnel established
  • ✅ Registered as discoverable

Step 6: Network Status Verification​

# Check if Border0 process is running
ps aux | grep border0

Output:

root       81944  1.4  4.7 1375344 92864 ?       Ssl  03:41   0:00 /usr/local/bin/border0 connector start
# Check if WireGuard port is listening
sudo ss -tulpn | grep 32442

Output:

udp   UNCONN 0      0               0.0.0.0:32442      0.0.0.0:*    users:(("border0",pid=81944,fd=17))
udp UNCONN 0 0 [::]:32442 [::]:* users:(("border0",pid=81944,fd=18))

✅ WireGuard listening on default port 32442 (both IPv4 and IPv6)

# Check network interfaces
ip link show

Output now includes:

1: lo: <LOOPBACK,UP,LOWER_UP>
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP>
3: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> ← Border0 WireGuard interface

✅ WireGuard tunnel interface created

Connector Recovery: COMPLETE ✓


Phase 3: Socket Configuration - Bastion Host​

Understanding Border0 Sockets​

What is a Socket?

A socket in Border0 represents a service that users can connect to. Think of it as a "door" through which authenticated users can access internal resources.

Socket Types:

  • SSH: Terminal access to servers
  • Database: MongoDB, PostgreSQL, MySQL connections
  • HTTP/HTTPS: Web applications
  • TCP: Generic TCP services
  • RDP: Windows Remote Desktop

Step 1: Planning the Bastion Socket​

Objective: Allow Border0 users to SSH directly into the bastion host itself.

Why this is needed:

  • Administrative access to the Border0 connector
  • Troubleshooting platform
  • Jump box for accessing other internal servers

Configuration Requirements:

ParameterValueReason
Socket TypeSSHTerminal access needed
Socket Namepixlr-prod-bastionDescriptive identifier
Upstream TypeAWS EC2 Instance ConnectAWS-native auth OR Standard SSH
Port22Standard SSH port
UsernameubuntuDefault user on Ubuntu AMI
Connectorpixlr-prodOur restored connector

Step 2: Initial Socket Creation Attempt​

Method: AWS EC2 Instance Connect

Rationale:

  • AWS-native authentication
  • No need for SSH certificate configuration
  • Leverages IAM permissions

Created Socket:

Socket Type: SSH
Socket Name: pixlr-prod-bastion
Description: Border0 bastion host for Pixlr Dev environment
Upstream Information:
SSH Service Type: AWS EC2 Instance Connect
Hostname: 10.44.10.88:22
Username Source: Defined
Username: ubuntu
EC2 Instance ID: i-056f0f2e6a49d02f8
EC2 Instance Region: us-east-1
Connector: pixlr-prod
Session Recording: Enabled

Socket Created Successfully in Portal:

  • Socket ID: Generated
  • DNS Name: pixlr-prod-bastion-pixlr-group.border0.io
  • Status: Alive
  • IP Address: 100.126.0.100 (Border0 internal network)

Step 3: Connection Test - EC2 Instance Connect​

Attempted Connection via Border0 Portal:

Click "Connect" → Select socket → Wait for connection

Error Encountered:

Error: failed to open a new session for the SSH client: ssh: unexpected packet in response to channel open: <nil>
[SSH session terminated]

Detailed Error from Socket:

ssh_ec2_instance_connect: failed to setup ec2 connect: failed to send ssh public key: 
operation error EC2 Instance Connect: SendSSHPublicKey,
https response error StatusCode: 400,
RequestID: f1a4f1f5-f68f-44d1-9221-12fa9c4ee87e,
api error AccessDeniedException:
User: arn:aws:sts::395785058010:assumed-role/EC2-SSM-Role/i-056f0f2e6a49d02f8
is not authorized to perform: ec2-instance-connect:SendSSHPublicKey
on resource: arn:aws:ec2:us-east-1:395785058010:instance/i-056f0f2e6a49d02f8
because no identity-based policy allows the ec2-instance-connect:SendSSHPublicKey action

Root Cause Analysis:

The bastion instance has an IAM role (EC2-SSM-Role) that allows Systems Manager access, but does NOT have permission to use EC2 Instance Connect.

IAM Role Required Permissions:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "ec2-instance-connect:SendSSHPublicKey",
"Resource": "arn:aws:ec2:us-east-1:395785058010:instance/i-056f0f2e6a49d02f8"
}
]
}

Decision: Rather than modify IAM policies (requires approval), switch to Standard SSH with Border0 Certificate authentication.

Step 4: SSH Certificate Authentication Setup​

What is SSH Certificate Authentication?

Instead of using SSH keys, Border0 acts as a Certificate Authority (CA). It:

  1. Issues short-lived SSH certificates to authenticated users
  2. Target SSH servers trust certificates signed by Border0's CA
  3. No need to distribute or manage SSH keys

Benefits:

  • ✅ No SSH keys to manage
  • ✅ Automatic certificate rotation
  • ✅ Certificate validity tied to user authentication
  • ✅ Centralized access control

Border0 SSH CA Public Key:

Retrieved from Border0 Portal → Organization Settings → Details:

ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ1HJQ3fctrXZ5gmQiVpyUEnlkiXj4pWH3ytGx9YnacbXEa4eFqyhTagxS1c9XtfPBPJM2QP1ZZXUXM9cuEIdIk= Border0 ssh CA for Pixlr Group

Step 5: Configuring the Bastion for Certificate Auth​

Connected to bastion via AWS SSM:

aws ssm start-session --target i-056f0f2e6a49d02f8

Create Border0 CA Certificate File​

Method 1 Attempt (heredoc - FAILED):

sudo bash -c 'cat > /etc/ssh-ca.pub << EOF
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyN.....= Border0 ssh CA for Pixlr Group
EOF'

Error:

bash: line 1: warning: here-document at line 1 delimited by end-of-file (wanted `EOF')
cat: ecdsa-sha2-nistp256: No such file or directory
cat: AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ1HJQ3fctrXZ5gmQiVpyUEnlkiXj4pWH3ytGx9YnacbXEa4eFqyhTagxS1c9XtfPBPJM2QP1ZZXUXM9cuEIdIk=: No such file or directory

Issue: Quote escaping problem with nested bash commands.

Method 2 (tee - SUCCESS):

sudo tee /etc/ssh-ca.pub > /dev/null << 'EOF'
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbml...= Border0 ssh CA for Pixlr Group
EOF

Verification:

cat /etc/ssh-ca.pub

Output:

ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTIt...= Border0 ssh CA for Pixlr Group

✅ CA certificate file created successfully

Configure SSH Daemon​

Check existing configuration:

grep "TrustedUserCAKeys" /etc/ssh/sshd_config
# Output: (empty - not configured)

grep "AuthorizedPrincipalsFile" /etc/ssh/sshd_config
# Output: #AuthorizedPrincipalsFile none

Add Border0 certificate trust:

# Add TrustedUserCAKeys directive
sudo sh -c 'echo "TrustedUserCAKeys /etc/ssh-ca.pub" >> /etc/ssh/sshd_config'

# Add AuthorizedPrincipalsFile directive
sudo sh -c 'echo "AuthorizedPrincipalsFile /etc/ssh/authorized_principals" >> /etc/ssh/sshd_config'

# Create authorized principals file
sudo sh -c 'echo "border0_ssh_signed" > /etc/ssh/authorized_principals'

Verify configuration:

sudo tail -5 /etc/ssh/sshd_config

Output:

#       AllowTcpForwarding no
# PermitTTY no
# ForceCommand cvs server
TrustedUserCAKeys /etc/ssh-ca.pub
AuthorizedPrincipalsFile /etc/ssh/authorized_principals
cat /etc/ssh/authorized_principals

Output:

border0_ssh_signed

✅ SSH configuration updated

Test SSH Configuration​

# Test configuration for syntax errors
sudo sshd -t

Warning (non-critical):

Missing privilege separation directory: /run/sshd

Note: This warning is normal and can be ignored. The directory is created when SSH starts.

Start/Restart SSH Service​

Check SSH service status:

systemctl list-unit-files | grep ssh

Output:

ssh.service                disabled        enabled
ssh.socket enabled enabled

Start SSH service:

sudo systemctl start ssh
sudo systemctl enable ssh

Verify SSH is running:

sudo systemctl status ssh

Output:

● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/usr/lib/systemd/system/ssh.service; enabled; preset: enabled)
Active: active (running) since Fri 2025-10-17 05:08:17 UTC; 5s ago
Main PID: 84373 (sshd)
Tasks: 1 (limit: 2213)
Memory: 1.2M (peak: 1.4M)
CPU: 20ms
CGroup: /system.slice/ssh.service
└─84373 "sshd: /usr/sbin/sshd -D"

Oct 17 05:08:17 ip-10-44-10-88 systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
Oct 17 05:08:17 ip-10-44-10-88 sshd[84373]: Server listening on 0.0.0.0 port 22.
Oct 17 05:08:17 ip-10-44-10-88 sshd[84373]: Server listening on :: port 22.
Oct 17 05:08:17 ip-10-44-10-88 systemd[1]: Started ssh.service - OpenBSD Secure Shell server.

✅ SSH service running and listening

# Verify SSH is listening on port 22
sudo ss -tlnp | grep :22

Output:

LISTEN 0      4096         0.0.0.0:22        0.0.0.0:*    users:(("sshd",pid=84373,fd=3),("systemd",pid=1,fd=114))
LISTEN 0 4096 [::]:22 [::]:* users:(("sshd",pid=84373,fd=4),("systemd",pid=1,fd=115))

✅ SSH listening on all interfaces (IPv4 and IPv6)

Step 6: Update Border0 Socket to Use Certificate Auth​

Changed Socket Configuration:

FROM:
Upstream Connection Type: AWS EC2 Instance Connect
EC2 Instance ID: i-056f0f2e6a49d02f8
EC2 Instance Region: us-east-1

TO:
Upstream Connection Type: Standard SSH
Authentication Strategy: Border0 Certificate
Hostname: 10.44.10.88
Port: 22
Username: ubuntu
Connector: pixlr-prod

Socket Updated Successfully

Step 7: Connection Test - SUCCESS!​

Attempted Connection:

Border0 Portal → Sockets → pixlr-prod-bastion → Connect

Result:

Welcome to Ubuntu 24.04.3 LTS (GNU/Linux 6.14.0-1011-aws x86_64)

* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro

System information as of Fri Oct 17 06:23:57 UTC 2025

System load: 0.0 Temperature: -273.1 C
Usage of /: 40.1% of 6.71GB Processes: 115
Memory usage: 23% Users logged in: 0
Swap usage: 0% IPv4 address for ens5: 10.44.10.88

Last login: Fri Oct 17 05:15:50 2025 from 10.44.10.88
ubuntu@ip-10-44-10-88:~$

✅ BASTION ACCESS WORKING!

Verification in auth.log:

sudo tail -f /var/log/auth.log

Output:

Oct 17 06:23:57 ip-10-44-10-88 sshd[84521]: Accepted publickey for ubuntu from 100.126.0.100 port 53847 ssh2: ECDSA-CERT SHA256:...
Oct 17 06:23:57 ip-10-44-10-88 sshd[84521]: pam_unix(sshd:session): session opened for user ubuntu(uid=1000)

Key observations:

  • ✅ Authentication method: publickey (certificate)
  • ✅ Source: 100.126.0.100 (Border0 network)
  • ✅ Certificate type: ECDSA-CERT
  • ✅ Session opened successfully

Bastion Socket Configuration: COMPLETE ✓


Phase 4: SSH Certificate Authentication Setup​

Now that the bastion works, we need to configure the same certificate authentication on target servers.

Target Servers​

From the Jira ticket requirements:

  1. Staging Node.js Server

    • Instance ID: i-0b405da8be80a8a2f
    • Name: pixlr-stag-nodejs-ub
    • Private IP: 10.44.10.79
    • Purpose: Staging application server
  2. Production Node.js Payment Server

    • Instance ID: i-04907f267ca9683a6
    • Name: pixlr-prod-nodejs-payment
    • Private IP: 10.44.10.61
    • Purpose: Production payment processing server

Challenge: No Direct Access​

Problem: These servers are in private subnets with:

  • ❌ No public IP addresses
  • ❌ No EC2 Instance Connect available (not in public subnet)
  • ❌ No SSM agent connected
  • ❌ No SSH keys on the bastion to access them

Error when trying EC2 Instance Connect:

No public IPv4 or IPv6 address assigned
Instance is not in public subnet
Associated subnet subnet-906b62b8 (apps-internal-nat) is not a public subnet.

Error when trying SSM:

An error occurred (TargetNotConnected) when calling the StartSession operation: 
i-0b405da8be80a8a2f is not connected.

Solution: Temporary EC2 Instance Connect from AWS CLI​

AWS EC2 Instance Connect can push a temporary SSH key (valid for 60 seconds) to an instance, even if it's in a private subnet, as long as the AWS API can reach it.

Generate SSH Key on Bastion​

# On pixlr-prod-bastion-border0 (already connected via Border0)
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ""

Output:

Generating public/private ed25519 key pair.
Your identification has been saved in /home/ubuntu/.ssh/id_ed25519
Your public key has been saved in /home/ubuntu/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:vnQhvnUShdGe9MZ...../DcDPFQ ubuntu@ip-10-44-10-88
cat ~/.ssh/id_ed25519.pub

Output:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAA... ubuntu@ip-10-44-10-88

Key generated: This will be used to initially access the target servers.


Phase 5: Target Server Configuration​

Configuring Staging Server (pixlr-stag-nodejs-ub)​

Step 1: Push Temporary SSH Key​

From local machine with AWS CLI:

aws ec2-instance-connect send-ssh-public-key \
--region us-east-1 \
--instance-id i-0b405da8be80a8a2f \
--instance-os-user ubuntu \
--ssh-public-key "ssh-ed25519 AAAAC3NzaC..."

Response:

{
"RequestId": "305a6785-bd0d-419b-9893-2469add25c42",
"Success": true
}

✅ Temporary key pushed successfully (valid for 60 seconds)

Step 2: SSH from Bastion (Within 60 seconds!)​

From bastion terminal:

ssh ubuntu@10.44.10.79

Success:

Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.15.0-1073-aws x86_64)

ubuntu@ip-10-44-10-79:~$

✅ Connected to staging server

Step 3: Configure Border0 Certificate Authentication​

Run configuration commands:

# Create Border0 CA certificate file
sudo tee /etc/ssh-ca.pub > /dev/null << 'EOF'
ecdsa-sha2-nistp256 AAAAE2VjZHNhLX....= Border0 ssh CA for Pixlr Group
EOF

# Add SSH configuration
sudo sh -c 'echo "TrustedUserCAKeys /etc/ssh-ca.pub" >> /etc/ssh/sshd_config'
sudo sh -c 'echo "AuthorizedPrincipalsFile /etc/ssh/authorized_principals" >> /etc/ssh/sshd_config'
sudo sh -c 'echo "border0_ssh_signed" > /etc/ssh/authorized_principals'

# Start and reload SSH
sudo systemctl start ssh
sudo systemctl reload ssh

Verification:

# Verify CA file
cat /etc/ssh-ca.pub
# Output: [Border0 CA certificate]

# Verify SSH config
sudo tail -3 /etc/ssh/sshd_config
# Output:
# TrustedUserCAKeys /etc/ssh-ca.pub
# AuthorizedPrincipalsFile /etc/ssh/authorized_principals

# Verify principals
cat /etc/ssh/authorized_principals
# Output: border0_ssh_signed

# Verify SSH status
sudo systemctl status ssh
# Output: active (running)

✅ Staging server configured

# Exit back to bastion
exit

Configuring Production Server (pixlr-prod-nodejs-payment)​

Step 1: Push Temporary SSH Key​

From local machine:

aws ec2-instance-connect send-ssh-public-key \
--region us-east-1 \
--instance-id i-04907f267ca9683a6 \
--instance-os-user ubuntu \
--ssh-public-key "ssh-ed25519 AAAAC3NzaC1lZDI..."

Response:

{
"RequestId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"Success": true
}

Step 2: SSH from Bastion​

ssh ubuntu@10.44.10.61

Success:

Welcome to Ubuntu 22.04.5 LTS (GNU/Linux 6.5.0-1025-aws x86_64)

ubuntu@ip-10-44-10-61:~$

✅ Connected to production payment server

Step 3: Configure Border0 Certificate Authentication​

# Create Border0 CA certificate file
sudo tee /etc/ssh-ca.pub > /dev/null << 'EOF'
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbm....= Border0 ssh CA for Pixlr Group
EOF

# Add SSH configuration
sudo sh -c 'echo "TrustedUserCAKeys /etc/ssh-ca.pub" >> /etc/ssh/sshd_config'
sudo sh -c 'echo "AuthorizedPrincipalsFile /etc/ssh/authorized_principals" >> /etc/ssh/sshd_config'
sudo sh -c 'echo "border0_ssh_signed" > /etc/ssh/authorized_principals'

# Start and reload SSH
sudo systemctl start ssh
sudo systemctl reload ssh

Verification:

cat /etc/ssh-ca.pub
sudo tail -3 /etc/ssh/sshd_config
cat /etc/ssh/authorized_principals
sudo systemctl status ssh

✅ Production server configured

exit

Creating Border0 Sockets for Target Servers​

Socket 1: Staging Node.js Server​

Border0 Portal → Sockets → Add New Socket

Configuration:

Socket Type: SSH
Socket Name: pixlr-stag-nodejs-ub
Description: Pixlr Staging Node.js Server

Upstream Configuration:
Connection Type: Standard SSH
Authentication Strategy: Border0 Certificate
Hostname: 10.44.10.79
Port: 22
SSH Username: ubuntu
Prompt Clients for User Name: Enter Manually

Connectors: pixlr-prod

Session Recording: Enabled

Socket Created:

  • DNS Name: pixlr-stag-nodejs-ub-pixlr-group.border0.io
  • IP Address: 100.126.0.79
  • Status: Alive

Socket 2: Production Node.js Payment Server​

Configuration:

Socket Type: SSH
Socket Name: pixlr-prod-nodejs-payment
Description: Pixlr Production Node.js Payment Server

Upstream Configuration:
Connection Type: Standard SSH
Authentication Strategy: Border0 Certificate
Hostname: 10.44.10.61
Port: 22
SSH Username: ubuntu
Prompt Clients for User Name: Enter Manually

Connectors: pixlr-prod

Session Recording: Enabled

Socket Created:

  • DNS Name: pixlr-prod-nodejs-payment-pixlr-group.border0.io
  • IP Address: 100.126.0.61
  • Status: Alive

Connection Testing​

Test Staging Server​

Border0 Portal → Sockets → pixlr-stag-nodejs-ub → Connect

Success:

Welcome to Ubuntu 20.04.6 LTS
ubuntu@ip-10-44-10-79:~$

✅ Staging server accessible via Border0

Test Production Server​

Border0 Portal → Sockets → pixlr-prod-nodejs-payment → Connect

Success:

Welcome to Ubuntu 22.04.5 LTS
ubuntu@ip-10-44-10-61:~$

✅ Production server accessible via Border0

All Sockets Operational: COMPLETE ✓


Complete Infrastructure Overview​

Architecture Diagram​

┌─────────────────────────────────────────────────────────────────────────┐
│ Users (Web Browser / CLI) │
└────────────────────────────────┬────────────────────────────────────────┘
│
│ HTTPS
│
┌────────────────────────────────▼────────────────────────────────────────┐
│ Border0 Control Plane (Cloud) │
│ portal.border0.com │
│ │
│ - User Authentication (Google SSO) │
│ - Policy Management │
│ - Session Recording │
│ - Audit Logging │
└────────────────────────────────┬────────────────────────────────────────┘
│
│ WireGuard (UDP 32442)
│
┌────────────────────────────────▼────────────────────────────────────────┐
│ Border0 Connector (pixlr-prod) │
│ EC2: i-056f0f2e6a49d02f8 (t3.small) │
│ pixlr-prod-bastion-border0 (10.44.10.88) │
│ │
│ - WireGuard Tunnel: 100.124.0.8/32 │
│ - Service IP Range: 100.126.0.0/16 │
│ - Status: Online │
│ - Version: v1.1-1755-g6b19208c │
└───────────────┬─────────────────┬──────────────────┬─────────────────────┘
│ │ │
│ SSH (22) │ SSH (22) │ SSH (22)
│ │ │
┌───────────────▼──────┐ ┌────────▼──────┐ ┌────────▼──────────────────┐
│ Socket: Bastion │ │ Socket: Stag │ │ Socket: Prod Payment │
│ 10.44.10.88:22 │ │ 10.44.10.79 │ │ 10.44.10.61 │
│ Border0 IP: │ │ Border0 IP: │ │ Border0 IP: │
│ 100.126.0.100 │ │ 100.126.0.79 │ │ 100.126.0.61 │
│ │ │ │ │ │
│ Self (bastion) │ │ Node.js Stag │ │ Node.js Prod Payment │
└──────────────────────┘ └───────────────┘ └───────────────────────────┘

Network Details​

AWS VPC:

  • VPC ID: Default VPC
  • CIDR: 10.44.0.0/16
  • Region: us-east-1 (N. Virginia)
  • Subnets: Multiple across AZs

Border0 Network:

  • Device Network: 100.124.0.0/16
  • Socket Network: 100.126.0.0/16
  • IPv6: fd62:6f72:6465:72xx::/64

Socket Summary​

Socket NameTypeUpstream TargetBorder0 IPStatusDNS Name
pixlr-prod-bastionSSH10.44.10.88:22100.126.0.100Activepixlr-prod-bastion-pixlr-group.border0.io
pixlr-stag-nodejs-ubSSH10.44.10.79:22100.126.0.79Activepixlr-stag-nodejs-ub-pixlr-group.border0.io
pixlr-prod-nodejs-paymentSSH10.44.10.61:22100.126.0.61Activepixlr-prod-nodejs-payment-pixlr-group.border0.io

Security Configuration​

SSH Certificate Authentication:

  • CA Type: ECDSA-SHA2-NISTP256
  • Principal: border0_ssh_signed
  • Certificate Lifetime: Session-based (short-lived)
  • Revocation: Automatic on policy change

Access Control:

  • Authentication: Google SSO
  • Authorization: Border0 policies
  • Session Recording: Enabled
  • Audit Logging: Complete

User Guide​

For End Users​

Prerequisites​

  1. Email Address: Must be added to Border0 policy
  2. Google Account: For SSO authentication
  3. Web Browser: Any modern browser (Chrome, Firefox, Safari, Edge)

Accessing Servers (Web Portal)​

Step 1: Login

  1. Go to https://portal.border0.com
  2. Click "Sign In"
  3. Select "Sign in with Google"
  4. Use your @pixlr.com email

Step 2: View Available Resources

After login, you'll see a list of servers:

  • pixlr-prod-bastion - Border0 bastion host
  • pixlr-stag-nodejs-ub - Staging application server
  • pixlr-prod-nodejs-payment - Production payment server

Step 3: Connect to a Server

  1. Click on the server name (e.g., pixlr-stag-nodejs-ub)
  2. Click the "Connect" button
  3. Wait 2-5 seconds for connection
  4. A terminal window opens in your browser
  5. You're now connected via SSH!

Example Session:

Connecting to pixlr-stag-nodejs-ub...
Connected successfully!

Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.15.0-1073-aws x86_64)

ubuntu@ip-10-44-10-79:~$ whoami
ubuntu

ubuntu@ip-10-44-10-79:~$ pwd
/home/ubuntu

ubuntu@ip-10-44-10-79:~$ # You can now run any commands

Step 4: Disconnect

  • Close the browser tab, or
  • Type exit and press Enter

Your session is automatically recorded and logged.

Using Border0 CLI (Optional - For Advanced Users)​

Installation:

# macOS
brew install borderzero/tap/border0

# Linux
curl -fsSL https://download.border0.com/linux_install.sh | bash

# Windows
# Download from https://download.border0.com/windows_amd64/border0.exe

Login:

border0 login

Opens browser for authentication.

List Available Sockets:

border0 socket ls

Output:

┌────────────────────────────┬──────────────┬────────────┐
│ NAME │ TYPE │ STATUS │
├────────────────────────────┼──────────────┼────────────┤
│ pixlr-prod-bastion │ SSH │ online │
│ pixlr-stag-nodejs-ub │ SSH │ online │
│ pixlr-prod-nodejs-payment │ SSH │ online │
└────────────────────────────┴──────────────┴────────────┘

Connect via CLI:

# Connect to staging
border0 client ssh pixlr-stag-nodejs-ub

# Connect to production
border0 client ssh pixlr-prod-nodejs-payment

# Connect to bastion
border0 client ssh pixlr-prod-bastion

SSH with Agent Forwarding:

border0 client ssh pixlr-stag-nodejs-ub -A

For Administrators​

Adding New Users​

Step 1: Access Border0 Portal

  1. Login to https://portal.border0.com
  2. Navigate to Policies section
  3. Click on default-org-policy (or relevant policy)

Step 2: Add User Email

  1. In the "WHO" section, click Edit
  2. Add user's email address (e.g., newuser@pixlr.com)
  3. Click Save

Result: User can immediately access allowed resources.

Removing Users​

Step 1: Access Policy

  1. Navigate to Policies
  2. Select the policy containing the user

Step 2: Remove User

  1. In the "WHO" section, click Edit
  2. Remove user's email address
  3. Click Save

Result: User loses access immediately. Active sessions are terminated.

Creating New Sockets​

Prerequisites:

  • Target server must be reachable from connector
  • SSH service running on target
  • Border0 CA certificate configured on target (for certificate auth)

Step 1: Configure Target Server

SSH to target server and run:

sudo tee /etc/ssh-ca.pub > /dev/null << 'EOF'
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbm.....= Border0 ssh CA for Pixlr Group
EOF

sudo sh -c 'echo "TrustedUserCAKeys /etc/ssh-ca.pub" >> /etc/ssh/sshd_config'
sudo sh -c 'echo "AuthorizedPrincipalsFile /etc/ssh/authorized_principals" >> /etc/ssh/sshd_config'
sudo sh -c 'echo "border0_ssh_signed" > /etc/ssh/authorized_principals'

sudo systemctl reload ssh

Step 2: Create Socket in Border0 Portal

  1. Navigate to Sockets
  2. Click Add New Socket
  3. Fill in configuration:
    • Socket Type: SSH
    • Socket Name: descriptive-name
    • Description: Purpose of the server
    • Connection Type: Standard SSH
    • Authentication: Border0 Certificate
    • Hostname: Private IP of target
    • Port: 22
    • Username: ubuntu (or appropriate)
    • Connector: pixlr-prod
  4. Click Create

Step 3: Test Connection

  1. Click on the newly created socket
  2. Click Connect
  3. Verify you can access the server

Monitoring and Troubleshooting​

Check Connector Status:

# SSH to bastion
border0 client ssh pixlr-prod-bastion

# Check service status
sudo systemctl status border0-connector

# View real-time logs
sudo journalctl -u border0-connector -f

# Check last 50 lines
sudo journalctl -u border0-connector -n 50 --no-pager

Check Network Connectivity:

# Verify WireGuard is listening
sudo ss -tulpn | grep 32442

# Check Border0 process
ps aux | grep border0

# Test connectivity to Border0 control plane
curl -I https://portal.border0.com

View Socket Status:

In Border0 Portal:

  1. Navigate to Sockets
  2. Check "Status" column - should show green "Alive"
  3. Click on socket to see details
  4. Check "Connectors" section - should show "pixlr-prod" online

View Session History:

  1. Navigate to Sessions
  2. Filter by user, socket, or date range
  3. Click on a session to view recording
  4. Review audit logs for compliance

Troubleshooting Guide​

Common Issues and Solutions​

Issue 1: "Socket shows as Offline"​

Symptoms:

  • Socket status shows red "Offline" indicator
  • Cannot connect to the socket
  • Connection timeout

Possible Causes & Solutions:

Cause 1: Connector is down

Check connector status:

sudo systemctl status border0-connector

If stopped, restart:

sudo systemctl restart border0-connector

Cause 2: Target server unreachable

Test connectivity from bastion:

# Ping target
ping -c 3 10.44.10.79

# Test SSH port
nc -zv 10.44.10.79 22

# Try SSH manually
ssh ubuntu@10.44.10.79

Cause 3: Security group blocking traffic

Verify security group allows SSH from bastion:

  • Go to AWS Console → EC2 → Security Groups
  • Find target server's security group
  • Ensure inbound rule: TCP 22 from bastion's security group

Cause 4: SSH service not running on target

SSH to target and check:

sudo systemctl status ssh
sudo systemctl start ssh

Issue 2: "Permission denied (publickey)"​

Symptoms:

  • Connection fails with authentication error
  • Works for some users but not others

Possible Causes & Solutions:

Cause 1: Border0 certificate not configured

Verify on target server:

# Check if CA file exists
cat /etc/ssh-ca.pub

# Check SSH config
grep "TrustedUserCAKeys" /etc/ssh/sshd_config
grep "AuthorizedPrincipalsFile" /etc/ssh/sshd_config

# Check principals file
cat /etc/ssh/authorized_principals

If missing, reconfigure:

sudo tee /etc/ssh-ca.pub > /dev/null << 'EOF'
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTIt...= Border0 ssh CA for Pixlr Group
EOF

sudo sh -c 'echo "TrustedUserCAKeys /etc/ssh-ca.pub" >> /etc/ssh/sshd_config'
sudo sh -c 'echo "AuthorizedPrincipalsFile /etc/ssh/authorized_principals" >> /etc/ssh/sshd_config'
sudo sh -c 'echo "border0_ssh_signed" > /etc/ssh/authorized_principals'

sudo systemctl reload ssh

Cause 2: User not in Border0 policy

Check Border0 Portal:

  1. Navigate to Policies
  2. Verify user's email is in the "WHO" section
  3. Add if missing

Cause 3: SSH daemon configuration error

Test SSH configuration:

sudo sshd -t

View detailed logs:

sudo tail -f /var/log/auth.log

While tailing logs, attempt connection to see specific error.

Issue 3: "Connector Token Invalid"​

Symptoms:

  • Connector fails to start
  • Logs show: "invalid token" or "unauthorized"

Solution:

Generate new connector token:

# Login to Border0
border0 login

# List connectors
border0 connector list

# Generate new token
border0 connector token create --connector-id 7d5a5d82-d38d-4c0c-badd-842302c16a6b

Update service file:

sudo nano /etc/systemd/system/border0-connector.service

# Replace the BORDER0_TOKEN environment variable value
# Save and exit

sudo systemctl daemon-reload
sudo systemctl restart border0-connector

Issue 4: "Connection timeout / Can't reach socket"​

Symptoms:

  • Connection hangs
  • Browser shows loading spinner indefinitely
  • CLI shows "connection timeout"

Diagnosis:

Step 1: Check Border0 control plane status

curl -I https://portal.border0.com

Step 2: Check connector connectivity

sudo journalctl -u border0-connector | grep -i "connected to relay"

Should see recent "connected to relay" messages.

Step 3: Check WireGuard tunnel

ip link show | grep wg
sudo wg show # If wireguard-tools installed

Step 4: Restart connector

sudo systemctl restart border0-connector
sleep 10
sudo systemctl status border0-connector

Issue 5: "SSH service won't start on target"​

Symptoms:

  • systemctl status ssh shows failed
  • SSH connections fail

Solution:

Check SSH configuration:

sudo sshd -t

Common errors:

Error: "Missing privilege separation directory"

sudo mkdir -p /run/sshd
sudo chmod 0755 /run/sshd

Error: "Bad configuration option"

# Review recent changes
sudo tail -20 /etc/ssh/sshd_config

# Remove or fix problematic lines
sudo nano /etc/ssh/sshd_config

Error: "Permission denied" on config files

sudo chmod 644 /etc/ssh/sshd_config
sudo chmod 644 /etc/ssh-ca.pub
sudo chmod 644 /etc/ssh/authorized_principals

Restart SSH:

sudo systemctl restart ssh

Issue 6: "Session recording not working"​

Symptoms:

  • Sessions listed but no recording available
  • "Recording unavailable" message

Solution:

  1. Check socket configuration:

    • Go to Border0 Portal → Sockets → [Your Socket]
    • Verify "Session Recording: Enabled"
  2. If disabled, enable it:

    • Click "Edit"
    • Enable "Session Recording"
    • Save changes

Note: Only sessions after enabling will be recorded.

Debug Commands Reference​

Connector Diagnostics:

# Service status
sudo systemctl status border0-connector

# Recent logs
sudo journalctl -u border0-connector -n 100 --no-pager

# Follow logs live
sudo journalctl -u border0-connector -f

# Check process
ps aux | grep border0

# Check listening ports
sudo ss -tulpn | grep border0

Network Diagnostics:

# Check WireGuard tunnel
ip addr show | grep -A 5 wg

# Test target connectivity
nc -zv <target-ip> 22

# Test DNS resolution
nslookg portal.border0.com

# Check routing
ip route show

SSH Diagnostics (on target):

# SSH configuration test
sudo sshd -t

# Check SSH is running
sudo systemctl status ssh

# Check SSH is listening
sudo ss -tlnp | grep :22

# View SSH logs
sudo tail -f /var/log/auth.log

# Verbose SSH connection test
ssh -vvv ubuntu@localhost

Border0 CLI Diagnostics:

# Check authentication
border0 whoami

# List connectors
border0 connector list

# List sockets
border0 socket ls

# Get socket details
border0 socket show <socket-name>

Security Considerations​

Current Security Posture​

Strengths ✅​

  1. Zero Trust Architecture

    • Every connection authenticated individually
    • No persistent VPN tunnels
    • Identity-based access control
  2. Certificate-Based Authentication

    • No static SSH keys to manage
    • Short-lived certificates
    • Automatic rotation
    • Centralized revocation
  3. Complete Audit Trail

    • All sessions recorded
    • Full audit logs
    • User identity tracking
    • Compliance-ready
  4. Network Segmentation

    • Bastion in DMZ-like configuration
    • Target servers in private subnets
    • No public IPs on application servers
  5. Encrypted Tunnels

    • WireGuard encryption
    • Modern cryptography
    • Perfect forward secrecy

Potential Improvements 🔄​

High Priority:

  1. Enable AWS Systems Manager Session Manager

    • Eliminates need for public IP on bastion
    • Additional layer of AWS-native security
    • Cost: Free (included with AWS)

    Implementation:

    # Attach IAM role with AmazonSSMManagedInstanceCore
    # Install SSM agent (usually pre-installed on Ubuntu AMIs)
    sudo snap install amazon-ssm-agent --classic
    sudo snap start amazon-ssm-agent
  2. Implement IP Allowlisting for Border0 Portal Access

    • Restrict portal access to office/VPN IPs
    • Reduces attack surface

    Configuration: Border0 Portal → Policies → Add IP restrictions

  3. Enable Multi-Factor Authentication (MFA)

    • Require MFA for all Border0 logins
    • Google Authenticator or hardware keys

    Configuration: Border0 Portal → Organization Settings → Security

Medium Priority:

  1. Implement Just-In-Time (JIT) Access

    • Time-limited access grants
    • Approval workflows for production
    • Automatic access expiration

    Configuration: Use Border0 advanced policies

  2. Enhanced Monitoring

    • Set up CloudWatch alarms for connector failures
    • Alert on unusual session patterns
    • Integration with SIEM system

    Implementation:

    # CloudWatch alarm for connector offline
    aws cloudwatch put-metric-alarm --alarm-name border0-connector-down \
    --alarm-description "Alert when Border0 connector is offline" \
    --metric-name StatusCheckFailed \
    --namespace AWS/EC2 \
    --statistic Average \
    --period 300 \
    --threshold 1 \
    --comparison-operator GreaterThanThreshold \
    --dimensions Name=InstanceId,Value=i-056f0f2e6a49d02f8
  3. Regular Security Audits

    • Monthly access review
    • Quarterly configuration audit
    • Annual penetration testing

Low Priority:

  1. Private VPC Endpoints for Border0

    • Direct connection to Border0 via AWS PrivateLink
    • Avoids internet routing
    • Cost: ~$7.20/month per endpoint

    Note: Check if Border0 supports AWS PrivateLink

  2. VPC Flow Logs

    • Monitor all network traffic
    • Cost: ~$5-10/month

    Implementation:

    aws ec2 create-flow-logs \
    --resource-type VPC \
    --resource-ids vpc-<your-vpc-id> \
    --traffic-type ALL \
    --log-destination-type cloud-watch-logs \
    --log-group-name /aws/vpc/flowlogs

Incident Response​

Suspected Compromise:

  1. Immediate Actions:

    # Disable affected user in Border0
    Border0 Portal → Policies → Remove user email

    # Generate new connector token
    border0 connector token create --connector-id 7d5a5d82-d38d-4c0c-badd-842302c16a6b

    # Update connector
    sudo nano /etc/systemd/system/border0-connector.service
    sudo systemctl daemon-reload
    sudo systemctl restart border0-connector
  2. Investigation:

    • Review Border0 session logs
    • Check AWS CloudTrail logs
    • Review system logs on affected servers
    • Identify scope of compromise
  3. Recovery:

    • Rotate all credentials
    • Patch vulnerabilities
    • Restore from clean backup if necessary
    • Update security policies

Connection Issues:

  1. Check connector status
  2. Review recent changes
  3. Test network connectivity
  4. Verify configurations
  5. Check Border0 service status page

Compliance and Auditing​

Session Recording​

What is recorded:

  • All terminal input/output
  • Connection metadata (user, time, duration)
  • Source IP addresses
  • Commands executed

Retention:

  • Default: 90 days (configurable)
  • Can be exported to S3 for long-term storage

Access:

  • Border0 Portal → Sessions
  • Filter by user, socket, date
  • Playback capability

Audit Logs​

Available logs:

  • User authentication events
  • Socket connection attempts
  • Policy changes
  • Configuration modifications
  • Connector status changes

Export options:

  • CSV download
  • API access
  • Webhook integration
  • SIEM forwarding

Compliance Frameworks​

Border0 supports compliance with:

  • SOC 2 Type II
  • ISO 27001
  • GDPR
  • HIPAA (with proper configuration)
  • PCI DSS

Documentation for auditors:

  1. Access control policies
  2. Session recordings
  3. Audit logs
  4. Configuration history
  5. Security architecture diagram

Maintenance Procedures​

Regular Maintenance Tasks​

Daily Checks (Automated)​

Monitoring alerts should notify of:

  • Connector offline
  • Socket failures
  • High connection failure rate
  • Unusual access patterns

Manual check (if needed):

# Quick connector health check
border0 connector list

Should show "online" status.

Weekly Tasks​

1. Review Failed Connections

Border0 Portal → Sessions → Filter: Failed

  • Investigate any recurring failures
  • Check for authentication issues
  • Verify user access needs

2. Check Connector Logs

# SSH to bastion
border0 client ssh pixlr-prod-bastion

# Check for errors/warnings
sudo journalctl -u border0-connector -p warning -n 100 --no-pager

3. Verify Socket Status

Border0 Portal → Sockets

All sockets should show green "Alive" status.

Monthly Tasks​

1. Access Review

  • Review all users in Border0 policies
  • Remove users who no longer need access
  • Verify access levels are appropriate

2. Session Activity Audit

  • Review session logs for anomalies
  • Check for after-hours production access
  • Verify no unauthorized access attempts

3. Update Check

# Check Border0 version
border0 version

# Compare with latest release
curl -s https://api.github.com/repos/borderzero/border0/releases/latest | grep tag_name

4. Security Group Review

  • Verify no unnecessary inbound rules
  • Confirm least privilege principle
  • Check for 0.0.0.0/0 sources (should be minimal)

Quarterly Tasks​

1. Rotate Connector Token

# Generate new token
border0 connector token create --connector-id 7d5a5d82-d38d-4c0c-badd-842302c16a6b

# Update service file
sudo nano /etc/systemd/system/border0-connector.service
# Replace BORDER0_TOKEN value

sudo systemctl daemon-reload
sudo systemctl restart border0-connector

2. Update Border0 Connector

# SSH to bastion
border0 client ssh pixlr-prod-bastion

# Download latest version
sudo curl -o /usr/local/bin/border0.new https://download.border0.com/linux_amd64/border0
sudo chmod +x /usr/local/bin/border0.new

# Test new version
/usr/local/bin/border0.new version

# If OK, replace old version
sudo systemctl stop border0-connector
sudo mv /usr/local/bin/border0.new /usr/local/bin/border0
sudo systemctl start border0-connector

# Verify
sudo systemctl status border0-connector

3. Review and Update Documentation

  • Update this document with any changes
  • Document new sockets created
  • Note any configuration changes

4. Disaster Recovery Test

Simulate connector failure and verify:

  • Monitoring alerts trigger
  • Recovery procedure works
  • Users are notified appropriately

Annual Tasks​

1. Full Security Audit

  • Review all configurations
  • Verify compliance requirements
  • Update security policies
  • Consider penetration testing

2. Architecture Review

  • Assess if current setup meets needs
  • Plan for scaling or improvements
  • Review cost optimization opportunities

3. Update Training Materials

  • Refresh user documentation
  • Update onboarding procedures
  • Create new training videos if needed

Backup and Recovery​

Configuration Backup​

What to backup:

  1. Border0 service file
  2. SSH configurations on all servers
  3. Border0 policy configurations
  4. Security group rules
  5. This documentation

Backup procedure:

# On bastion
sudo cp /etc/systemd/system/border0-connector.service ~/backups/
sudo cp /etc/ssh-ca.pub ~/backups/
sudo cp /etc/ssh/authorized_principals ~/backups/
sudo tar -czf ~/backups/border0-config-$(date +%Y%m%d).tar.gz ~/backups/*.service ~/backups/*.pub ~/backups/authorized_principals

Store securely:

  • Encrypted S3 bucket
  • Password manager (for small configs)
  • Internal documentation system

Disaster Recovery Scenarios​

Scenario 1: Bastion Instance Failure

Recovery steps:

  1. Launch new EC2 instance:

    # Use same AMI, VPC, and security groups
    # Instance type: t3.small
    # Key pair: existing admin key
  2. Install Border0:

    sudo curl -o /usr/local/bin/border0 https://download.border0.com/linux_amd64/border0
    sudo chmod +x /usr/local/bin/border0
  3. Create service file:

    sudo nano /etc/systemd/system/border0-connector.service
    # Paste backed-up configuration
  4. Start service:

    sudo systemctl daemon-reload
    sudo systemctl enable border0-connector
    sudo systemctl start border0-connector
  5. Verify:

    sudo systemctl status border0-connector
    border0 connector list # Should show online

Expected downtime: 10-15 minutes

Scenario 2: Connector Token Compromised

Recovery steps:

  1. Generate new token:

    border0 connector token create --connector-id 7d5a5d82-d38d-4c0c-badd-842302c16a6b
  2. Update all systems using the token:

    # On bastion
    sudo nano /etc/systemd/system/border0-connector.service
    # Update BORDER0_TOKEN

    sudo systemctl daemon-reload
    sudo systemctl restart border0-connector
  3. Revoke old token:

    • Border0 Portal → Connectors → Tokens → Revoke old token
  4. Audit access:

    • Review recent session logs
    • Check for unauthorized access
    • Investigate suspicious activity

Expected downtime: 1-2 minutes

Scenario 3: Target Server Failure

Recovery steps:

  1. Launch replacement instance (or restore from snapshot)

  2. Configure Border0 certificate auth:

    sudo tee /etc/ssh-ca.pub > /dev/null << 'EOF'
    ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNo...= Border0 ssh CA for Pixlr Group
    EOF

    sudo sh -c 'echo "TrustedUserCAKeys /etc/ssh-ca.pub" >> /etc/ssh/sshd_config'
    sudo sh -c 'echo "AuthorizedPrincipalsFile /etc/ssh/authorized_principals" >> /etc/ssh/sshd_config'
    sudo sh -c 'echo "border0_ssh_signed" > /etc/ssh/authorized_principals'

    sudo systemctl reload ssh
  3. Update Border0 socket (if IP changed):

    • Border0 Portal → Sockets → [Socket] → Edit
    • Update hostname to new IP
    • Save
  4. Test connection

Expected downtime: Depends on replacement speed (5-30 minutes)

Scaling Considerations​

Adding More Connectors​

When to add:

  • High latency to current connector
  • Geographic distribution of users
  • Redundancy requirements
  • Load distribution

Process:

  1. Launch new EC2 instance in target region/AZ

  2. Create new connector in Border0:

    border0 connector create \
    --name pixlr-prod-region2 \
    --description "Connector in us-west-2"
  3. Install and configure: (Same as initial setup)

  4. Assign sockets to new connector:

    • Edit existing sockets
    • Add new connector to connector list
    • Border0 will load-balance automatically

Adding More Sockets​

Planning:

  • Identify which services need Border0 access
  • Determine access requirements (who needs access)
  • Plan security configurations

Implementation:

  • Follow "Creating New Sockets" procedure in User Guide
  • Test thoroughly before granting wide access
  • Update documentation

Lessons Learned​

Technical Insights​

1. Border0 Command Syntax Matters​

Problem: Using connector run instead of connector start

Learning: Border0 CLI commands must be exact. The difference between run and start:

  • run: Interactive/foreground mode (for testing)
  • start: Daemon mode (for production services)

Best Practice: Always verify commands in Border0 documentation before creating systemd services.

2. Token Persistence is Critical​

Problem: User login tokens don't persist for service use

Learning: Border0 has two authentication mechanisms:

  • User tokens: For management operations (login, create resources)
  • Connector tokens: For service/daemon operation

Best Practice: Always generate connector-specific tokens for systemd services. Store in environment variables within the service file.

3. SSH Certificate Authentication is Superior​

Problem: EC2 Instance Connect requires IAM permissions and complex setup

Learning: SSH certificate authentication via Border0:

  • No AWS IAM dependencies
  • Centralized certificate management
  • Automatic rotation
  • Simpler configuration

Best Practice: Use Border0 certificate authentication as default. Only use AWS-native methods when specifically required.

4. Temporary SSH Key Method for Initial Setup​

Problem: Need to configure servers without existing access

Learning: AWS EC2 Instance Connect's send-ssh-public-key can push temporary keys (60-second validity) even to private instances.

Best Practice: Use EC2 Instance Connect for initial bootstrap, then configure permanent certificate-based auth immediately after.

5. systemd Service Configuration Pitfalls​

Problem: Service file modifications not taking effect

Learning: Multiple issues encountered:

  • Text editors with permission issues
  • Need for daemon-reload after changes
  • Quote escaping in bash commands

Best Practice:

  • Use sudo nano for direct editing
  • Always run systemctl daemon-reload after changes
  • Test with systemctl status before assuming it works
  • Check logs with journalctl

Process Improvements​

1. Documentation is Essential​

Lesson: Without this detailed documentation, reproducing this setup would require another 4-6 hours of troubleshooting.

Action: Maintain living documentation that includes:

  • Every configuration step
  • All error messages encountered
  • Solutions that worked and didn't work
  • Command reference

2. Test in Stages​

Lesson: Testing each component (connector, bastion socket, target sockets) separately made troubleshooting manageable.

Action: Always:

  1. Get connector online first
  2. Test with bastion socket
  3. Only then configure additional sockets
  4. Test each socket individually

3. Understand AWS Networking​

Lesson: Understanding VPC, subnets, security groups, and EC2 connectivity options was crucial for troubleshooting.

Action: Maintain network diagrams and document:

  • Which instances are in which subnets
  • Security group rules
  • Network paths between components

4. Have Multiple Access Paths​

Lesson: Having AWS SSM access when direct SSH failed was invaluable.

Action: Always maintain at least two ways to access critical infrastructure:

  • Border0 (primary)
  • AWS SSM (backup)
  • Emergency SSH key (last resort)

Appendix​

A. Command Reference​

Border0 CLI Commands​

# Authentication
border0 login
border0 logout
border0 whoami

# Connector Management
border0 connector list
border0 connector show <connector-id>
border0 connector token create --connector-id <id>
border0 connector token list --connector-id <id>
border0 connector token revoke --token-id <id>

# Socket Management
border0 socket ls
border0 socket show <socket-name>
border0 socket create ssh --name <name> --connector <connector> --host <ip> --port <port> --username <user>
border0 socket update <socket-name> [options]
border0 socket delete <socket-name>

# Client Connection
border0 client ssh <socket-name>
border0 client ssh <socket-name> -A # with agent forwarding
border0 client tcp --host <socket-name> --listener <local-port>

# Policy Management
border0 policy list
border0 policy show <policy-name>
border0 policy create [options]
border0 policy update <policy-name> [options]
border0 policy delete <policy-name>

Linux System Commands​

# systemd Service Management
sudo systemctl start <service>
sudo systemctl stop <service>
sudo systemctl restart <service>
sudo systemctl reload <service>
sudo systemctl status <service>
sudo systemctl enable <service>
sudo systemctl disable <service>
sudo systemctl daemon-reload