Border0 Implementation - Complete Documentation

Pixlr Infrastructure Access Solution

Table of Contents

1. Executive Summary
2. Phase 1: Initial Discovery and Diagnosis
3. Phase 2: Connector Recovery
4. Phase 3: Socket Configuration - Bastion Host
5. Phase 4: SSH Certificate Authentication Setup
6. Phase 5: Target Server Configuration
7. Complete Infrastructure Overview
8. User Guide
9. Troubleshooting Guide
10. Security Considerations
11. Maintenance Procedures
12. Lessons Learned

Executive Summary

This document chronicles the complete Border0 implementation journey at Pixlr, including:

• Recovery of an offline Border0 connector
• Configuration of SSH certificate-based authentication
• Setup of multiple secure access sockets
• Resolution of complex authentication and networking challenges

What is Border0?

Border0 is a zero-trust access platform that provides secure, identity-based access to infrastructure without traditional VPNs. It eliminates the need for:

• VPN clients and configurations
• Exposed inbound ports
• Manual SSH key management
• Bastion host public access

Why Border0 at Pixlr?

Before Border0:

• ❌ Required VPN for server access
• ❌ Manual SSH key distribution
• ❌ Not ideal bastion host management

After Border0:

• ✅ Web-based access (no VPN)
• ✅ Automatic certificate-based authentication
• ✅ Zero standing privileges
• ✅ Complete session recording
• ✅ Instant access provisioning

Implementation Results:

Metric	Result
Connector Status	Online and Stable
Sockets Configured	3 (Bastion + 2 Node.js servers)
Authentication Method	SSH Certificate (passwordless)
Session Recording	Enabled
Access Time	< 5 seconds
Setup Time per User	< 5 minutes

---

Phase 1: Initial Discovery and Diagnosis

Problem Statement

Initial Issue: Border0 connector showing as "Offline" in the portal with the following details:

Connector: pixlr-prod
Status: Offline
Last Seen: October 9, 2025, 07:35:32 GMT+8
Hostname: ip-10-44-10-88
Public IP: 52.2.193.239 (Ashburn, VA)
Private IP: 100.124.0.8

Step 1: Instance Identification

Challenge: Multiple EC2 instances existed, needed to identify which hosted the connector.

EC2 Infrastructure Inventory:

Total Instances: 32
Region: us-east-1 (N. Virginia)
VPC: Default VPC (10.44.0.0/16)

Key Instances:

Instance ID	Name	Type	Private IP	Public IP	Role
i-056f0f2e6a49d02f8	pixlr-prod-bastion-border0	t3.small	10.44.10.88	34.239.121.154	Border0 Connector Host
i-0b405da8be80a8a2f	pixlr-stag-nodejs-ub	t3a.medium	10.44.10.79	None	Staging Node.js Server
i-04907f267ca9683a6	pixlr-prod-nodejs-payment	c6a.xlarge	10.44.10.61	None	Production Payment Server
i-0ee43fe8d759a7b4f	pixlr-prod-bastion-developer	t3a.nano	-	3.233.65.175	Legacy Bastion
i-036edef128b9d0306	pixlr-prod-bastion-new	t4g.medium	-	100.25.202.135	Alternative Bastion

Discovery Process:

# Connected via AWS Systems Manager (SSM)
aws ssm start-session --target i-056f0f2e6a49d02f8

# Verified hostname matched Border0 connector
hostname
# Output: ip-10-44-10-88

# Confirmed private IP
ip addr show ens5
# Output: 10.44.10.88/27

# Retrieved instance metadata
curl -s http://169.254.169.254/latest/meta-data/instance-id
# Output: i-056f0f2e6a49d02f8

Confirmed: pixlr-prod-bastion-border0 (i-056f0f2e6a49d02f8) was the correct instance.

Step 2: Initial System Diagnostics

Commands Executed:

# Check for Border0 service
sudo systemctl status border0-connector
# Result: Unit border0-connector.service could not be found.

sudo systemctl status border0
# Result: Unit border0.service could not be found.

# Search for any Border0-related services
systemctl list-units --all | grep -i border
# Result: No Border0 services found

# Check for running Border0 processes
ps aux | grep -i border
# Result: No processes found (only grep itself)

# Comprehensive system search for Border0
sudo find / -name "*border0*" -type f 2>/dev/null

Critical Finding:

/usr/local/bin/border0

✅ Border0 binary exists but no service configured to run it.

Step 3: Network and Connectivity Checks

# Check if WireGuard port is listening (Border0 uses WireGuard)
sudo ss -tulpn | grep 32442
# Result: No service listening on port 32442

# Check all listening ports
sudo netstat -tlnp
# Result: Only SSH (22) and SSM agent ports open

# Check network interfaces
ip link show
# Output: Only ens5 (primary network) and lo (loopback)
# Missing: WireGuard tunnel interface (wg0 or utun)

# Test outbound connectivity to Border0 control plane
curl -I https://portal.border0.com
# Result: Connection successful

# Test DNS resolution
nslookup portal.border0.com
# Result: Resolved correctly

Diagnosis:

• Border0 binary installed but not running
• No active WireGuard tunnel
• Network connectivity to Border0 cloud is functional
• Issue is configuration/service-related, not network-related

Step 4: Error Log Discovery

Searched system logs:

# Check syslog for Border0 entries
sudo grep -r "border0" /var/log/

Critical Error Logs Found:

{
  "level": "fatal",
  "ts": "2025-10-16T10:21:52Z",
  "msg": "failed to get connector (v2) configuration",
  "config_path": "",
  "error": "no Border0 token in credentials chain"
}

{
  "level": "error",
  "ts": "2025-10-16T10:27:40Z",
  "msg": "not running as root: only non-network sockets are supported, sockets will not have local DNS entries"
}

Root Causes Identified:

1. Missing Authentication Token: Connector cannot authenticate with Border0 control plane
2. Permission Issues: Connector not running with root privileges
3. No Service Management: No systemd service to keep connector running

---

Phase 2: Connector Recovery

Step 1: Authentication Investigation

Attempted Login Methods:

Attempt 1: Basic User Login

sudo /usr/local/bin/border0 login

Process:

1. Command opened URL in browser
2. Successfully authenticated via Google SSO
3. Token supposedly saved

Verification:

sudo /usr/local/bin/border0 whoami

Error:

failed to get authenticated identity details: invalid token, please login

Analysis: Token not persisting between commands.

Attempt 2: Token Storage Investigation

# Check user's Border0 directory
ls -la ~/.border0/
# Result: ls: cannot access '/home/ssm-user/.border0/': No such file or directory

# Check root's Border0 directory
sudo ls -la /root/.border0/
# Result: ls: cannot access '/root/.border0/': No such file or directory

# Search for any Border0 config files
sudo find /etc -name "*border0*" 2>/dev/null
# Result: No config files found

# Check for environment variables
env | grep -i border
# Result: No Border0 environment variables set

# Check for credential files
find ~ -name "*border0*" -type f 2>/dev/null
# Result: No credential files found

Problem Identified: No persistent token storage configured.

Attempt 3: Root User Authentication

# Switch to root
sudo -i

# Attempt login as root
border0 login
# Result: Login successful

# Test authentication
border0 whoami
# Result: Still shows invalid token error

Conclusion: Login process works but token storage mechanism is broken.

Step 2: Connector Token Generation

Understanding Border0 Authentication:

Border0 uses two types of tokens:

1. User Tokens: For CLI management operations (login, create resources)
2. Connector Tokens: For connector service authentication (long-lived)

Solution: Generate a connector-specific token.

List Available Connectors

# First, successfully login and immediately run command
sudo border0 login && border0 connector list

Output:

┌──────────────────────────────────────┬────────────┬──────────────────────────────────┐
│ CONNECTOR ID                         │ NAME       │ DESCRIPTION                      │
├──────────────────────────────────────┼────────────┼──────────────────────────────────┤
│ 7d5a5d82-d38d-4c0c-badd-842302c16a6b │ pixlr-prod │ Pixlr Staging / Prod Environment │
└──────────────────────────────────────┴────────────┴──────────────────────────────────┘

✅ Connector exists in Border0 organization.

Generate Connector Token

border0 connector token create --connector-id 7d5a5d82-d38d-4c0c-badd-842302c16a6b

Success Output:

✓ Connector token created successfully

Token: eyJhbGciOiJFZERTQSIsImtpZCI6IjEzN....

⚠️  This token will only be displayed once. Store it securely.

Token Details:

• Type: Connector Authentication Token
• Connector ID: 7d5a5d82-d38d-4c0c-badd-842302c16a6b
• Organization: bade2394-d87d-41ec-8cb6-7e9ea3e4d034
• Issued: 1760672213 (Unix timestamp)
• Validity: Long-lived (no expiration)

Step 3: Systemd Service Creation

Challenge: Create a reliable service to run Border0 connector 24/7.

Attempt 1: Initial Service File (INCORRECT)

sudo cat > /etc/systemd/system/border0-connector.service << 'EOF'
[Unit]
Description=Border0 Connector
After=network.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/border0 connector run
Restart=always
RestartSec=10
Environment=BORDER0_TOKEN=eyJhbGciOiJFZERTQSIs....

[Install]
WantedBy=multi-user.target
EOF

Attempt to Start:

sudo systemctl daemon-reload
sudo systemctl start border0-connector
sudo systemctl status border0-connector

Error:

● border0-connector.service - Border0 Connector
   Loaded: loaded (/etc/systemd/system/border0-connector.service; disabled)
   Active: failed (Result: exit-code)
  Process: ExitCode=1

Error: unknown command "run" for "connector"

Problem: Border0 CLI uses connector start, not connector run.

Attempt 2: Service File Correction - Using sed

# Attempt to fix with sed
sudo sed -i 's/connector run/connector start/g' /etc/systemd/system/border0-connector.service

# Verify change
sudo cat /etc/systemd/system/border0-connector.service | grep ExecStart

Result: File still showed old content due to permission/caching issues.

Attempt 3: Manual Edit with nano (SUCCESSFUL)

# Open in text editor
sudo nano /etc/systemd/system/border0-connector.service

# Manually changed:
# FROM: ExecStart=/usr/local/bin/border0 connector run
# TO:   ExecStart=/usr/local/bin/border0 connector start

# Save: Ctrl+O, Enter
# Exit: Ctrl+X

Final Working Service File:

[Unit]
Description=Border0 Connector
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/border0 connector start
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
Environment=BORDER0_TOKEN=eyJhbGciOiJFZERTQSIsImtpZ....

[Install]
WantedBy=multi-user.target

Step 4: Service Activation and Verification

# Reload systemd to recognize changes
sudo systemctl daemon-reload

# Start the service
sudo systemctl start border0-connector

# Enable auto-start on boot
sudo systemctl enable border0-connector

# Check status
sudo systemctl status border0-connector

SUCCESS! Service Status:

● border0-connector.service - Border0 Connector
     Loaded: loaded (/etc/systemd/system/border0-connector.service; enabled; preset: enabled)
     Active: active (running) since Fri 2025-10-17 03:41:28 UTC; 8s ago
   Main PID: 81944 (border0)
      Tasks: 9 (limit: 2213)
     Memory: 22.6M (peak: 23.6M)
        CPU: 389ms
     CGroup: /system.slice/border0-connector.service
             └─81944 /usr/local/bin/border0 connector start

✅ Connector is now running!

Step 5: Connection Verification

# Monitor logs in real-time
sudo journalctl -u border0-connector -f

Success Logs:

{
  "level": "info",
  "ts": "2025-10-17T03:41:29Z",
  "msg": "starting Border0 connector",
  "version": "v1.1-1755-g6b19208c",
  "commit": "6b19208c"
}

{
  "level": "info",
  "ts": "2025-10-17T03:41:30Z",
  "msg": "connected to relay",
  "relay_pub": "BsMAvFScqUuMVfQFKxxDvQByaNNLngsfdHK7EMuDUAQ="
}

{
  "level": "info",
  "ts": "2025-10-17T03:41:30Z",
  "msg": "received STUN response for udp4",
  "old_address": "<nil>",
  "new_address": "52.2.193.239:62310",
  "changed": true
}

{
  "level": "info",
  "ts": "2025-10-17T03:41:30Z",
  "msg": "sending discoverability message",
  "discoverable": true,
  "addrs": [
    {
      "from_stun": true,
      "ip_address": "52.2.193.239",
      "port": 62310
    },
    {
      "iface_name": "ens5",
      "iface_cidr": "10.44.10.64/27",
      "ip_address": "10.44.10.88",
      "port": 32442
    }
  ]
}

{
  "level": "info",
  "ts": "2025-10-17T03:41:30Z",
  "msg": "got network state",
  "network_id": "e86a4b3e-965b-466a-a5e6-b42c80e34398",
  "ipv4": "100.124.0.8/32",
  "ipv6": "fd62:6f72:6465:7230::8/128",
  "devices_cidr_v4": "100.124.0.0/16",
  "devices_cidr_v6": "fd62:6f72:6465:7230::/64",
  "resources_cidr_v4": "100.126.0.0/16",
  "resources_cidr_v6": "fd62:6f72:6465:7231::/64"
}

{
  "level": "info",
  "ts": "2025-10-17T03:41:30Z",
  "msg": "re-configured wireguard peers successfully"
}

Key Success Indicators:

• ✅ Connected to Border0 relay server
• ✅ STUN discovered public IP and port
• ✅ Received network configuration
• ✅ WireGuard tunnel established
• ✅ Registered as discoverable

Step 6: Network Status Verification

# Check if Border0 process is running
ps aux | grep border0

Output:

root       81944  1.4  4.7 1375344 92864 ?       Ssl  03:41   0:00 /usr/local/bin/border0 connector start

# Check if WireGuard port is listening
sudo ss -tulpn | grep 32442

Output:

udp   UNCONN 0      0               0.0.0.0:32442      0.0.0.0:*    users:(("border0",pid=81944,fd=17))
udp   UNCONN 0      0                  [::]:32442         [::]:*    users:(("border0",pid=81944,fd=18))

✅ WireGuard listening on default port 32442 (both IPv4 and IPv6)

# Check network interfaces
ip link show

Output now includes:

1: lo: <LOOPBACK,UP,LOWER_UP>
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP>
3: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP>  ← Border0 WireGuard interface

✅ WireGuard tunnel interface created

Connector Recovery: COMPLETE ✓

---

Phase 3: Socket Configuration - Bastion Host

Understanding Border0 Sockets

What is a Socket?

A socket in Border0 represents a service that users can connect to. Think of it as a "door" through which authenticated users can access internal resources.

Socket Types:

• SSH: Terminal access to servers
• Database: MongoDB, PostgreSQL, MySQL connections
• HTTP/HTTPS: Web applications
• TCP: Generic TCP services
• RDP: Windows Remote Desktop

Step 1: Planning the Bastion Socket

Objective: Allow Border0 users to SSH directly into the bastion host itself.

Why this is needed:

• Administrative access to the Border0 connector
• Troubleshooting platform
• Jump box for accessing other internal servers

Configuration Requirements:

Parameter	Value	Reason
Socket Type	SSH	Terminal access needed
Socket Name	pixlr-prod-bastion	Descriptive identifier
Upstream Type	AWS EC2 Instance Connect	AWS-native auth OR Standard SSH
Port	22	Standard SSH port
Username	ubuntu	Default user on Ubuntu AMI
Connector	pixlr-prod	Our restored connector

Step 2: Initial Socket Creation Attempt

Method: AWS EC2 Instance Connect

Rationale:

• AWS-native authentication
• No need for SSH certificate configuration
• Leverages IAM permissions

Created Socket:

Socket Type: SSH
Socket Name: pixlr-prod-bastion
Description: Border0 bastion host for Pixlr Dev environment
Upstream Information:
  SSH Service Type: AWS EC2 Instance Connect
  Hostname: 10.44.10.88:22
  Username Source: Defined
  Username: ubuntu
  EC2 Instance ID: i-056f0f2e6a49d02f8
  EC2 Instance Region: us-east-1
Connector: pixlr-prod
Session Recording: Enabled

Socket Created Successfully in Portal:

• Socket ID: Generated
• DNS Name: pixlr-prod-bastion-pixlr-group.border0.io
• Status: Alive
• IP Address: 100.126.0.100 (Border0 internal network)

Step 3: Connection Test - EC2 Instance Connect

Attempted Connection via Border0 Portal:

Click "Connect" → Select socket → Wait for connection

Error Encountered:

Error: failed to open a new session for the SSH client: ssh: unexpected packet in response to channel open: <nil>
[SSH session terminated]

Detailed Error from Socket:

ssh_ec2_instance_connect: failed to setup ec2 connect: failed to send ssh public key: 
operation error EC2 Instance Connect: SendSSHPublicKey, 
https response error StatusCode: 400, 
RequestID: f1a4f1f5-f68f-44d1-9221-12fa9c4ee87e, 
api error AccessDeniedException: 
User: arn:aws:sts::395785058010:assumed-role/EC2-SSM-Role/i-056f0f2e6a49d02f8 
is not authorized to perform: ec2-instance-connect:SendSSHPublicKey 
on resource: arn:aws:ec2:us-east-1:395785058010:instance/i-056f0f2e6a49d02f8 
because no identity-based policy allows the ec2-instance-connect:SendSSHPublicKey action

Root Cause Analysis:

The bastion instance has an IAM role (EC2-SSM-Role) that allows Systems Manager access, but does NOT have permission to use EC2 Instance Connect.

IAM Role Required Permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ec2-instance-connect:SendSSHPublicKey",
            "Resource": "arn:aws:ec2:us-east-1:395785058010:instance/i-056f0f2e6a49d02f8"
        }
    ]
}

Decision: Rather than modify IAM policies (requires approval), switch to Standard SSH with Border0 Certificate authentication.

Step 4: SSH Certificate Authentication Setup

What is SSH Certificate Authentication?

Instead of using SSH keys, Border0 acts as a Certificate Authority (CA). It:

1. Issues short-lived SSH certificates to authenticated users
2. Target SSH servers trust certificates signed by Border0's CA
3. No need to distribute or manage SSH keys

Benefits:

• ✅ No SSH keys to manage
• ✅ Automatic certificate rotation
• ✅ Certificate validity tied to user authentication
• ✅ Centralized access control

Border0 SSH CA Public Key:

Retrieved from Border0 Portal → Organization Settings → Details:

ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ1HJQ3fctrXZ5gmQiVpyUEnlkiXj4pWH3ytGx9YnacbXEa4eFqyhTagxS1c9XtfPBPJM2QP1ZZXUXM9cuEIdIk= Border0 ssh CA for Pixlr Group

Step 5: Configuring the Bastion for Certificate Auth

Connected to bastion via AWS SSM:

aws ssm start-session --target i-056f0f2e6a49d02f8

Create Border0 CA Certificate File

Method 1 Attempt (heredoc - FAILED):

sudo bash -c 'cat > /etc/ssh-ca.pub << EOF
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyN.....= Border0 ssh CA for Pixlr Group
EOF'

Error:

bash: line 1: warning: here-document at line 1 delimited by end-of-file (wanted `EOF')
cat: ecdsa-sha2-nistp256: No such file or directory
cat: AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ1HJQ3fctrXZ5gmQiVpyUEnlkiXj4pWH3ytGx9YnacbXEa4eFqyhTagxS1c9XtfPBPJM2QP1ZZXUXM9cuEIdIk=: No such file or directory

Issue: Quote escaping problem with nested bash commands.

Method 2 (tee - SUCCESS):

sudo tee /etc/ssh-ca.pub > /dev/null << 'EOF'
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbml...= Border0 ssh CA for Pixlr Group
EOF

Verification:

cat /etc/ssh-ca.pub

Output:

ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTIt...= Border0 ssh CA for Pixlr Group

✅ CA certificate file created successfully

Configure SSH Daemon

Check existing configuration:

grep "TrustedUserCAKeys" /etc/ssh/sshd_config
# Output: (empty - not configured)

grep "AuthorizedPrincipalsFile" /etc/ssh/sshd_config
# Output: #AuthorizedPrincipalsFile none

Add Border0 certificate trust:

# Add TrustedUserCAKeys directive
sudo sh -c 'echo "TrustedUserCAKeys /etc/ssh-ca.pub" >> /etc/ssh/sshd_config'

# Add AuthorizedPrincipalsFile directive
sudo sh -c 'echo "AuthorizedPrincipalsFile /etc/ssh/authorized_principals" >> /etc/ssh/sshd_config'

# Create authorized principals file
sudo sh -c 'echo "border0_ssh_signed" > /etc/ssh/authorized_principals'

Verify configuration:

sudo tail -5 /etc/ssh/sshd_config

Output:

#       AllowTcpForwarding no
#       PermitTTY no
#       ForceCommand cvs server
TrustedUserCAKeys /etc/ssh-ca.pub
AuthorizedPrincipalsFile /etc/ssh/authorized_principals

cat /etc/ssh/authorized_principals

Output:

border0_ssh_signed

✅ SSH configuration updated

Test SSH Configuration

# Test configuration for syntax errors
sudo sshd -t

Warning (non-critical):

Missing privilege separation directory: /run/sshd

Note: This warning is normal and can be ignored. The directory is created when SSH starts.

Start/Restart SSH Service

Check SSH service status:

systemctl list-unit-files | grep ssh

Output:

ssh.service                disabled        enabled
ssh.socket                 enabled         enabled

Start SSH service:

sudo systemctl start ssh
sudo systemctl enable ssh

Verify SSH is running:

sudo systemctl status ssh

Output:

● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/usr/lib/systemd/system/ssh.service; enabled; preset: enabled)
     Active: active (running) since Fri 2025-10-17 05:08:17 UTC; 5s ago
   Main PID: 84373 (sshd)
      Tasks: 1 (limit: 2213)
     Memory: 1.2M (peak: 1.4M)
        CPU: 20ms
     CGroup: /system.slice/ssh.service
             └─84373 "sshd: /usr/sbin/sshd -D"

Oct 17 05:08:17 ip-10-44-10-88 systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
Oct 17 05:08:17 ip-10-44-10-88 sshd[84373]: Server listening on 0.0.0.0 port 22.
Oct 17 05:08:17 ip-10-44-10-88 sshd[84373]: Server listening on :: port 22.
Oct 17 05:08:17 ip-10-44-10-88 systemd[1]: Started ssh.service - OpenBSD Secure Shell server.

✅ SSH service running and listening

# Verify SSH is listening on port 22
sudo ss -tlnp | grep :22

Output:

LISTEN 0      4096         0.0.0.0:22        0.0.0.0:*    users:(("sshd",pid=84373,fd=3),("systemd",pid=1,fd=114))
LISTEN 0      4096            [::]:22           [::]:*    users:(("sshd",pid=84373,fd=4),("systemd",pid=1,fd=115))

✅ SSH listening on all interfaces (IPv4 and IPv6)

Step 6: Update Border0 Socket to Use Certificate Auth

Changed Socket Configuration:

FROM:
  Upstream Connection Type: AWS EC2 Instance Connect
  EC2 Instance ID: i-056f0f2e6a49d02f8
  EC2 Instance Region: us-east-1

TO:
  Upstream Connection Type: Standard SSH
  Authentication Strategy: Border0 Certificate
  Hostname: 10.44.10.88
  Port: 22
  Username: ubuntu
  Connector: pixlr-prod

Socket Updated Successfully

Step 7: Connection Test - SUCCESS!

Attempted Connection:

Border0 Portal → Sockets → pixlr-prod-bastion → Connect

Result:

Welcome to Ubuntu 24.04.3 LTS (GNU/Linux 6.14.0-1011-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/pro

System information as of Fri Oct 17 06:23:57 UTC 2025

  System load:  0.0               Temperature:           -273.1 C
  Usage of /:   40.1% of 6.71GB   Processes:             115
  Memory usage: 23%               Users logged in:       0
  Swap usage:   0%                IPv4 address for ens5: 10.44.10.88

Last login: Fri Oct 17 05:15:50 2025 from 10.44.10.88
ubuntu@ip-10-44-10-88:~$

✅ BASTION ACCESS WORKING!

Verification in auth.log:

sudo tail -f /var/log/auth.log

Output:

Oct 17 06:23:57 ip-10-44-10-88 sshd[84521]: Accepted publickey for ubuntu from 100.126.0.100 port 53847 ssh2: ECDSA-CERT SHA256:...
Oct 17 06:23:57 ip-10-44-10-88 sshd[84521]: pam_unix(sshd:session): session opened for user ubuntu(uid=1000)

Key observations:

• ✅ Authentication method: publickey (certificate)
• ✅ Source: 100.126.0.100 (Border0 network)
• ✅ Certificate type: ECDSA-CERT
• ✅ Session opened successfully

Bastion Socket Configuration: COMPLETE ✓

---

Phase 4: SSH Certificate Authentication Setup

Now that the bastion works, we need to configure the same certificate authentication on target servers.

Target Servers

From the Jira ticket requirements:

1. Staging Node.js Server

   • Instance ID: i-0b405da8be80a8a2f
   • Name: pixlr-stag-nodejs-ub
   • Private IP: 10.44.10.79
   • Purpose: Staging application server

2. Production Node.js Payment Server

   • Instance ID: i-04907f267ca9683a6
   • Name: pixlr-prod-nodejs-payment
   • Private IP: 10.44.10.61
   • Purpose: Production payment processing server

Challenge: No Direct Access

Problem: These servers are in private subnets with:

• ❌ No public IP addresses
• ❌ No EC2 Instance Connect available (not in public subnet)
• ❌ No SSM agent connected
• ❌ No SSH keys on the bastion to access them

Error when trying EC2 Instance Connect:

No public IPv4 or IPv6 address assigned
Instance is not in public subnet
Associated subnet subnet-906b62b8 (apps-internal-nat) is not a public subnet.

Error when trying SSM:

An error occurred (TargetNotConnected) when calling the StartSession operation: 
i-0b405da8be80a8a2f is not connected.

Solution: Temporary EC2 Instance Connect from AWS CLI

AWS EC2 Instance Connect can push a temporary SSH key (valid for 60 seconds) to an instance, even if it's in a private subnet, as long as the AWS API can reach it.

Generate SSH Key on Bastion

# On pixlr-prod-bastion-border0 (already connected via Border0)
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ""

Output:

Generating public/private ed25519 key pair.
Your identification has been saved in /home/ubuntu/.ssh/id_ed25519
Your public key has been saved in /home/ubuntu/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:vnQhvnUShdGe9MZ...../DcDPFQ ubuntu@ip-10-44-10-88

cat ~/.ssh/id_ed25519.pub

Output:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAA... ubuntu@ip-10-44-10-88

Key generated: This will be used to initially access the target servers.

---

Phase 5: Target Server Configuration

Configuring Staging Server (pixlr-stag-nodejs-ub)

Step 1: Push Temporary SSH Key

From local machine with AWS CLI:

aws ec2-instance-connect send-ssh-public-key \
  --region us-east-1 \
  --instance-id i-0b405da8be80a8a2f \
  --instance-os-user ubuntu \
  --ssh-public-key "ssh-ed25519 AAAAC3NzaC..."

Response:

{
    "RequestId": "305a6785-bd0d-419b-9893-2469add25c42",
    "Success": true
}

✅ Temporary key pushed successfully (valid for 60 seconds)

Step 2: SSH from Bastion (Within 60 seconds!)

From bastion terminal:

ssh ubuntu@10.44.10.79

Success:

Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.15.0-1073-aws x86_64)

ubuntu@ip-10-44-10-79:~$

✅ Connected to staging server

Step 3: Configure Border0 Certificate Authentication

Run configuration commands:

# Create Border0 CA certificate file
sudo tee /etc/ssh-ca.pub > /dev/null << 'EOF'
ecdsa-sha2-nistp256 AAAAE2VjZHNhLX....= Border0 ssh CA for Pixlr Group
EOF

# Add SSH configuration
sudo sh -c 'echo "TrustedUserCAKeys /etc/ssh-ca.pub" >> /etc/ssh/sshd_config'
sudo sh -c 'echo "AuthorizedPrincipalsFile /etc/ssh/authorized_principals" >> /etc/ssh/sshd_config'
sudo sh -c 'echo "border0_ssh_signed" > /etc/ssh/authorized_principals'

# Start and reload SSH
sudo systemctl start ssh
sudo systemctl reload ssh

Verification:

# Verify CA file
cat /etc/ssh-ca.pub
# Output: [Border0 CA certificate]

# Verify SSH config
sudo tail -3 /etc/ssh/sshd_config
# Output:
# TrustedUserCAKeys /etc/ssh-ca.pub
# AuthorizedPrincipalsFile /etc/ssh/authorized_principals

# Verify principals
cat /etc/ssh/authorized_principals
# Output: border0_ssh_signed

# Verify SSH status
sudo systemctl status ssh
# Output: active (running)

✅ Staging server configured

# Exit back to bastion
exit

Configuring Production Server (pixlr-prod-nodejs-payment)

Step 1: Push Temporary SSH Key

From local machine:

aws ec2-instance-connect send-ssh-public-key \
  --region us-east-1 \
  --instance-id i-04907f267ca9683a6 \
  --instance-os-user ubuntu \
  --ssh-public-key "ssh-ed25519 AAAAC3NzaC1lZDI..."

Response:

{
    "RequestId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "Success": true
}

Step 2: SSH from Bastion

ssh ubuntu@10.44.10.61

Success:

Welcome to Ubuntu 22.04.5 LTS (GNU/Linux 6.5.0-1025-aws x86_64)

ubuntu@ip-10-44-10-61:~$

✅ Connected to production payment server

Step 3: Configure Border0 Certificate Authentication

# Create Border0 CA certificate file
sudo tee /etc/ssh-ca.pub > /dev/null << 'EOF'
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbm....= Border0 ssh CA for Pixlr Group
EOF

# Add SSH configuration
sudo sh -c 'echo "TrustedUserCAKeys /etc/ssh-ca.pub" >> /etc/ssh/sshd_config'
sudo sh -c 'echo "AuthorizedPrincipalsFile /etc/ssh/authorized_principals" >> /etc/ssh/sshd_config'
sudo sh -c 'echo "border0_ssh_signed" > /etc/ssh/authorized_principals'

# Start and reload SSH
sudo systemctl start ssh
sudo systemctl reload ssh

Verification:

cat /etc/ssh-ca.pub
sudo tail -3 /etc/ssh/sshd_config
cat /etc/ssh/authorized_principals
sudo systemctl status ssh

✅ Production server configured

exit

Creating Border0 Sockets for Target Servers

Socket 1: Staging Node.js Server

Border0 Portal → Sockets → Add New Socket

Configuration:

Socket Type: SSH
Socket Name: pixlr-stag-nodejs-ub
Description: Pixlr Staging Node.js Server

Upstream Configuration:
  Connection Type: Standard SSH
  Authentication Strategy: Border0 Certificate
  Hostname: 10.44.10.79
  Port: 22
  SSH Username: ubuntu
  Prompt Clients for User Name: Enter Manually
  
Connectors: pixlr-prod

Session Recording: Enabled

Socket Created:

• DNS Name: pixlr-stag-nodejs-ub-pixlr-group.border0.io
• IP Address: 100.126.0.79
• Status: Alive

Socket 2: Production Node.js Payment Server

Configuration:

Socket Type: SSH
Socket Name: pixlr-prod-nodejs-payment
Description: Pixlr Production Node.js Payment Server

Upstream Configuration:
  Connection Type: Standard SSH
  Authentication Strategy: Border0 Certificate
  Hostname: 10.44.10.61
  Port: 22
  SSH Username: ubuntu
  Prompt Clients for User Name: Enter Manually
  
Connectors: pixlr-prod

Session Recording: Enabled

Socket Created:

• DNS Name: pixlr-prod-nodejs-payment-pixlr-group.border0.io
• IP Address: 100.126.0.61
• Status: Alive

Connection Testing

Test Staging Server

Border0 Portal → Sockets → pixlr-stag-nodejs-ub → Connect

Success:

Welcome to Ubuntu 20.04.6 LTS
ubuntu@ip-10-44-10-79:~$

✅ Staging server accessible via Border0

Test Production Server

Border0 Portal → Sockets → pixlr-prod-nodejs-payment → Connect

Success:

Welcome to Ubuntu 22.04.5 LTS
ubuntu@ip-10-44-10-61:~$

✅ Production server accessible via Border0

All Sockets Operational: COMPLETE ✓

---

Complete Infrastructure Overview

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────┐
│                         Users (Web Browser / CLI)                        │
└────────────────────────────────┬────────────────────────────────────────┘
                                 │
                                 │ HTTPS
                                 │
┌────────────────────────────────▼────────────────────────────────────────┐
│                     Border0 Control Plane (Cloud)                        │
│                         portal.border0.com                               │
│                                                                           │
│  - User Authentication (Google SSO)                                      │
│  - Policy Management                                                     │
│  - Session Recording                                                     │
│  - Audit Logging                                                         │
└────────────────────────────────┬────────────────────────────────────────┘
                                 │
                                 │ WireGuard (UDP 32442)
                                 │
┌────────────────────────────────▼────────────────────────────────────────┐
│                    Border0 Connector (pixlr-prod)                        │
│              EC2: i-056f0f2e6a49d02f8 (t3.small)                         │
│            pixlr-prod-bastion-border0 (10.44.10.88)                      │
│                                                                           │
│  - WireGuard Tunnel: 100.124.0.8/32                                      │
│  - Service IP Range: 100.126.0.0/16                                      │
│  - Status: Online                                                        │
│  - Version: v1.1-1755-g6b19208c                                          │
└───────────────┬─────────────────┬──────────────────┬─────────────────────┘
                │                 │                  │
                │ SSH (22)        │ SSH (22)         │ SSH (22)
                │                 │                  │
┌───────────────▼──────┐ ┌────────▼──────┐ ┌────────▼──────────────────┐
│  Socket: Bastion     │ │ Socket: Stag  │ │ Socket: Prod Payment      │
│  10.44.10.88:22      │ │ 10.44.10.79   │ │ 10.44.10.61               │
│  Border0 IP:         │ │ Border0 IP:   │ │ Border0 IP:               │
│  100.126.0.100       │ │ 100.126.0.79  │ │ 100.126.0.61              │
│                      │ │               │ │                           │
│  Self (bastion)      │ │ Node.js Stag  │ │ Node.js Prod Payment      │
└──────────────────────┘ └───────────────┘ └───────────────────────────┘

Network Details

AWS VPC:

• VPC ID: Default VPC
• CIDR: 10.44.0.0/16
• Region: us-east-1 (N. Virginia)
• Subnets: Multiple across AZs

Border0 Network:

• Device Network: 100.124.0.0/16
• Socket Network: 100.126.0.0/16
• IPv6: fd62:6f72:6465:72xx::/64

Socket Summary

Socket Name	Type	Upstream Target	Border0 IP	Status	DNS Name
pixlr-prod-bastion	SSH	10.44.10.88:22	100.126.0.100	Active	pixlr-prod-bastion-pixlr-group.border0.io
pixlr-stag-nodejs-ub	SSH	10.44.10.79:22	100.126.0.79	Active	pixlr-stag-nodejs-ub-pixlr-group.border0.io
pixlr-prod-nodejs-payment	SSH	10.44.10.61:22	100.126.0.61	Active	pixlr-prod-nodejs-payment-pixlr-group.border0.io

Security Configuration

SSH Certificate Authentication:

• CA Type: ECDSA-SHA2-NISTP256
• Principal: border0_ssh_signed
• Certificate Lifetime: Session-based (short-lived)
• Revocation: Automatic on policy change

Access Control:

• Authentication: Google SSO
• Authorization: Border0 policies
• Session Recording: Enabled
• Audit Logging: Complete

---

User Guide

For End Users

Prerequisites

1. Email Address: Must be added to Border0 policy
2. Google Account: For SSO authentication
3. Web Browser: Any modern browser (Chrome, Firefox, Safari, Edge)

Accessing Servers (Web Portal)

Step 1: Login

1. Go to https://portal.border0.com
2. Click "Sign In"
3. Select "Sign in with Google"
4. Use your @pixlr.com email

Step 2: View Available Resources

After login, you'll see a list of servers:

• pixlr-prod-bastion - Border0 bastion host
• pixlr-stag-nodejs-ub - Staging application server
• pixlr-prod-nodejs-payment - Production payment server

Step 3: Connect to a Server

1. Click on the server name (e.g., pixlr-stag-nodejs-ub)
2. Click the "Connect" button
3. Wait 2-5 seconds for connection
4. A terminal window opens in your browser
5. You're now connected via SSH!

Example Session:

Connecting to pixlr-stag-nodejs-ub...
Connected successfully!

Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.15.0-1073-aws x86_64)

ubuntu@ip-10-44-10-79:~$ whoami
ubuntu

ubuntu@ip-10-44-10-79:~$ pwd
/home/ubuntu

ubuntu@ip-10-44-10-79:~$ # You can now run any commands

Step 4: Disconnect

• Close the browser tab, or
• Type exit and press Enter

Your session is automatically recorded and logged.

Using Border0 CLI (Optional - For Advanced Users)

Installation:

# macOS
brew install borderzero/tap/border0

# Linux
curl -fsSL https://download.border0.com/linux_install.sh | bash

# Windows
# Download from https://download.border0.com/windows_amd64/border0.exe

Login:

border0 login

Opens browser for authentication.

List Available Sockets:

border0 socket ls

Output:

┌────────────────────────────┬──────────────┬────────────┐
│ NAME                       │ TYPE         │ STATUS     │
├────────────────────────────┼──────────────┼────────────┤
│ pixlr-prod-bastion         │ SSH          │ online     │
│ pixlr-stag-nodejs-ub       │ SSH          │ online     │
│ pixlr-prod-nodejs-payment  │ SSH          │ online     │
└────────────────────────────┴──────────────┴────────────┘

Connect via CLI:

# Connect to staging
border0 client ssh pixlr-stag-nodejs-ub

# Connect to production
border0 client ssh pixlr-prod-nodejs-payment

# Connect to bastion
border0 client ssh pixlr-prod-bastion

SSH with Agent Forwarding:

border0 client ssh pixlr-stag-nodejs-ub -A

For Administrators

Adding New Users

Step 1: Access Border0 Portal

1. Login to https://portal.border0.com
2. Navigate to Policies section
3. Click on default-org-policy (or relevant policy)

Step 2: Add User Email

1. In the "WHO" section, click Edit
2. Add user's email address (e.g., newuser@pixlr.com)
3. Click Save

Result: User can immediately access allowed resources.

Removing Users

Step 1: Access Policy

1. Navigate to Policies
2. Select the policy containing the user

Step 2: Remove User

1. In the "WHO" section, click Edit
2. Remove user's email address
3. Click Save

Result: User loses access immediately. Active sessions are terminated.

Creating New Sockets

Prerequisites:

• Target server must be reachable from connector
• SSH service running on target
• Border0 CA certificate configured on target (for certificate auth)

Step 1: Configure Target Server

SSH to target server and run:

sudo tee /etc/ssh-ca.pub > /dev/null << 'EOF'
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbm.....= Border0 ssh CA for Pixlr Group
EOF

sudo sh -c 'echo "TrustedUserCAKeys /etc/ssh-ca.pub" >> /etc/ssh/sshd_config'
sudo sh -c 'echo "AuthorizedPrincipalsFile /etc/ssh/authorized_principals" >> /etc/ssh/sshd_config'
sudo sh -c 'echo "border0_ssh_signed" > /etc/ssh/authorized_principals'

sudo systemctl reload ssh

Step 2: Create Socket in Border0 Portal

1. Navigate to Sockets
2. Click Add New Socket
3. Fill in configuration:
   • Socket Type: SSH
   • Socket Name: descriptive-name
   • Description: Purpose of the server
   • Connection Type: Standard SSH
   • Authentication: Border0 Certificate
   • Hostname: Private IP of target
   • Port: 22
   • Username: ubuntu (or appropriate)
   • Connector: pixlr-prod
4. Click Create

Step 3: Test Connection

1. Click on the newly created socket
2. Click Connect
3. Verify you can access the server

Monitoring and Troubleshooting

Check Connector Status:

# SSH to bastion
border0 client ssh pixlr-prod-bastion

# Check service status
sudo systemctl status border0-connector

# View real-time logs
sudo journalctl -u border0-connector -f

# Check last 50 lines
sudo journalctl -u border0-connector -n 50 --no-pager

Check Network Connectivity:

# Verify WireGuard is listening
sudo ss -tulpn | grep 32442

# Check Border0 process
ps aux | grep border0

# Test connectivity to Border0 control plane
curl -I https://portal.border0.com

View Socket Status:

In Border0 Portal:

1. Navigate to Sockets
2. Check "Status" column - should show green "Alive"
3. Click on socket to see details
4. Check "Connectors" section - should show "pixlr-prod" online

View Session History:

1. Navigate to Sessions
2. Filter by user, socket, or date range
3. Click on a session to view recording
4. Review audit logs for compliance

---

Troubleshooting Guide

Common Issues and Solutions

Issue 1: "Socket shows as Offline"

Symptoms:

• Socket status shows red "Offline" indicator
• Cannot connect to the socket
• Connection timeout

Possible Causes & Solutions:

Cause 1: Connector is down

Check connector status:

sudo systemctl status border0-connector

If stopped, restart:

sudo systemctl restart border0-connector

Cause 2: Target server unreachable

Test connectivity from bastion:

# Ping target
ping -c 3 10.44.10.79

# Test SSH port
nc -zv 10.44.10.79 22

# Try SSH manually
ssh ubuntu@10.44.10.79

Cause 3: Security group blocking traffic

Verify security group allows SSH from bastion:

• Go to AWS Console → EC2 → Security Groups
• Find target server's security group
• Ensure inbound rule: TCP 22 from bastion's security group

Cause 4: SSH service not running on target

SSH to target and check:

sudo systemctl status ssh
sudo systemctl start ssh

Issue 2: "Permission denied (publickey)"

Symptoms:

• Connection fails with authentication error
• Works for some users but not others

Possible Causes & Solutions:

Cause 1: Border0 certificate not configured

Verify on target server:

# Check if CA file exists
cat /etc/ssh-ca.pub

# Check SSH config
grep "TrustedUserCAKeys" /etc/ssh/sshd_config
grep "AuthorizedPrincipalsFile" /etc/ssh/sshd_config

# Check principals file
cat /etc/ssh/authorized_principals

If missing, reconfigure:

sudo tee /etc/ssh-ca.pub > /dev/null << 'EOF'
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTIt...= Border0 ssh CA for Pixlr Group
EOF

sudo sh -c 'echo "TrustedUserCAKeys /etc/ssh-ca.pub" >> /etc/ssh/sshd_config'
sudo sh -c 'echo "AuthorizedPrincipalsFile /etc/ssh/authorized_principals" >> /etc/ssh/sshd_config'
sudo sh -c 'echo "border0_ssh_signed" > /etc/ssh/authorized_principals'

sudo systemctl reload ssh

Cause 2: User not in Border0 policy

Check Border0 Portal:

1. Navigate to Policies
2. Verify user's email is in the "WHO" section
3. Add if missing

Cause 3: SSH daemon configuration error

Test SSH configuration:

sudo sshd -t

View detailed logs:

sudo tail -f /var/log/auth.log

While tailing logs, attempt connection to see specific error.

Issue 3: "Connector Token Invalid"

Symptoms:

• Connector fails to start
• Logs show: "invalid token" or "unauthorized"

Solution:

Generate new connector token:

# Login to Border0
border0 login

# List connectors
border0 connector list

# Generate new token
border0 connector token create --connector-id 7d5a5d82-d38d-4c0c-badd-842302c16a6b

Update service file:

sudo nano /etc/systemd/system/border0-connector.service

# Replace the BORDER0_TOKEN environment variable value
# Save and exit

sudo systemctl daemon-reload
sudo systemctl restart border0-connector

Issue 4: "Connection timeout / Can't reach socket"

Symptoms:

• Connection hangs
• Browser shows loading spinner indefinitely
• CLI shows "connection timeout"

Diagnosis:

Step 1: Check Border0 control plane status

curl -I https://portal.border0.com

Step 2: Check connector connectivity

sudo journalctl -u border0-connector | grep -i "connected to relay"

Should see recent "connected to relay" messages.

Step 3: Check WireGuard tunnel

ip link show | grep wg
sudo wg show  # If wireguard-tools installed

Step 4: Restart connector

sudo systemctl restart border0-connector
sleep 10
sudo systemctl status border0-connector

Issue 5: "SSH service won't start on target"

Symptoms:

• systemctl status ssh shows failed
• SSH connections fail

Solution:

Check SSH configuration:

sudo sshd -t

Common errors:

Error: "Missing privilege separation directory"

sudo mkdir -p /run/sshd
sudo chmod 0755 /run/sshd

Error: "Bad configuration option"

# Review recent changes
sudo tail -20 /etc/ssh/sshd_config

# Remove or fix problematic lines
sudo nano /etc/ssh/sshd_config

Error: "Permission denied" on config files

sudo chmod 644 /etc/ssh/sshd_config
sudo chmod 644 /etc/ssh-ca.pub
sudo chmod 644 /etc/ssh/authorized_principals

Restart SSH:

sudo systemctl restart ssh

Issue 6: "Session recording not working"

Symptoms:

• Sessions listed but no recording available
• "Recording unavailable" message

Solution:

1. Check socket configuration:

   • Go to Border0 Portal → Sockets → [Your Socket]
   • Verify "Session Recording: Enabled"

2. If disabled, enable it:

   • Click "Edit"
   • Enable "Session Recording"
   • Save changes

Note: Only sessions after enabling will be recorded.

Debug Commands Reference

Connector Diagnostics:

# Service status
sudo systemctl status border0-connector

# Recent logs
sudo journalctl -u border0-connector -n 100 --no-pager

# Follow logs live
sudo journalctl -u border0-connector -f

# Check process
ps aux | grep border0

# Check listening ports
sudo ss -tulpn | grep border0

Network Diagnostics:

# Check WireGuard tunnel
ip addr show | grep -A 5 wg

# Test target connectivity
nc -zv <target-ip> 22

# Test DNS resolution
nslookg portal.border0.com

# Check routing
ip route show

SSH Diagnostics (on target):

# SSH configuration test
sudo sshd -t

# Check SSH is running
sudo systemctl status ssh

# Check SSH is listening
sudo ss -tlnp | grep :22

# View SSH logs
sudo tail -f /var/log/auth.log

# Verbose SSH connection test
ssh -vvv ubuntu@localhost

Border0 CLI Diagnostics:

# Check authentication
border0 whoami

# List connectors
border0 connector list

# List sockets
border0 socket ls

# Get socket details
border0 socket show <socket-name>

---

Security Considerations

Current Security Posture

Strengths ✅

1. Zero Trust Architecture

   • Every connection authenticated individually
   • No persistent VPN tunnels
   • Identity-based access control

2. Certificate-Based Authentication

   • No static SSH keys to manage
   • Short-lived certificates
   • Automatic rotation
   • Centralized revocation

3. Complete Audit Trail

   • All sessions recorded
   • Full audit logs
   • User identity tracking
   • Compliance-ready

4. Network Segmentation

   • Bastion in DMZ-like configuration
   • Target servers in private subnets
   • No public IPs on application servers

5. Encrypted Tunnels

   • WireGuard encryption
   • Modern cryptography
   • Perfect forward secrecy

Potential Improvements 🔄

High Priority:

1. Enable AWS Systems Manager Session Manager

   • Eliminates need for public IP on bastion
   • Additional layer of AWS-native security
   • Cost: Free (included with AWS)

   Implementation:

   # Attach IAM role with AmazonSSMManagedInstanceCore
   # Install SSM agent (usually pre-installed on Ubuntu AMIs)
   sudo snap install amazon-ssm-agent --classic
   sudo snap start amazon-ssm-agent

2. Implement IP Allowlisting for Border0 Portal Access

   • Restrict portal access to office/VPN IPs
   • Reduces attack surface

   Configuration: Border0 Portal → Policies → Add IP restrictions

3. Enable Multi-Factor Authentication (MFA)

   • Require MFA for all Border0 logins
   • Google Authenticator or hardware keys

   Configuration: Border0 Portal → Organization Settings → Security

Medium Priority:

4. Implement Just-In-Time (JIT) Access

   • Time-limited access grants
   • Approval workflows for production
   • Automatic access expiration

   Configuration: Use Border0 advanced policies

5. Enhanced Monitoring

   • Set up CloudWatch alarms for connector failures
   • Alert on unusual session patterns
   • Integration with SIEM system

   Implementation:

   # CloudWatch alarm for connector offline
   aws cloudwatch put-metric-alarm --alarm-name border0-connector-down \
     --alarm-description "Alert when Border0 connector is offline" \
     --metric-name StatusCheckFailed \
     --namespace AWS/EC2 \
     --statistic Average \
     --period 300 \
     --threshold 1 \
     --comparison-operator GreaterThanThreshold \
     --dimensions Name=InstanceId,Value=i-056f0f2e6a49d02f8

6. Regular Security Audits

   • Monthly access review
   • Quarterly configuration audit
   • Annual penetration testing

Low Priority:

7. Private VPC Endpoints for Border0

   • Direct connection to Border0 via AWS PrivateLink
   • Avoids internet routing
   • Cost: ~$7.20/month per endpoint

   Note: Check if Border0 supports AWS PrivateLink

8. VPC Flow Logs

   • Monitor all network traffic
   • Cost: ~$5-10/month

   Implementation:

   aws ec2 create-flow-logs \
     --resource-type VPC \
     --resource-ids vpc-<your-vpc-id> \
     --traffic-type ALL \
     --log-destination-type cloud-watch-logs \
     --log-group-name /aws/vpc/flowlogs

Incident Response

Suspected Compromise:

1. Immediate Actions:

   # Disable affected user in Border0
   Border0 Portal → Policies → Remove user email
   
   # Generate new connector token
   border0 connector token create --connector-id 7d5a5d82-d38d-4c0c-badd-842302c16a6b
   
   # Update connector
   sudo nano /etc/systemd/system/border0-connector.service
   sudo systemctl daemon-reload
   sudo systemctl restart border0-connector

2. Investigation:

   • Review Border0 session logs
   • Check AWS CloudTrail logs
   • Review system logs on affected servers
   • Identify scope of compromise

3. Recovery:

   • Rotate all credentials
   • Patch vulnerabilities
   • Restore from clean backup if necessary
   • Update security policies

Connection Issues:

1. Check connector status
2. Review recent changes
3. Test network connectivity
4. Verify configurations
5. Check Border0 service status page

Compliance and Auditing

Session Recording

What is recorded:

• All terminal input/output
• Connection metadata (user, time, duration)
• Source IP addresses
• Commands executed

Retention:

• Default: 90 days (configurable)
• Can be exported to S3 for long-term storage

Access:

• Border0 Portal → Sessions
• Filter by user, socket, date
• Playback capability

Audit Logs

Available logs:

• User authentication events
• Socket connection attempts
• Policy changes
• Configuration modifications
• Connector status changes

Export options:

• CSV download
• API access
• Webhook integration
• SIEM forwarding

Compliance Frameworks

Border0 supports compliance with:

• SOC 2 Type II
• ISO 27001
• GDPR
• HIPAA (with proper configuration)
• PCI DSS

Documentation for auditors:

1. Access control policies
2. Session recordings
3. Audit logs
4. Configuration history
5. Security architecture diagram

---

Maintenance Procedures

Regular Maintenance Tasks

Daily Checks (Automated)

Monitoring alerts should notify of:

• Connector offline
• Socket failures
• High connection failure rate
• Unusual access patterns

Manual check (if needed):

# Quick connector health check
border0 connector list

Should show "online" status.

Weekly Tasks

1. Review Failed Connections

Border0 Portal → Sessions → Filter: Failed

• Investigate any recurring failures
• Check for authentication issues
• Verify user access needs

2. Check Connector Logs

# SSH to bastion
border0 client ssh pixlr-prod-bastion

# Check for errors/warnings
sudo journalctl -u border0-connector -p warning -n 100 --no-pager

3. Verify Socket Status

Border0 Portal → Sockets

All sockets should show green "Alive" status.

Monthly Tasks

1. Access Review

• Review all users in Border0 policies
• Remove users who no longer need access
• Verify access levels are appropriate

2. Session Activity Audit

• Review session logs for anomalies
• Check for after-hours production access
• Verify no unauthorized access attempts

3. Update Check

# Check Border0 version
border0 version

# Compare with latest release
curl -s https://api.github.com/repos/borderzero/border0/releases/latest | grep tag_name

4. Security Group Review

• Verify no unnecessary inbound rules
• Confirm least privilege principle
• Check for 0.0.0.0/0 sources (should be minimal)

Quarterly Tasks

1. Rotate Connector Token

# Generate new token
border0 connector token create --connector-id 7d5a5d82-d38d-4c0c-badd-842302c16a6b

# Update service file
sudo nano /etc/systemd/system/border0-connector.service
# Replace BORDER0_TOKEN value

sudo systemctl daemon-reload
sudo systemctl restart border0-connector

2. Update Border0 Connector

# SSH to bastion
border0 client ssh pixlr-prod-bastion

# Download latest version
sudo curl -o /usr/local/bin/border0.new https://download.border0.com/linux_amd64/border0
sudo chmod +x /usr/local/bin/border0.new

# Test new version
/usr/local/bin/border0.new version

# If OK, replace old version
sudo systemctl stop border0-connector
sudo mv /usr/local/bin/border0.new /usr/local/bin/border0
sudo systemctl start border0-connector

# Verify
sudo systemctl status border0-connector

3. Review and Update Documentation

• Update this document with any changes
• Document new sockets created
• Note any configuration changes

4. Disaster Recovery Test

Simulate connector failure and verify:

• Monitoring alerts trigger
• Recovery procedure works
• Users are notified appropriately

Annual Tasks

1. Full Security Audit

• Review all configurations
• Verify compliance requirements
• Update security policies
• Consider penetration testing

2. Architecture Review

• Assess if current setup meets needs
• Plan for scaling or improvements
• Review cost optimization opportunities

3. Update Training Materials

• Refresh user documentation
• Update onboarding procedures
• Create new training videos if needed

Backup and Recovery

Configuration Backup

What to backup:

1. Border0 service file
2. SSH configurations on all servers
3. Border0 policy configurations
4. Security group rules
5. This documentation

Backup procedure:

# On bastion
sudo cp /etc/systemd/system/border0-connector.service ~/backups/
sudo cp /etc/ssh-ca.pub ~/backups/
sudo cp /etc/ssh/authorized_principals ~/backups/
sudo tar -czf ~/backups/border0-config-$(date +%Y%m%d).tar.gz ~/backups/*.service ~/backups/*.pub ~/backups/authorized_principals

Store securely:

• Encrypted S3 bucket
• Password manager (for small configs)
• Internal documentation system

Disaster Recovery Scenarios

Scenario 1: Bastion Instance Failure

Recovery steps:

1. Launch new EC2 instance:

   # Use same AMI, VPC, and security groups
   # Instance type: t3.small
   # Key pair: existing admin key

2. Install Border0:

   sudo curl -o /usr/local/bin/border0 https://download.border0.com/linux_amd64/border0
   sudo chmod +x /usr/local/bin/border0

3. Create service file:

   sudo nano /etc/systemd/system/border0-connector.service
   # Paste backed-up configuration

4. Start service:

   sudo systemctl daemon-reload
   sudo systemctl enable border0-connector
   sudo systemctl start border0-connector

5. Verify:

   sudo systemctl status border0-connector
   border0 connector list  # Should show online

Expected downtime: 10-15 minutes

Scenario 2: Connector Token Compromised

Recovery steps:

1. Generate new token:

   border0 connector token create --connector-id 7d5a5d82-d38d-4c0c-badd-842302c16a6b

2. Update all systems using the token:

   # On bastion
   sudo nano /etc/systemd/system/border0-connector.service
   # Update BORDER0_TOKEN
   
   sudo systemctl daemon-reload
   sudo systemctl restart border0-connector

3. Revoke old token:

   • Border0 Portal → Connectors → Tokens → Revoke old token

4. Audit access:

   • Review recent session logs
   • Check for unauthorized access
   • Investigate suspicious activity

Expected downtime: 1-2 minutes

Scenario 3: Target Server Failure

Recovery steps:

1. Launch replacement instance (or restore from snapshot)

2. Configure Border0 certificate auth:

   sudo tee /etc/ssh-ca.pub > /dev/null << 'EOF'
   ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNo...= Border0 ssh CA for Pixlr Group
   EOF
   
   sudo sh -c 'echo "TrustedUserCAKeys /etc/ssh-ca.pub" >> /etc/ssh/sshd_config'
   sudo sh -c 'echo "AuthorizedPrincipalsFile /etc/ssh/authorized_principals" >> /etc/ssh/sshd_config'
   sudo sh -c 'echo "border0_ssh_signed" > /etc/ssh/authorized_principals'
   
   sudo systemctl reload ssh

3. Update Border0 socket (if IP changed):

   • Border0 Portal → Sockets → [Socket] → Edit
   • Update hostname to new IP
   • Save

4. Test connection

Expected downtime: Depends on replacement speed (5-30 minutes)

Scaling Considerations

Adding More Connectors

When to add:

• High latency to current connector
• Geographic distribution of users
• Redundancy requirements
• Load distribution

Process:

1. Launch new EC2 instance in target region/AZ

2. Create new connector in Border0:

   border0 connector create \
     --name pixlr-prod-region2 \
     --description "Connector in us-west-2"

3. Install and configure: (Same as initial setup)

4. Assign sockets to new connector:

   • Edit existing sockets
   • Add new connector to connector list
   • Border0 will load-balance automatically

Adding More Sockets

Planning:

• Identify which services need Border0 access
• Determine access requirements (who needs access)
• Plan security configurations

Implementation:

• Follow "Creating New Sockets" procedure in User Guide
• Test thoroughly before granting wide access
• Update documentation

---

Lessons Learned

Technical Insights

1. Border0 Command Syntax Matters

Problem: Using connector run instead of connector start

Learning: Border0 CLI commands must be exact. The difference between run and start:

• run: Interactive/foreground mode (for testing)
• start: Daemon mode (for production services)

Best Practice: Always verify commands in Border0 documentation before creating systemd services.

2. Token Persistence is Critical

Problem: User login tokens don't persist for service use

Learning: Border0 has two authentication mechanisms:

• User tokens: For management operations (login, create resources)
• Connector tokens: For service/daemon operation

Best Practice: Always generate connector-specific tokens for systemd services. Store in environment variables within the service file.

3. SSH Certificate Authentication is Superior

Problem: EC2 Instance Connect requires IAM permissions and complex setup

Learning: SSH certificate authentication via Border0:

• No AWS IAM dependencies
• Centralized certificate management
• Automatic rotation
• Simpler configuration

Best Practice: Use Border0 certificate authentication as default. Only use AWS-native methods when specifically required.

4. Temporary SSH Key Method for Initial Setup

Problem: Need to configure servers without existing access

Learning: AWS EC2 Instance Connect's send-ssh-public-key can push temporary keys (60-second validity) even to private instances.

Best Practice: Use EC2 Instance Connect for initial bootstrap, then configure permanent certificate-based auth immediately after.

5. systemd Service Configuration Pitfalls

Problem: Service file modifications not taking effect

Learning: Multiple issues encountered:

• Text editors with permission issues
• Need for daemon-reload after changes
• Quote escaping in bash commands

Best Practice:

• Use sudo nano for direct editing
• Always run systemctl daemon-reload after changes
• Test with systemctl status before assuming it works
• Check logs with journalctl

Process Improvements

1. Documentation is Essential

Lesson: Without this detailed documentation, reproducing this setup would require another 4-6 hours of troubleshooting.

Action: Maintain living documentation that includes:

• Every configuration step
• All error messages encountered
• Solutions that worked and didn't work
• Command reference

2. Test in Stages

Lesson: Testing each component (connector, bastion socket, target sockets) separately made troubleshooting manageable.

Action: Always:

1. Get connector online first
2. Test with bastion socket
3. Only then configure additional sockets
4. Test each socket individually

3. Understand AWS Networking

Lesson: Understanding VPC, subnets, security groups, and EC2 connectivity options was crucial for troubleshooting.

Action: Maintain network diagrams and document:

• Which instances are in which subnets
• Security group rules
• Network paths between components

4. Have Multiple Access Paths

Lesson: Having AWS SSM access when direct SSH failed was invaluable.

Action: Always maintain at least two ways to access critical infrastructure:

• Border0 (primary)
• AWS SSM (backup)
• Emergency SSH key (last resort)

Appendix

A. Command Reference

Border0 CLI Commands

# Authentication
border0 login
border0 logout
border0 whoami

# Connector Management
border0 connector list
border0 connector show <connector-id>
border0 connector token create --connector-id <id>
border0 connector token list --connector-id <id>
border0 connector token revoke --token-id <id>

# Socket Management
border0 socket ls
border0 socket show <socket-name>
border0 socket create ssh --name <name> --connector <connector> --host <ip> --port <port> --username <user>
border0 socket update <socket-name> [options]
border0 socket delete <socket-name>

# Client Connection
border0 client ssh <socket-name>
border0 client ssh <socket-name> -A  # with agent forwarding
border0 client tcp --host <socket-name> --listener <local-port>

# Policy Management
border0 policy list
border0 policy show <policy-name>
border0 policy create [options]
border0 policy update <policy-name> [options]
border0 policy delete <policy-name>

Linux System Commands

# systemd Service Management
sudo systemctl start <service>
sudo systemctl stop <service>
sudo systemctl restart <service>
sudo systemctl reload <service>
sudo systemctl status <service>
sudo systemctl enable <service>
sudo systemctl disable <service>
sudo systemctl daemon-reload