8000 fix: harden install cleanup and agent startup by angelorc · Pull Request #40 · angelorc/vmsan · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .changeset/late-pianos-help.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
"vmsan": patch
---

Harden installer and VM networking reliability across mixed Linux hosts.

- fix branch/commit installs and uninstalls in `install.sh`, including safer cleanup of per-VM iptables rules
- migrate the default VM subnet to `198.19.x.x` while preserving compatibility with legacy persisted `172.16.x.x` states
- keep stopped VM slots reserved, tighten persisted IP parsing, and restore agent connectivity on hosts with restrictive local firewalls
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ docs/ Documentation site (vmsan.dev)
### How it works

1. **vmsan** uses [Firecracker](https://github.com/firecracker-microvm/firecracker) to create lightweight microVMs with a jailer for security isolation
2. Each VM gets a TAP network device with its own `/30` subnet (`172.16.{slot}.0/30`)
2. Each VM gets a TAP network device with its own `/30` subnet (`198.19.{slot}.0/30`)
3. A Go-based **agent** runs inside the VM, exposing an HTTP API for command execution, file operations, and shell access
4. The CLI communicates with the agent over the host-guest network

Expand Down
2 changes: 1 addition & 1 deletion docs/content/2.guide/2.networking.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ navigation:
icon: i-lucide-shield
---

Each vmsan microVM gets its own TAP network device with a `/30` subnet (`172.16.{slot}.0/30`). You can control outbound network access with policies.
Each vmsan microVM gets its own TAP network device with a `/30` subnet (`198.19.{slot}.0/30`). You can control outbound network access with policies.

## Set a network policy at creation

Expand Down
2 changes: 1 addition & 1 deletion docs/content/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ description: Powerful isolation primitives wrapped in an intuitive developer exp
vmsan list --json
[{"id":"vm-a3f7b2c","status":"running",
"runtime":"node22","memory":128,
"cpus":1,"ip":"172.16.1.2",
"cpus":1,"ip":"198.19.1.2",
"networkPolicy":"allow-all"}]
```
::::
Expand Down
143 changes: 127 additions & 16 deletions install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ set -euo pipefail
# Install a commit: curl -fsSL https://vmsan.dev/install | bash -s -- --sha <commit>
# Uninstall: curl -fsSL https://vmsan.dev/install | bash -s -- --uninstall

VMSAN_DIR="${VMSAN_DIR:-$HOME/.vmsan}"
VMSAN_DIR="${VMSAN_DIR:-}"
VMSAN_REPO="angelorc/vmsan"
VMSAN_INSTALL_BOOTSTRAPPED="${VMSAN_INSTALL_BOOTSTRAPPED:-}"
VMSAN_INSTALL_SOURCE_SHA="${VMSAN_INSTALL_SOURCE_SHA:-}"
Expand All @@ -29,6 +29,24 @@ success() { printf "\033[1;32m[ok]\033[0m %s\n" "$*"; }
warn() { printf "\033[1;33m[warn]\033[0m %s\n" "$*"; }
error() { printf "\033[1;31m[error]\033[0m %s\n" "$*" >&2; exit 1; }

resolve_vmsan_dir() {
if [ -n "$VMSAN_DIR" ]; then
printf '%s\n' "$VMSAN_DIR"
return
fi

if [ -n "${SUDO_USER:-}" ] && command -v getent >/dev/null 2>&1; then
local sudo_home
sudo_home="$(getent passwd "$SUDO_USER" 2>/dev/null | cut -d: -f6)"
if [ -n "$sudo_home" ]; then
printf '%s/.vmsan\n' "$sudo_home"
return
fi
fi

printf '%s/.vmsan\n' "$HOME"
}

print_usage() {
cat <<EOF
Usage:
Expand All @@ -43,6 +61,96 @@ need_cmd() {
command -v "$1" >/dev/null 2>&1 || error "Required command not found: $1"
}

default_iface() {
ip route show default 2>/dev/null | awk '/default/ {print $5; exit}'
}
Comment on lines +64 to +66
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Parse the dev token instead of assuming field 5.

ip route show default can also render as default dev <iface> .... In that form, $5 is empty and the uninstall path silently skips MASQUERADE/DNAT cleanup.

💡 Suggested change
 default_iface() {
-  ip route show default 2>/dev/null | awk '/default/ {print $5; exit}'
+  ip route show default 2>/dev/null | awk '{
+    for (i = 1; i < NF; i++) {
+      if ($i == "dev") {
+        print $(i + 1)
+        exit
+      }
+    }
+  }'
 }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
default_iface() {
ip route show default 2>/dev/null | awk '/default/ {print $5; exit}'
}
default_iface() {
ip route show default 2>/dev/null | awk '{
for (i = 1; i < NF; i++) {
if ($i == "dev") {
print $(i + 1)
exit
}
}
}'
}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@install.sh` around lines 64 - 66, The default_iface() function assumes the
interface is field 5 from `ip route show default`, which fails when the output
uses the "dev <iface>" form; update default_iface to parse the token "dev" and
return the next field (and fall back to the existing awk pattern if necessary)
so the uninstall logic that removes MASQUERADE/DNAT rules uses the correct
interface name; locate the default_iface function and change its awk/shell
parsing to search for "dev" and emit the following token as the interface,
preserving the original behavior as a fallback.


json_string_field() {
local key="$1" file="$2"
grep -oP "\"$key\"\\s*:\\s*\"\\K[^\"]+" "$file" 2>/dev/null | head -n1 || true
}

json_bool_field() {
local key="$1" file="$2"
grep -oP "\"$key\"\\s*:\\s*\\K(true|false)" "$file" 2>/dev/null | head -n1 || true
}

json_ports_field() {
local file="$1"
sed -n '/"publishedPorts"\s*:/,/\]/p' "$file" 2>/dev/null | grep -oP '\d+' || true
}

cidr30_from_ip() {
local ip="$1"
IFS=. read -r a b c _ <<EOF
$ip
EOF
[ -n "${a:-}" ] && [ -n "${b:-}" ] && [ -n "${c:-}" ] || return 1
printf '%s.%s.%s.0/30\n' "$a" "$b" "$c"
}

iptables_delete_rule() {
iptables "$@" 2>/dev/null || true
}

delete_iptables_rule_spec() {
local rule="$1"
local -a rule_parts
read -r -a rule_parts <<<"$rule"
[ "${#rule_parts[@]}" -gt 0 ] || return 0
[ "${rule_parts[0]}" = "-A" ] || return 0
rule_parts[0]="-D"
iptables_delete_rule "${rule_parts[@]}"
}

cleanup_iptables_rules_for_vm() {
local state_file="$1" default_iface_name="$2"
local tap_device guest_ip skip_dnat slot veth_host guest_cidr

tap_device="$(json_string_field tapDevice "$state_file")"
guest_ip="$(json_string_field guestIp "$state_file")"
skip_dnat="$(json_bool_field skipDnat "$state_file")"

[ -n "$guest_ip" ] || return
guest_cidr="$(cidr30_from_ip "$guest_ip" || true)"

if [ -n "$default_iface_name" ] && [ -n "$guest_cidr" ]; then
iptables_delete_rule -t nat -D POSTROUTING -s "$guest_cidr" -o "$default_iface_name" -j MASQUERADE
iptables_delete_rule -D OUTPUT -d "$guest_ip" -j ACCEPT
iptables_delete_rule -D INPUT -s "$guest_ip" -j ACCEPT

if [ "$skip_dnat" != "true" ]; then
while IFS= read -r port; do
[ -n "$port" ] || continue
iptables_delete_rule -t nat -D PREROUTING -i "$default_iface_name" -p tcp --dport "$port" -j DNAT --to-destination "${guest_ip}:${port}"
iptables_delete_rule -D FORWARD -p tcp -d "$guest_ip" --dport "$port" -j ACCEPT
done < <(json_ports_field "$state_file")
fi
fi
Comment on lines +117 to +129
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Don’t gate OUTPUT/INPUT cleanup on default-route discovery.

The guest-specific -D OUTPUT -d "$guest_ip" and -D INPUT -s "$guest_ip" rules do not depend on $default_iface_name or $guest_cidr. If default-interface lookup fails, uninstall currently leaves those vmsan-owned ACCEPT rules behind.

💡 Suggested change
+  iptables_delete_rule -D OUTPUT -d "$guest_ip" -j ACCEPT
+  iptables_delete_rule -D INPUT -s "$guest_ip" -j ACCEPT
+
   if [ -n "$default_iface_name" ] && [ -n "$guest_cidr" ]; then
     iptables_delete_rule -t nat -D POSTROUTING -s "$guest_cidr" -o "$default_iface_name" -j MASQUERADE
-    iptables_delete_rule -D OUTPUT -d "$guest_ip" -j ACCEPT
-    iptables_delete_rule -D INPUT -s "$guest_ip" -j ACCEPT
 
     if [ "$skip_dnat" != "true" ]; then
       while IFS= read -r port; do
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if [ -n "$default_iface_name" ] && [ -n "$guest_cidr" ]; then
iptables_delete_rule -t nat -D POSTROUTING -s "$guest_cidr" -o "$default_iface_name" -j MASQUERADE
iptables_delete_rule -D OUTPUT -d "$guest_ip" -j ACCEPT
iptables_delete_rule -D INPUT -s "$guest_ip" -j ACCEPT
if [ "$skip_dnat" != "true" ]; then
while IFS= read -r port; do
[ -n "$port" ] || continue
iptables_delete_rule -t nat -D PREROUTING -i "$default_iface_name" -p tcp --dport "$port" -j DNAT --to-destination "${guest_ip}:${port}"
iptables_delete_rule -D FORWARD -p tcp -d "$guest_ip" --dport "$port" -j ACCEPT
done < <(json_ports_field "$state_file")
fi
fi
iptables_delete_rule -D OUTPUT -d "$guest_ip" -j ACCEPT
iptables_delete_rule -D INPUT -s "$guest_ip" -j ACCEPT
if [ -n "$default_iface_name" ] && [ -n "$guest_cidr" ]; then
iptables_delete_rule -t nat -D POSTROUTING -s "$guest_cidr" -o "$default_iface_name" -j MASQUERADE
if [ "$skip_dnat" != "true" ]; then
while IFS= read -r port; do
[ -n "$port" ] || continue
iptables_delete_rule -t nat -D PREROUTING -i "$default_iface_name" -p tcp --dport "$port" -j DNAT --to-destination "${guest_ip}:${port}"
iptables_delete_rule -D FORWARD -p tcp -d "$guest_ip" --dport "$port" -j ACCEPT
done < <(json_ports_field "$state_file")
fi
fi
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@install.sh` around lines 107 - 119, The cleanup currently requires both
$default_iface_name and $guest_cidr before removing guest-specific ACCEPT rules,
which leaves the OUTPUT/INPUT rules for $guest_ip behind if default-route
discovery fails; change the logic in the teardown so that iptables_delete_rule
-D OUTPUT -d "$guest_ip" -j ACCEPT and iptables_delete_rule -D INPUT -s
"$guest_ip" -j ACCEPT are executed unconditionally (or at least independent of
$default_iface_name/$guest_cidr), while keeping the NAT/PREROUTING and FORWARD
port cleanup (the loop reading json_ports_field "$state_file" and the
iptables_delete_rule -t nat ... PREROUTING / -D FORWARD ... rules) gated by the
existing $default_iface_name/$guest_cidr check and the skip_dnat flag; locate
these calls around the iptables_delete_rule invocations and adjust their
placement so guest_ip ACCEPT rule deletions run even when default_iface_name or
guest_cidr are empty.


slot="$(printf '%s\n' "$tap_device" | sed -n 's/^fhvm//p')"
if [ -n "$default_iface_name" ] && [ -n "$slot" ] && [ -n "$guest_cidr" ]; then
veth_host="veth-h-$slot"
iptables_delete_rule -D FORWARD -i "$veth_host" -o "$default_iface_name" -s "$guest_cidr" -j ACCEPT
iptables_delete_rule -D FORWARD -i "$default_iface_name" -o "$veth_host" -d "$guest_cidr" -m state --state RELATED,ESTABLISHED -j ACCEPT
fi

if [ -n "$tap_device" ]; then
iptables -S 2>/dev/null | grep -E "(^-A .* -i ${tap_device}( |$)|^-A .* -o ${tap_device}( |$))" | while IFS= read -r rule; do
delete_iptables_rule_spec "$rule"
done || true
fi

if [ -n "$slot" ]; then
veth_host="veth-h-$slot"
iptables -S 2>/dev/null | grep -E "(^-A .* -i ${veth_host}( |$)|^-A .* -o ${veth_host}( |$))" | while IFS= read -r rule; do
delete_iptables_rule_spec "$rule"
done || true
fi
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
}

VMSAN_DIR="$(resolve_vmsan_dir)"

download() {
local url="$1" dest="$2"
info "Downloading $(basename "$dest")..."
Expand Down Expand Up @@ -82,7 +190,6 @@ bootstrap_ref_installer() {
shift
local script_tmp
script_tmp="$(mktemp)"
trap 'rm -f "$script_tmp"' EXIT

download "https://raw.githubusercontent.com/$VMSAN_REPO/$resolved_sha/install.sh" "$script_tmp"
chmod +x "$script_tmp"
Expand All @@ -94,7 +201,10 @@ bootstrap_ref_installer() {
"VMSAN_INSTALL_REQUESTED_REF=$REQUESTED_REF"
"VMSAN_INSTALL_REQUESTED_SHA=$REQUESTED_SHA"
)
env "${env_args[@]}" bash "$script_tmp" "$@"

exec 3<"$script_tmp"
rm -f -- "$script_tmp"
exec env "${env_args[@]}" bash -s -- "$@" <&3
}

while [ $# -gt 0 ]; do
Expand Down Expand Up @@ -191,6 +301,19 @@ if [ "$UNINSTALL" -eq 1 ]; then
success "vmsan CLI not installed"
fi

# Clean up iptables rules owned by vmsan using per-VM state instead of broad 172.16.* matching.
DEFAULT_IFACE="$(default_iface || true)"
IPTABLES_CLEANED=0
if [ -d "$VMSAN_DIR/vms" ]; then
for state_file in "$VMSAN_DIR"/vms/*.json; do
[ -f "$state_file" ] || continue
[ "$IPTABLES_CLEANED" -eq 0 ] && info "Cleaning vmsan iptables rules..."
cleanup_iptables_rules_for_vm "$state_file" "$DEFAULT_IFACE"
IPTABLES_CLEANED=1
done
fi
[ "$IPTABLES_CLEANED" -eq 1 ] && success "iptables rules cleaned"

# Clean up TAP and veth interfaces (host namespace)
NET_COUNT=0
for iface_path in /sys/class/net/fhvm* /sys/class/net/veth-h-* /sys/class/net/veth-g-*; do
Expand All @@ -200,24 +323,12 @@ if [ "$UNINSTALL" -eq 1 ]; then
done

# Clean up network namespaces created by vmsan
for ns in $(ip netns list 2>/dev/null | awk '{print $1}' | grep '^vmsan-'); do
for ns in $(ip netns list 2>/dev/null | awk '{print $1}' | grep '^vmsan-' || true); do
ip netns delete "$ns" 2>/dev/null && NET_COUNT=$((NET_COUNT + 1))
done

[ "$NET_COUNT" -gt 0 ] && success "Cleaned up $NET_COUNT network resources"

# Clean up iptables rules referencing vmsan
iptables-save 2>/dev/null | grep -cE 'fhvm|172\.16\.' >/dev/null 2>&1 && {
info "Cleaning iptables rules..."
iptables -t nat -S 2>/dev/null | grep -E 'fhvm|172\.16\.' | while IFS= read -r rule; do
eval "iptables -t nat $(echo "$rule" | sed 's/^-A/-D/')" 2>/dev/null || true
done
iptables -S FORWARD 2>/dev/null | grep -E 'fhvm|172\.16\.' | while IFS= read -r rule; do
eval "iptables $(echo "$rule" | sed 's/^-A/-D/')" 2>/dev/null || true
done
success "iptables rules cleaned"
}

# Delete Cloudflare tunnel via API before removing local files
if [ -f "$VMSAN_DIR/cloudflare/cloudflare.json" ]; then
CF_TOKEN="$(grep -oP '"token"\s*:\s*"\K[^"]+' "$VMSAN_DIR/cloudflare/cloudflare.json" 2>/dev/null || true)"
Expand Down
7 changes: 6 additions & 1 deletion src/lib/jailer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,7 @@ export class Jailer {
// Mount rootfs to configure DNS and inject services
const tmpMount = join(paths.rootDir, "tmp-mount");
mkdirSync(tmpMount, { recursive: true });
let prepareError: unknown;
try {
execSync(`sudo mount -o loop "${paths.rootfsPath}" "${tmpMount}"`, {
stdio: "pipe",
Expand Down Expand Up @@ -200,7 +201,8 @@ export class Jailer {
}

execSync(`sudo umount "${tmpMount}"`, { stdio: "pipe" });
} catch {
} catch (error) {
prepareError = error;
// Mount point may already be unmounted or directory removed
try {
execSync(`sudo umount "${tmpMount}" 2>/dev/null`, { stdio: "pipe" });
Expand All @@ -213,6 +215,9 @@ export class Jailer {
} catch {
// Temp mount directory may already be removed
}
if (prepareError) {
throw prepareError;
}

// If restoring from snapshot, copy snapshot files into chroot
if (config.snapshot) {
Expand Down
76 changes: 76 additions & 0 deletions src/lib/network-address.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
const VM_NETWORK_FIRST_OCTET = 198;
const VM_NETWORK_SECOND_OCTET = 19;

export const VM_SUBNET_MASK: string = "255.255.255.252";
export const VM_NETWORK_PREFIX: string = `${VM_NETWORK_FIRST_OCTET}.${VM_NETWORK_SECOND_OCTET}`;
export const SUPPORTED_VM_ADDRESS_BLOCKS: readonly string[] = ["198.19.0.0/16", "172.16.0.0/16"];

function assertValidSlot(slot: number): void {
if (!Number.isInteger(slot) || slot < 0 || slot > 254) {
throw new Error(`invalid VM network slot: ${slot}`);
}
}

function parseIpv4OrNull(ip: string): [number, number, number, number] | null {
const octets = ip.split(".");
if (octets.length !== 4 || octets.some((part) => !/^\d+$/.test(part))) {
return null;
}

const parts = octets.map((part) => Number.parseInt(part, 10));
if (parts.some((part) => !Number.isInteger(part) || part < 0 || part > 255)) {
return null;
}

return parts as [number, number, number, number];
}

function parseIpv4(ip: string): [number, number, number, number] {
const parts = parseIpv4OrNull(ip);
if (!parts) {
throw new Error(`invalid IPv4 address: ${ip}`);
}
return parts;
}

export function vmHostIp(slot: number): string {
assertValidSlot(slot);
return `${VM_NETWORK_PREFIX}.${slot}.1`;
}

export function vmGuestIp(slot: number): string {
assertValidSlot(slot);
return `${VM_NETWORK_PREFIX}.${slot}.2`;
}

export function slotFromVmHostIpOrNull(hostIp: string): number | null {
const parts = parseIpv4OrNull(hostIp);
if (!parts) {
return null;
}

const slot = parts[2];
if (!Number.isInteger(slot) || slot < 0 || slot > 254) {
return null;
}

return slot;
}

export function slotFromVmHostIp(hostIp: string): number {
const slot = slotFromVmHostIpOrNull(hostIp);
if (slot === null) {
throw new Error(`invalid VM host IP: ${hostIp}`);
}
return slot;
}

export function vmLinkCidrFromIp(ip: string): string {
const [first, second, third] = parseIpv4(ip);
return `${first}.${second}.${third}.0/30`;
}

export function vmAddressBlockCidrFromIp(ip: string): string {
const [first, second] = parseIpv4(ip);
return `${first}.${second}.0.0/16`;
}
Loading
0