Benjamin Nied
Site Reliability Engineer
This site is an interactive terminal and needs JavaScript. What follows is the same resume as plain text.
ABOUT
Site Reliability Engineer with nearly two decades of professional experience building and operating infrastructure at scale, specializing in distributed databases and Kubernetes automation.
Experienced. Independent. Results-oriented.
Certified Kubernetes Administrator (CKA), 2020
CONTACT
email bnied@spaceduck.org
sourcetube source.tube/spaceduck
github github.com/bnied
linkedin linkedin.com/in/bnied
SKILLS
Distributed Databases
- Cassandra at fleet scale: upgrades, schema and replication changes, performance tuning, incident response
- PostgreSQL as a managed service: major-version rollouts and fleet-wide patching
- Backup and restore design, including self-service restores
Kubernetes & Containers
- Self-healing Kubernetes automation in Go
- Cluster provisioning from scratch, bare metal -> Kubernetes migrations
- Helm charts, operators, Kubespray; Docker Swarm -> Kubernetes, zero-downtime upgrades
- Maintaining and repairing clusters for full lifetime
Observability
- OpenTelemetry, metrics pipelines, SLO reporting, alerting, PagerDuty
Configuration Management
Operating Systems
- RHEL, Oracle Linux, CentOS, Fedora, Ubuntu
- macOS, FreeBSD, OpenBSD
Languages
- Go, Java (JDK 8 -> 25), Python, Rust, Ruby
- C, C++, Objective-C, Kotlin, Swift
Hardware & Infrastructure
- Assembly, configuration, diagnosis, migration, repair, replacement
- Extensive experience in systems design, development, deployment, and operations at scale
EXPERIENCE
APPLE 2018 - Present
SRE // ASE Cassandra 2021 - Present
SRE // ACI Postgres 2020 - 2021
SRE // ACI Observability 2018 - 2020
LINKEDIN, INC 2016 - 2018
Senior Site Reliability Engineer
WORK MARKET, INC 2015 - 2016
Senior DevOps Engineer
SHUTTERSTOCK, INC 2013 - 2015
Site Reliability Engineer
DATAPIPE, INC 2008 - 2013
Datacenter Technician / Operational Support Engineer
For details, run: experience <role> e.g. 'experience apple-cassandra' available roles: apple-cassandra apple-postgres apple-observability linkedin workmarket shutterstock datapipe
APPLE // ASE Cassandra
Site Reliability Engineer 2021 - Present
- Owned the group's Slackbot, growing it from a skunkworks project into the team's primary ChatOps platform for operating Cassandra fleets: peer-approved cluster operations, change-freeze enforcement, on-call paging, cluster visualizations, and a full web UI, with ~300 projects and fixes to date
- Drove remediation of a comprehensive automated security review of that platform, moved it to single sign-on, and kept it current from Java 11 through 25 with test-coverage gates on every change
- Designed & built self-healing Kubernetes automation in Go that replaces failed database pods and moves them off unhealthy nodes on its own, paced so a cluster never loses replicas all at once
- Frontline production support across customer clusters: incident response through post-incident reviews, latency investigations, credential lifecycle, and data recovery; wrote the runbooks for the team's major alerts
- Diagnosed platform-level Kubernetes issues affecting stateful workloads on shared clusters: CNI/IPAM bugs, storage-provisioner gaps, and stateful-controller reliability
- Planned & ran fleet-wide changes across dozens of production clusters: replication changes, Cassandra upgrades, and safer default settings
- Headed the project to migrate the entire fleet to a new monitoring solution with better dashboarding, flexible querying, and alerting; later evaluated a second metrics-platform move, and automated the monthly SLO reporting
- Designed self-service restores so customer teams can recover their own data from backups without waiting on an SRE
- Ran proof-of-concept and QA cycles for the team's Kubernetes-hosted Cassandra offering, and helped design its Kubernetes-native control plane
- Built the team-wide documentation site end to end and SRE courses on Kubernetes, observability, and SLOs; ran a monthly SRE knowledge forum, presented at org all-hands meetings, and mentored and onboarded new team members
APPLE // ACI Postgres
Site Reliability Engineer 2020 - 2021
- Planned & executed migration of all 2000+ customer PostgreSQL instances to a newer major version ahead of end-of-life, coordinating with many other teams to ensure the rollout landed safely and completely
- Led the build and release of a newer PostgreSQL major version, from build pipelines through production enablement
- Retired legacy PostgreSQL versions, added new instance sizes, and wrote the customer-facing docs and setup guides
- Added failure-rate and latency alerting across the Postgres control plane: API, monitoring, and dashboard services
- Migrated the team's production automation stack onto Kubernetes and enhanced automated bad-node handling
- Devised, wrote, and presented a stability-validation procedure letting services prove their stability to the organization and bypass internal freeze directives during critical periods
- Devised, piloted, and implemented team-level systems ensuring vital functions stayed covered in a way that was fair to the whole team, with visibility and progress properly communicated
- Ensured ongoing consistency of prioritization and current state between the team and management
- Improved team documentation, runbooks, procedures, processes, and on-call load
- Inherited, maintained, and led an org-wide initiative improving learning resources for SREs joining Apple Cloud Services (ACS), spanning multiple teams and including SREs, developers, and cloud advocates from many parts of ACS: built onboarding courses on Kubernetes fundamentals, intro to SRE, and observability, restructured into 30-60 minute modules
APPLE // ACI Observability
Site Reliability Engineer 2018 - 2020
- Technical lead for ACI SRE Observability: helped set technical direction for the SRE teams around improved operation and servicing of existing platforms, while developing migration plans to move customers onto new platforms
- Mentored junior team members on coding and code-release practices, enabling them to write the automation that completed key projects while raising the overall skill level of the team
- Built, maintained, and serviced a series of self-hosted Kubernetes clusters across production, QA, and test, including upgrades, rebuilds, and traffic cutovers
- Ran the Ceph storage layer for those clusters, including failure planning and multi-datacenter setup
- Planned & executed migration of the service frontend from Docker Swarm to self-hosted Kubernetes alongside other senior team members: 20+ applications in several languages, along with several data backends
- Created, maintained, and deployed the Helm charts used to migrate applications into Kubernetes
- Wrote runbooks and docs on all aspects of Kubernetes ownership: cluster creation, cluster migration, and repairing parts of a cluster without downtime or performance impact
- Maintained a fleet of Kubernetes clusters for the team's use, including maintenances, repair, and zero-downtime upgrades
- Stood up core services of the next-generation telemetry platform on Kubernetes (alerting rule engine, downsampling, workspace services) and authored its deployment architecture proposal
- Headed an initiative to get SREs involved in and educated on the next-generation telemetry offering
- Built canary release tooling and backup coverage for stateful services as part of the Kubernetes migration
- Peer-reviewed code for team members to ensure solutions met requirements without introducing additional issues
- Created & maintained an initiative to reduce operational toil by streamlining alerts, removing unnecessary ones, and keeping a clear view into the state of the observability system
- Supported the team as an educational resource after departure
LINKEDIN, INC
Senior Site Reliability Engineer 2016 - 2018
- Led a team of engineers building host-level chaos engineering as part of LinkedIn's Waterbear initiative, measurably improving infrastructure reliability and availability
- Wrote the code enabling CPU-level and network-level chaos engineering for Waterbear
- Presented the host-level chaos initiative at SaltConf17
- Led the weekly design review meeting in the New York office, letting developers present plans to a smaller audience before opening RFCs to wider company-wide scrutiny
- Contributed code to the company monitoring system to enable monitoring on new services and adjust it on existing ones
- Created solutions for developer teams to more effectively monitor the health of their services at a glance
- Created the frontend and API for a company-wide log aggregation and searching service
- Peer-reviewed code to ensure developer strategies made sense and didn't detract from the reliability of their services
WORK MARKET, INC
Senior DevOps Engineer 2015 - 2016
- Contributed manifests and modules to the Puppet configuration management tree to facilitate deployment of new technologies and services
- Contributed code to the Fabric tree for administration tasks and code deployments
- Created solutions to facilitate easy provisioning and configuration of new EC2-backed microservice deployments
- Assisted in automating RDS provisioning and configuration to allow rapid deployment of new services
- Streamlined AMI creation into a single Packer pipeline built on CIS hardening guidelines
- Created a system patching strategy allowing the entire infrastructure to be patched and rebooted with zero downtime
- Created and documented production maintenance procedures for better notification, communication, and documentation
- Assisted in documenting procedures for rapidly creating from-scratch Puppet modules for new microservice deployments
- Added Puppet manifests to automatically install and configure collectd on all servers for system metrics collection
- Added Puppet code configuring collectd to collect MySQL metrics for diagnosing and troubleshooting MySQL problems
- Started migrating configuration management from the aging Puppet codebase to a newer, streamlined Saltstack codebase
SHUTTERSTOCK, INC
Site Reliability Engineer 2013 - 2015
- Contributed manifests and modules to the Puppet configuration management tree to facilitate deployment of new technologies and services
- Architected the solution that let Puppet configuration management scale beyond a single master node
- Helped spearhead the Chef-Solo migration, improving CM scale by moving to a master-less, decentralized infrastructure
- Ported Chef cookbooks to Ubuntu 14.04 and CentOS 7 for eventual use in place of CentOS 6
- Improved automation for creating and removing DNS records
- Improved automation for managing LDAP accounts
- Created tools to notify users of on-call assignments
- Maintained the Linux provisioning system; created automation for much faster provisioning across larger numbers of servers
- Maintained and improved the CMDB and project deployment systems
- Maintained and improved Jenkins CI; wrote a dashboard to quickly separate good/bad/unbuilt builds into groups and graph them
- Migrated services from the older build system to the new Jenkins-based build/deploy system
DATAPIPE, INC
Datacenter Technician / Operational Support 2008 - 2013
- Designed, implemented, and maintained the new UNIX provisioning system that replaced the company's previous solution
- Tested Linux distributions for hardware compatibility issues and resolved any issues surfaced in testing
- Developed custom client images tailored to specific needs
- Designed and built a CD/DVD burning solution letting datacenter personnel create installation discs remotely
- Implemented and maintained a new monitoring system complementing the company's existing system
- Designed supplemental applications for the new monitoring system; wrote integration scripts generating incidents in the new ticketing system
- Implemented and maintained the new ticketing/CMDB system, expanding its capacity and capabilities as needed
- Designed and implemented custom maintenance scripts to facilitate system administration functions
- Maintained servers for the Development team to ensure stability and uptime
- Designed, implemented, and maintained a system tracking all server builds while in progress
- Designed, implemented, and maintained a system tracking pulled servers and notifying personnel when drive grace periods expired
- Designed, implemented, and maintained a system tracking inventory through the company's RMA/QA process
- Assembled server and networking hardware per customer orders
- Installed Windows and UNIX OSes on customer servers via PXE, including post-installation steps
- Racked/cabled servers and networking hardware, ran power cables, and set VLANs on networking switches
- Tested Windows and UNIX deployment systems
- Tested hardware for operating system compatibility
PROJECTS
neofsn
source.tube/spaceduck/neofsn
A spatial filesystem navigator for macOS - a modern homage to SGI's FSN, the 3D UNIX interface from Jurassic Park. Built with SwiftUI and SceneKit.
kfsn
source.tube/spaceduck/kfsn
Qt/KDE port of neofsn, built with Qt Quick 3D and KDE Frameworks 6.
BSPRenderer
source.tube/spaceduck/BSPRenderer
A DOOM-style software renderer ported to six languages: Swift, C++, Go, Python, Rust, and Zig.
ls-rust
source.tube/spaceduck/ls-rust
An implementation of ls written in Rust.
kernel-ml-aufs
github.com/bnied/kernel-ml-aufs
Mainline Linux kernel packages for RHEL/CentOS/OEL 7 & 8 with AUFS storage-driver support for Docker.
kernel-lt-aufs
github.com/bnied/kernel-lt-aufs
Longterm support kernel packages for RHEL/CentOS 6 & 7 with AUFS support for Docker.
CONTRIBUTIONS
optopus
github.com/optopus/optopus
CMDB for tracking servers and networking equipment.
optopus-ldap-admin
github.com/optopus/optopus-ldap-admin
Optopus plugin for LDAP administration.
zfs_motd
github.com/spikegrobstein/zfs_motd
Ubuntu-style MOTD generator with ZFS filesystem metrics.
EDUCATION
Fairleigh Dickinson University 2005 - 2007
B.S. in Computer Science
Coursework: high-level mathematics, programming (C++, C#, Assembly, Java), computing theory, and networking
Sussex County Community College 2003 - 2005
A.S. in Computer Science