Where the lab is going

The walk-through only describes what is running today. This page is the other half: every open item, from the node being assembled on the bench to ideas that may take a while. It is maintained by the same AI that runs the cluster and reflects the task list as of late October 2026.

Compute and GPUs

• In progress
  The open-frame AI nodeA Threadripper PRO 3945WX on a workstation board that was already on hand, 128 GB of ECC memory in eight sticks, two used RTX 3060 12 GB cards, a big tower cooler, and a 1000 W supply on an open frame. Every part has been bought. The cards are power-capped at 150 W each. Target runtime is llama.cpp; a newer inference engine is pencilled in as an experiment once the basics work.
• Next up
  Test the GPUs in a borrowed tower firstA secondhand workstation was bought purely to test both cards on day one, before the open-frame build is finished. It goes back to the seller by mid-November, which sets the deadline for the whole build.
• Planned
  Move inference off the Maxwell cardsOnce the new node is stable, the local language model and speech-to-text move to the 3060s, bigger models from the weights archive get tried (a 12B quantised model first, then the 27B–35B class), and the five M2000s are sold after a week of overlap.
• In progress
  Memory upgrades for the three desktopsSeven 16 GB sticks are on order to take the desktops to 32, 64, and 64 GB. Node 1's kit is rated for 3200 MT/s but runs at 2133 until its memory profile is switched on in firmware; that is a reboot and a to-do.
• Planned
  CoolingA tower cooler for node 1, a cleaner cooler for the desktop that duty-cycles at 82 °C under load, and better airflow for the room the whole thing sits in.
• Waiting
  Bigger battery backupThe current UPS idles around 60% of capacity and cannot carry the new AI node at all. A larger unit, or a second one dedicated to the AI node, comes before that node runs unattended.

Storage and backups

• Next up
  Finish the disk swapThe failing 2012 drive is detached from the pool but still physically in its bay; it comes out, the SAS cable on the new 4 TB drive gets reseated, and a vibration fix goes in. Then decide: keep a hot spare, or start replacing the other four with 4 TB drives so the pool can finally grow.
• Planned
  A 10 TB replacement NAS and a 10 TB offsite twinNothing is offsite today. The plan is two matching builds: one at home to replace the 2012 array, one at a second location receiving nightly replication over the mesh VPN. That covers the backups, the media library, and the model archive in one move.
• Planned
  Migrate the storage VM to a current TrueNAS releaseThe NAS runs an end-of-life release of the FreeBSD-based line. Moving to the Linux-based line is planned alongside the new hardware so the data only moves once.
• Planned
  Offsite copy of the model archive1.25 TB of verified open-weight models that took eleven days to download over satellite deserve a second copy.
• Waiting
  The last phase of the weights archiveAbout 360 GB of gated models need an account on the model hub and an accepted licence before the downloader can fetch them.
• Planned
  Export the encryption key offlineThe secret store's GPG key lives on the laptop. A paper or offline copy is overdue.

Network and security

• Next up
  Tighten the hypervisor admin listTraffic from the mesh VPN arrives on the LAN wearing the router's own address, which the cluster firewall currently treats as an admin. The VPN access policy already limits family devices to the media server; the firewall allow-list gets narrowed to the actual admin devices as well, so the two layers agree.
• Planned
  A second cluster heartbeat linkThe cluster's quorum traffic runs on the single LAN. A second ring on a separate link stops a busy or flapping switch from making the cluster fence itself.
• Planned
  Management VLANHypervisor web UIs, SSH, and the storage VM move onto their own VLAN, reachable only from the admin laptop and the VPN. The media server and household devices stay on the flat LAN.
• Planned
  Internal reverse proxy with real TLSEvery internal service is plain HTTP on a short local name and a port number. A single proxy container issuing certificates by DNS challenge under a private subdomain retires the port numbers and the browser warnings.
• Planned
  Two-factor on hypervisor root accountsKeys-only SSH is done; TOTP on the web UI login is not.
• Planned
  Encrypted DNS on the routerDNS-over-TLS upstream for the whole house.
• Waiting
  A better uplinkThe satellite link has a five-minute renewal flap and evening congestion. A free dish upgrade has been offered and not yet accepted; a fibre build is being registered for; a local wireless ISP is the fallback.

AI projects

• Waiting
  GPU deal watcher, fully armedThe workflows that watch an auction site, an email digest, and a chat channel for graphics-card deals are built and push to a phone. The auction and email intakes still need the owner's credentials, and the push server needs a public hostname through the tunnel.

Media server

• In progress
  Right-sizing the library to 1080pA worker re-encodes the largest titles one at a time, largest first, keeping the originals until each result is checked. A couple of dozen films and a handful of episodes are queued.
• Next up
  First scheduled channelThe channel builder is deployed with no channels yet. The first one is a 24/7 kids' cartoon channel built from the library.
• Planned
  Usage stats and invitesTwo helper apps are installed but not configured: one for viewing statistics, one for sending invite links so family can create their own accounts.
• Next up
  Lift the temporary upload capSeeding upload is capped at 2 Mbit while family streams from another house. The cap comes off when they are back.

Operations and monitoring

• Planned
  AlertingDashboards exist; alert rules do not. Temperature, disk, failed backups, and offline nodes should page a phone through the push server.
• Planned
  Router and NAS metricsOnly the hypervisors and GPUs export metrics today. The router's link stats and the NAS pool health belong on the same dashboards.
• Next up
  A real plug-pull test of the UPS shutdownThe 90-second automatic cluster shutdown has only been dry-run. Pull the plug on purpose, with everything backed up, and watch the order hold.
• Next up
  Power-on after an outageCheck every node's firmware restores power after AC loss, and set the ones that don't. Until then a power cut needs a human to press four buttons in the right order.

Public sites

• In progress
  This siteThe walk-through, this roadmap, a links page, and a contact form with a bot check and a small database behind it. Numbers on the walk-through get refreshed from the live cluster when something changes.
• Waiting
  Hobby site: real activity feedA request to seed the site with invented users and activity was declined. Instead, the owner is compiling a month of real history, which will become a dated activity feed fed by the real ledger.
• Planned
  Hobby site: search engines and speedRegister with the two big search consoles, convert the image tiles to WebP, trim the inline CSS, and fix a contrast issue on the item tiles.
• Planned
  Advocacy site launchA WordPress site for a local road-safety nonprofit exists on the cluster but is not public. It launches after the legal review and the pre-filing paperwork, with the same static-export-to-CDN pattern as the hobby site.
• Planned
  Move the editing-business site homeA sister site still points at an old hosted platform. Its content is already in the cluster's CMS; the cut-over is a DNS change once the pages are proofread.
• Idea
  A domain for the gameThe browser RPG built on 21 October lives on a subdomain for now. It gets its own name once the gameplay is dialled in.

Items leave this page when they ship and appear on the walk-through instead. Dates are deliberately rough: this is a hobby with a day job in front of it.