This is a composite scenario for illustration.

Summary

Consider a case we see in small shops more often than anyone would like. A monitoring appliance goes up for a quick internal test. It gets exposed to the internet for convenience. Someone creates a new admin user with a strong password, and the original default admin account is forgotten, still on its default password. The test quietly becomes production. Eventually someone logs in with those default credentials and uses the appliance's remote script and command feature to run a payload on the hosts it monitors.

The example is a monitoring appliance, but the pattern applies to almost any device, appliance, container, or VM that starts life as "just a test." The software usually works exactly as designed. The gap is in how it drifted from test to production without anyone deciding it should.

The scenario

It starts as a test. An admin wants better visibility into servers and hypervisors, so they deploy a monitoring appliance. It is meant to be temporary, a proof of concept to see whether the tool is worth adopting.

It gets exposed for convenience. Checking dashboards from home or a phone is easier if the web console is reachable from the internet, so a port gets opened. The plan is to lock it down later.

A new admin account is created. Being careful, the admin creates their own named account with a long, strong password and uses that from then on. The built-in admin account that shipped with the appliance is never touched again. It is not disabled, not renamed, and not re-passworded.

The test becomes production. The tool works well. Agents get rolled out to more hosts. Alerts start going to the on-call phone. Settings get tweaked here and there as needs come up, and nobody writes them down. Nobody ever decides it is production. It just is. In a small shop with a lot on everyone's plate, this is extremely common.

Someone logs in. Default credentials for common appliances are documented and widely known. An attacker, or an automated tool, tries them against the exposed console. They work on the first try, so there is no burst of failed logins to trigger an alert.

The remote script feature does the rest. Most monitoring appliances can run scripts or commands on monitored hosts. That is a legitimate feature for routine operations. With an admin session, the attacker creates a script, runs it against monitored hosts, and deletes it to tidy up. In this case the payload was a crypto-miner, set to survive reboots and process kills through crontab entries and systemd units.

The first sign is often just a host running hot for no obvious reason.

This is not just about monitoring tools

Almost anything with a web console, a default login, and a network connection can follow the same path from test bench to production. Categories worth scanning your own environment for:

Monitoring and observability servers
Firewalls
VPN concentrators
Switches and routers
Wireless controllers and access points
Load balancers
UPS and PDU network management cards
IP cameras and network video recorders (NVRs)
Access control and door entry systems
HVAC and building management controllers (BMS)
PBX systems and VoIP phones
Print servers and multifunction printers (MFPs)
NAS and SAN storage
Backup appliances
Hypervisor management and out-of-band management consoles (IPMI and BMC style)
KVM-over-IP devices
Log collectors and SIEM collectors
DNS and DHCP appliances
Email gateways
Web filters
IoT hubs
Digital signage players
POS terminals
Environmental sensors (temperature, humidity, water)
Lab containers and VMs spun up from templates

Many of these ship with a default admin login, a web console, and some way to run commands, push configuration, or reach other systems. Any of them can be "temporary" for years.

Why this keeps happening

None of the individual decisions is reckless. Testing a tool is good. Remote access is useful. Creating a named admin account is a best practice. The problem is that no single moment ever forces anyone to look at the whole system again before real data and real reach depend on it.

A monitoring appliance with remote commands enabled is effectively an admin console for every host it watches. If it is protected like a dashboard instead of like an admin console, it becomes the easiest way into everything else.

Core lessons

1. Test appliances quietly become production

Assume that anything you spin up for a test will still be running a year from now with people depending on it. Plan for that from day one: give it an owner, a name in your inventory, and a planned review date.

2. Keep it on the VPN or internal network until a deliberate go-live

Until a system is deliberately ready to go live, it should not be reachable from the open internet. Use your VPN, an internal network, or an IP allowlist for remote viewing. "Temporary" exposure has a way of becoming permanent.

3. Audit every account, including the forgotten default admin

Adding a new admin does not remove the old one. On any new appliance, container, or VM, list every account, including built-in, initial, and service accounts. Disable, rename, or re-password the default admin, and check what groups and roles it still holds.

4. Review for config drift with change control at the go-live gate

Between the first test and the day people depend on it, settings change. Ports get opened, features get turned on to try something, accounts get added, and debug options get left behind. At the test-to-production gate, compare the current configuration to a known baseline, review every difference, and decide on purpose what stays. From then on, put changes through change control, even if that is just a short written record of who changed what and why. If nobody can say when a test system became production, it skipped the gate.

Supporting lessons

5. Turn on MFA

A correct password on the first try sails past lockout and failed-login alerts. MFA closes that gap. Any console that can run commands on other systems deserves MFA.

6. Restrict remote script and command features

Disable remote command or script execution features you do not use. Where you need them, limit them to specific commands, specific hosts, and a small set of users. Treat the right to create a script as equal to root on every host that script can reach. Scope API tokens to least-privilege users, give them expiration dates, and rotate them.

7. Monitor the audit log

Alert on creation, change, deletion, and execution of anything that can run code. A script that appears and disappears within minutes is a strong signal. Alert on successful logins to admin accounts, especially at odd hours or from unfamiliar addresses, not only on failed logins. Ship audit logs off the box so a cleanup attempt does not erase them.

8. Check hosts for persistence

If remote commands may have been abused, stopping a process is not the same as cleaning a host. Check crontabs for every user and /etc/cron*, review systemd units and timers in system and user locations, and look for unexplained sustained CPU usage. Where rootkit-style persistence is involved, rebuilding the host is often safer than trusting a cleanup.

If it already happened

Isolate affected hosts before you start cleaning, so you can capture evidence. Rotate passwords and API tokens and end active sessions. Remove attacker-created scripts and disable remote commands until you have reviewed them. Sweep every host the appliance could reach, not just the one that looked wrong.

Closing

The pattern is simple: a test system, a convenience shortcut, a forgotten default account, untracked changes, and a powerful feature working as designed. Breaking any one link would have stopped it. The cheapest links to break are the first four, and they cost nothing but a deliberate review.

If you have an appliance, device, container, or VM that started as a test and is now something people depend on, take an hour this week. Check what is exposed, list every account, compare the configuration to what you meant it to be, and decide on purpose whether it is ready for production. If you would like a second set of eyes on that review, that is exactly the kind of work we do.

Download the free Appliance Go-Live Checklist (PDF).

Avery Parker, Diversified Tech Solutions

The Test Box That Became Production: A Case Study in Forgotten Default Accounts