The 3-2-1 backup rule has long been a staple of the IT industry. However, in the modern-day risk environment, the 3-2-1 rule of thumb is not sufficient on its own to protect against ransomware or stolen cloud admin credentials. How those backups are protected matters as well.
In the event of a ransomware attack, it is not unheard of for attackers to compromise the backup system as well. By 2014, ransomware variants were already deleting Windows Volume Shadow Copies. In the cloud environment, how do you protect your admin credentials? How do you protect your backups in the case where those credentials do get stolen? Maybe the question isn't so much about admin credentials, but rather about the roles or systems that have privileged access to both production and backup data? In any case, for disaster recovery—or rather, for integrity and availability—one should consider the worst case when forming a backup plan.
So, what can you do? Well, wherever you can, keep recovery copies that the systems and credentials used in production cannot overwrite or delete. One-way replication can be part of that. In the case of cloud storage, this could look like continuous, asynchronous replication from one S3 bucket to another, with the destination under separate administrative control. Versioning needs to be enabled on both buckets, though versioning alone does not prevent someone with sufficient privileges from deleting older versions. This is where immutable retention comes in—specifically, a thing called WORM (write once, read many), which S3 supports through Object Lock. And of course, test that you can actually restore from those backups. This is but one way.
Where Offsite Backups Still Break Down
Teams often assume that because a backup copy lives in another region or on a separate storage appliance, it is out of reach when something goes wrong in production. Hardware failures work that way. An attacker with stolen credentials does not.
Consider an on-premises backup server joined to the same Active Directory domain as the rest of the company. If someone compromises a Domain Controller, they generally inherit administrative access to that backup server right along with everything else. A similar problem shows up in AWS when RDS snapshots, EBS snapshots, and S3 backup buckets all live inside the same account as the application itself. One leaked role with broad permissions—or one compromised deployment runner—can delete production data and clear out the snapshots in the same script.
Even teams that turn on S3 Object Lock sometimes stop half a step short. Object Lock has two retention modes: GOVERNANCE and COMPLIANCE. Governance mode sounds strict enough, and for accidental deletions by well-meaning engineers it usually is, except anyone holding the s3:BypassGovernanceRetention permission can override the lock and remove objects immediately. Compliance mode closes that door so that no IAM user, not even the account root user, can shorten retention or delete a protected object version before its clock runs out.
Is Compliance mode the end of the story? Almost, though AWS accounts have one administrative edge case worth knowing about: closing the member account itself. Closing an AWS account begins a post-closure window after which resources are eventually reclaimed. Putting the backup vault inside its own dedicated member account and attaching an AWS Organizations Service Control Policy (SCP) that denies account closure keeps a compromised role inside the workload account—or even inside the vault account—from tearing the vault down from above.
| Backup Concern | Conventional 3-2-1 Setup | Separated, Immutable Setup |
|---|---|---|
| Copy count and placement | Primary data plus two backups, with one stored offsite | Live production, a fast warm recovery copy, and an isolated immutable vault copy |
| Administrative boundary | Shared Active Directory domain or shared cloud account | Separate authentication plane with no shared production admin roles |
| Offsite destination | Second bucket or replicated NAS under the same team credentials | Dedicated backup account in another Organizational Unit and region |
| Deletion protection | Bucket versioning or standard snapshot schedules | WORM retention via S3 Object Lock COMPLIANCE mode, Backup Vault Lock, or hardened Linux XFS |
| Verification | Checking whether nightly backup emails came back green | Scheduled restores into an isolated environment to confirm applications and databases actually run |
Setting Up a Cross-Account S3 Vault
If your workloads run in AWS, a clean pattern is to create a dedicated backup AWS account inside its own Organizational Unit. Production engineers and CI pipelines do not need interactive sign-in access there. Replication or backup jobs push encrypted copies across the account boundary, and the destination bucket applies a default COMPLIANCE retention period—often somewhere between 14 and 30 days on incremental chains so storage bills stay reasonable while still giving you weeks of clean recovery points.
# Create a dedicated vault bucket with Object Lock enabled and a 30-day COMPLIANCE retention floor
aws s3api create-bucket \
--bucket example-vault-immutable-usw2 \
--region us-west-2 \
--create-bucket-configuration LocationConstraint=us-west-2 \
--object-lock-enabled-for-bucket
aws s3api put-object-lock-configuration \
--bucket example-vault-immutable-usw2 \
--object-lock-configuration '{
"ObjectLockEnabled": "Enabled",
"Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 30 } }
}'
At the AWS Organizations root or Organizational Unit level, an SCP can then block anyone inside that member account from disabling Object Lock, deleting recovery points, scheduling KMS key deletion, or closing the account:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DenyBackupVaultDestructionAndAccountClosure",
"Effect": "Deny",
"Action": [
"organizations:LeaveOrganization",
"account:CloseAccount",
"s3:PutBucketObjectLockConfiguration",
"s3:BypassGovernanceRetention",
"backup:DeleteBackupVault",
"backup:DeleteBackupVaultLockConfiguration",
"backup:DeleteRecoveryPoint",
"kms:ScheduleKeyDeletion",
"kms:DisableKey"
],
"Resource": "*"
}
]
}
Notice that KMS keys sit in that deny list right alongside the S3 and AWS Backup actions. Destroying the encryption key renders ciphertext just as unrecoverable as deleting the objects themselves, which is an easy detail to overlook when people focus only on bucket policies.
When Local Recovery Speed Matters Too
Cloud vaults are hard to beat for durability and administrative separation, but what if you run several terabytes of local VMware or Hyper-V virtual machines in a plant, clinic, or office? Pulling all of that back down over a business internet circuit during an outage can turn a four-hour recovery target into a three-day wait.
In that situation, pairing an offsite cloud vault with a local hardened Linux repository—Ubuntu LTS or RHEL formatted with XFS and reflink=1 fast-clone support—gives you local restore speed without handing control back to Active Directory. Keep the Linux host completely off the Windows domain. Register the backup transport using single-use credentials, run the data-mover service under an unprivileged local user without sudo, and turn sshd off once setup is finished (systemctl disable --now ssh.socket ssh.service).
Two smaller details matter on physical Linux vaults as well. One is out-of-band management: an iDRAC, iLO, or IPMI interface left reachable on the general server VLAN lets an attacker mount a recovery ISO and wipe disks below the operating system, so those management ports belong unplugged or on a physically isolated management network inside a locked rack. The other is timekeeping. Point chronyd at external Network Time Security (NTS) servers rather than an internal Domain Controller, so a compromised NTP server cannot jump the system clock forward six weeks and trick retention rules into expiring immutable files early.
Proving That a Restore Actually Works
A backup job finishing without errors tells you that blocks were written somewhere. It does not tell you whether PostgreSQL write-ahead logs replay cleanly to a specific timestamp, whether an application config file was left out of the backup selection, or how long the restore actually takes when nobody has practiced it since last year.
Pick a cadence your team will genuinely keep—monthly or quarterly for critical systems—and restore into an isolated subnet where nothing touches live traffic. Boot the database, run a real query, check the elapsed clock against what leadership expects, and write down whatever tripped you up while doing it. If you want a structured way to run those exercises, our disaster recovery test plan guide and DR runbook template lay out the steps, and our managed backup services page covers how we set up and test isolated vaults for clients.