Moving the Storage Machine or an Isolated Recovery Host

You may need to move the storage machine or an Isolated Recovery Host, for example onto new hardware or from a pilot setup into production. The two moves work differently:

  Storage machine Isolated Recovery Host
What is lost Every restore point. Backups start again from zero on the new machine. Nothing. The test restore history stays.
Who can do it An operator holding both agents.manage and backup.manage, or two operators: one asks, and one holding the other capability approves. An operator holding agents.manage.
Old and new side by side The new storage agent can be installed beside the old one, but it does no work until the storage machine is replaced with it. Yes. Any number of approved hosts share the test restores.

The same machine at a new IP address is not a move. Change the address, keep it fixed, and restart the agent on that machine.

One storage machine, and candidates

An installation has one storage machine. The first storage agent to enroll becomes it. Any storage agent that enrolls after that is a candidate: it is listed on the Agents screen as Candidate, given no work, it is sent no backups, restores, or configuration backups, and it raises the alert Storage agents that are not the storage machine until it is used or removed. The storage machine is the only one the backups, the configuration backups, the storage alerts, and the Repository screen read.

That alert is also what an unexpected storage enrollment looks like. If nobody installed the candidate, uninstall its agent on that machine, delete its record with cdr agent forget, and treat the enrollment token as known to whoever installed it.

The storage machine itself cannot be uninstalled or forgotten from the Manager; the Manager refuses with It is the storage machine. Without a storage machine, the next storage agent to enroll would become it with nobody deciding. Replace it, and then remove it.

Replacing the storage machine

What a replacement does

  • Restore points are not moved. Every restore point on the current storage machine stops being restorable when the replacement is carried out. The backup history stays, in backup jobs, reports, and the audit trail; only the ability to restore from those backups is lost. Restore points ended this way show Storage machine replaced as the rule that removed them. The confirmation gives the number, for example 412 restore points across 37 DCs will no longer be restorable; backups start again from zero on the new machine.
  • Backups that were never committed are given up. Their content is on the old machine.
  • Every domain controller with an enabled backup policy is backed up at once to the new machine, within the usual limit on concurrent backups, instead of at its next scheduled time. Until its backup finishes, a domain controller has no restore point.
  • The configuration backup is sent to the new machine straight away. See Manager Configuration Backups.
  • OS media and recovery images—The old machine's OS media and recovery images leave the library. The new machine builds recovery images from its own media folder, so copy the installation ISOs there. Images that had passed a test restore are tested again.
  • Retention floors are not carried over. The retention floor and the minimum versions per DC are set in each storage machine's own appsettings.json and cannot be set over the network. The confirmation shows both machines' values side by side and marks the new machine's as Weaker where it protects less. Set them on the new machine before you replace. See Settings Reference.
  • The old machine still holds every backup it took, each domain controller's directory database included. Nothing in Cygna Forest Recovery can delete it, because no network path deletes vault content. Uninstall its agent and wipe its storage root afterward.

Before you begin

The replacement is refused, and a request for it stays pending, while any of these is true:

  • A restore, a recovery onto a new VM, or a test restore has not finished.
  • A recovery plan is approved, running, or halted.
  • A backup is running. Wait for it, or cancel it on the Backup Jobs screen or with cdr jobs cancel.
  • The new storage agent is not connected.

To replace the storage machine

  1. Prepare the new machine as you would any storage machine: not joined to a domain, with a fixed IP address, a volume for the storage root, and the staging share. If it is to hold OS media, copy the ISOs into its media folder. If the old machine's retention floor or minimum versions per DC were changed from the defaults, set them in the new machine's appsettings.json. See Setting Up the Storage Machine.
  2. Install the storage agent on it with the same Manager address, CA fingerprint, and enrollment token as any agent:

    msiexec /i CygnaDR.StorageAgent.msi /qn MANAGERADDRESS=https://10.0.0.10:5211 MANAGERCAFINGERPRINT=<fingerprint> STORAGEROOT=E:\CygnaDR ENROLLMENTTOKEN=<token>

    It enrolls as a candidate and is given no work.

  3. Look at what the replacement would do. On the Agents screen, open the candidate's row menu and select Replace the storage machine with machine…, or run:

    cdr storage replace <candidate agent id>

    Nothing changes yet. Check the number of restore points that end, the comparison of the two machines' local settings, and anything that blocks the replacement.

  4. Ask for it. In the dialog, enter the candidate's agent ID and select Replace the storage machine or Ask for the replacement, or run:

    cdr storage replace <candidate agent id> --confirm <candidate agent id> --note "Moving to production"
    • If you hold both agents.manage and backup.manage, as the break-glass account does, the storage machine is replaced in this one step.
    • Otherwise this creates a request that waits for an operator holding the capability you lack. A System Admin's request waits for a Backup Operator, and a Backup Operator's for a System Admin. The alert Requests waiting for a second operator is raised, and emailed with the other alerts, and the request appears on the Agents and Repository screens.
  5. The second operator approves it. On the Repository screen, select Review and approve… on the request, or run:

    cdr request show <request id>
    cdr request approve <request id> --confirm <candidate agent id>

    Approving shows what the replacement would do now, which can differ from when it was asked for, and asks for the candidate's agent ID again. Select Approve and replace the storage machine. It is carried out at once.

  6. Watch the first backups on the Backup Jobs screen. A domain controller whose backup could not start is named in the result and in the Manager's log.
  7. Retire the old machine. Uninstall its agent from the Agents screen or with cdr agent uninstall, delete its record with cdr agent forget, and then wipe its storage root and remove its staging share. Until it is wiped, it holds a copy of every domain controller's directory database.

A replaced machine is listed on the Agents screen as Replaced and is given no work. To use it again, delete its record with cdr agent forget and let its agent enroll again; it comes back as a candidate.

Requests

An action that needs two capabilities, such as replacing the storage machine, runs as a request when the operator who asks holds only one of them. Nothing happens until it is approved.

  • A request lapses after 24 hours if nobody decides it. It is withdrawn if the candidate's record is deleted or another storage agent becomes the storage machine first.
  • The requester cannot approve or decline their own request, and can cancel it. An operator who could approve it can decline it, with a note. Any operator holding one of the two capabilities can cancel it.
  • One storage replacement can be pending at a time.
  • Pending requests, and those decided in the last week, are shown on the Repository screen. The Agents screen shows the pending ones. cdr request list lists them, and --all lists every request. The command exits with code 4 while one is pending.
  • The audit trail records each step: Request made, waiting for a second operator, Request approved, Request declined, Request cancelled, Request lapsed, Request withdrawn, and Storage machine replaced with the counts.

For the commands, see Storage and Media Commands.

Moving an Isolated Recovery Host

Nothing is lost when a host moves. Each test restore keeps the name of the host it ran on, so its history outlives the host, and test restores go to whichever approved host can run one. A move is therefore: add the new host, drain the old one, and remove it.

Approving and draining hosts

A host is given test restores only once an operator approves it. A newly installed host enrolls Awaiting approval, however ready it is, and raises the alert Isolated Recovery Hosts awaiting approval. Approve only a host you installed: any machine holding the enrollment token can enroll one.

Draining a host stops new test restores going to it. The test restore running on it finishes. A test restore that was planned for it but has not started is refused there; plan it again and it goes to another host. Once nothing runs on it, the host reads Drained and can be removed. You can put a drained host back in use.

Approving, draining, and putting a host back in use need agents.manage, and each is recorded in the audit trail. Do them in the host's panel on the Recovery hosts tab of the Agents screen with Approve…, Drain, and Use it again, or with cdr recovery-host approve, drain, and undrain. Backups show as not verified only while no approved host is ready.

To move an Isolated Recovery Host

  1. Prepare the new host and install the Isolated Recovery Host agent on it. See Setting Up an Isolated Recovery Host.
  2. Give it its own settings, if it needs any. The folder for verification VMs and the recovery network prefix apply to every host unless a host has its own value as an override. An override belongs to the old host's agent and does not carry over: read the old host's overrides on the Configuration screen, and set the ones the new host needs with Override a setting on this machine… or cdr config set --agent.
  3. When the new host reads Ready, approve it: select Approve… in its panel on the Agents screen, or run cdr recovery-host approve <host agent id>.
  4. Drain the old host: select Drain in its panel, or run cdr recovery-host drain <old host agent id>. Wait until its panel, or cdr recovery-host list, says it is drained.
  5. Remove the old host: uninstall its agent with cdr agent uninstall, and then delete its record with cdr agent forget. Deleting the record is refused while a test restore still names the host. If the machine stays in service for something else, remove the CygnaDR Recovery switch, its NAT, and the folder for verification VMs.

Related topics