Recovering the Manager

Use this topic when the Manager machine has failed, been encrypted, or been rebuilt, and you need a working Manager again. If the Manager was compromised, do not follow it: a compromised Manager is replaced with a new CA, not recovered. See Re-Keying the Manager CA.

Before you begin

  • Have the offline record from the Manager Day-One Checklist: the recovery passphrase, the enrollment token, and the break-glass credential.
  • Have the CA fingerprint the agents trust, from your recovery notes or from a surviving agent. See The fingerprint to compare against.
  • Choose the route from what you have. A backup of the whole Manager machine is simplest. Without one, build a new Manager from a configuration backup on the storage machine, which brings back everything a database snapshot does, plus the enrollment token and the secret settings. A configuration backup and a database snapshot both need the recovery passphrase; without it, re-key the CA instead.

Note: Important: Sign in as soon as the Manager service starts on a new installation. A Manager that starts on an empty database offers its one-time setup to whoever reaches the console first.

The fingerprint to compare against

A recovered Manager must have the same CA as before, or every agent refuses it. To check, you compare the recovered Manager's fingerprint with one that did not come from the recovered Manager: the copy in your recovery notes, or the fingerprint a surviving agent was installed with.

Each agent keeps the fingerprint it trusts in the ManagerCaFingerprint value under its registry key:

  • Storage agent—HKLM\SOFTWARE\Cygna Labs\Disaster Recovery Suite\Storage Agent\Configuration\CygnaDR
  • DC agent—HKLM\SOFTWARE\Cygna Labs\Disaster Recovery Suite\DC Agent\Configuration\CygnaDR
  • Isolated Recovery Host agent—HKLM\SOFTWARE\Cygna Labs\Disaster Recovery Suite\Isolated Recovery Host Agent\Configuration\CygnaDR

The storage machine is usually the best source, because it is outside the domain and is the machine most likely to survive. For example, on the storage machine:

Get-ItemPropertyValue -Path 'HKLM:\SOFTWARE\Cygna Labs\Disaster Recovery Suite\Storage Agent\Configuration\CygnaDR' -Name ManagerCaFingerprint

The stored value is written as it was given at installation, so it may use colons or spaces instead of hyphens. Compare fingerprints ignoring letter case and the separators between the groups of digits.

Restoring the whole machine

To recover the Manager from a whole-machine backup:

  1. Restore the Manager machine or virtual machine from its backup, and start it.

    The CA, the address, the database, and the enrollment token all come back with the machine, so the Manager starts as it was.

  2. Sign in to the console, and select Administration > Agents.

    Agents reconnect on their own within a few minutes.

  3. Start a backup of every domain controller you back up.

    See After any Manager recovery.

Building a new Manager from a configuration backup

You need a configuration backup from the storage machine, the recovery passphrase that was current when it was written, and the Manager package of the same version as the old Manager or newer. For what a configuration backup holds, see Manager Configuration Backups.

To build a new Manager from a configuration backup:

  1. On the storage machine, sign in with an account in the CygnaDR Administrators group, and copy the newest backup from <StorageRoot>\manager-backups.

    Backups are named cygnadr-config-<yyyyMMdd-HHmmss>.cdrbundle, with the time they were written in UTC. If the storage root is not where you expect, use the folder that the old Manager's Configuration backups card and cdr config backup showed, if you recorded it. If you have changed the recovery passphrase since, choose a backup written under the passphrase you have.

  2. Build the machine and give it the old Manager's IP address.

    If it cannot have the old address, carry on, and re-point the agents afterwards as described in If the new Manager has a different address.

  3. Install the Manager:

    msiexec /i CygnaDR.Manager.msi /qn ENROLLMENTTOKEN=<token>

    Use the old Manager's enrollment token if you have it. If you do not, give any token: the import puts the old one back. If the old Manager used values other than the defaults for RESTPORT, GRPCPORT, or LISTENADDRESS, give the same values again. Do not open the console.

  4. Stop the Manager service:

    Stop-Service CygnaDRManager
  5. Copy the backup file to the new Manager, open an elevated command prompt in the Manager's program folder, %ProgramFiles%\Cygna Labs\Disaster Recovery Suite Manager, and check the file:

    CygnaDR.Manager.exe bundle inspect <file.cdrbundle>

    The command needs no passphrase. It shows when the backup was written, by which Manager and version, and its CA fingerprint, which must match the one the agents trust. See The fingerprint to compare against.

  6. Import the backup:

    CygnaDR.Manager.exe bundle import <file.cdrbundle>

    The command shows the backup again and the database it is about to replace. It refuses a backup written by a newer Manager version; install that version or newer first.

  7. Enter the recovery passphrase when asked.

    You have three tries, and a wrong passphrase changes nothing.

  8. Type IMPORT to continue.

    The import keeps the new installation's database beside it as cygnadr.db.before-import-<time>, puts the backup's database in place, protects the CA key for this machine, restores the secret settings, writes the enrollment token back if this installation's differs, and records the import in the audit trail with the Windows account that ran it. It then prints what it did.

    If it reports that the CA could not be unlocked with this passphrase, the database's escrow copy was set under a different passphrase. Run CygnaDR.Manager.exe ca recover with that passphrase before you start the service. If it lists settings that were not in the backup, enter them again on the Configuration screen after you sign in.

  9. Start the Manager service:

    Start-Service CygnaDRManager

    The Application log records the CA fingerprint at every start. It must match the one in step 5.

  10. Sign in with a named account or the break-glass account.

    Accounts, roles, settings, and backup schedules are as they were when the backup was written. The SMTP password came back with the backup, so you do not need to enter it again. Operators who use multi-factor authentication sign in with their authenticator app as before: the backup carries their MFA enrollment.

  11. Select Administration > Agents, and wait for the agents to reconnect.

    Agents reconnect within a few minutes with no change on their side. An agent that enrolled after the backup enrolls again with its kept token and appears after one more reconnect.

  12. Start a backup of every domain controller you back up.

  13. On the License screen, check that the Configuration backups card shows a new backup stored within the day.

Building a new Manager from a snapshot

Use this route when you have a database snapshot but no configuration backup. You need a database snapshot file (cygnadr-snapshot-*.db, from the snapshots folder in your file backup), the recovery passphrase, the enrollment token, and the Manager package of the same version as the old Manager or newer.

To build a new Manager from a database snapshot:

  1. Build the machine and give it the old Manager's IP address.

    If it cannot have the old address, carry on, and re-point the agents afterwards as described in If the new Manager has a different address.

  2. Install the Manager with the old Manager's enrollment token:

    msiexec /i CygnaDR.Manager.msi /qn ENROLLMENTTOKEN=<token>

    A new installation uses the default RESTPORT, GRPCPORT, and LISTENADDRESS. If the old Manager used other values, give the same values again. The service starts on an empty database and creates a CA of its own, which you are about to replace. Do not open the console.

  3. Stop the Manager service:

    Stop-Service CygnaDRManager
  4. In %ProgramData%\Cygna Labs\Disaster Recovery Suite Manager, delete cygnadr.db, cygnadr.db-wal, and cygnadr.db-shm.

    No -wal or -shm file from the new installation may remain beside the snapshot.

  5. Copy the snapshot into the same folder and rename it cygnadr.db.

  6. Open an elevated command prompt in the Manager's program folder, %ProgramFiles%\Cygna Labs\Disaster Recovery Suite Manager, and check the CA:

    CygnaDR.Manager.exe ca status

    Expect Usable here: NO and Escrow copy: yes. The fingerprint shown must match the one the agents trust. If Escrow copy is NO, the recovery passphrase was never set for this database: re-key instead.

  7. Recover the CA:

    CygnaDR.Manager.exe ca recover

    Enter the recovery passphrase when asked. You have three tries per run, and a wrong passphrase changes nothing. On success the CA key is protected for this machine, and an audit entry names the Windows account that ran the command.

  8. Start the Manager service:

    Start-Service CygnaDRManager

    The Application log records the CA fingerprint at every start. It must match the one in step 6.

  9. Sign in with a named account or the break-glass account.

    Accounts, roles, and passwords are as they were when the snapshot was taken.

    Multi-factor authentication enrollments are protected for the old machine and cannot be read here. An operator who used MFA signs in with one of their recovery codes, then turns MFA off and sets it up again; without a recovery code, an administrator resets their MFA. The break-glass account signs in with its password alone, and MFA is turned off for it, with an entry in the audit trail. Turn it on again from the account menu. See Multi-Factor Authentication.

  10. If the Manager sends email, enter the SMTP password again: select Administration > Configuration, and on the Manager tab, in the Email group, enter the password and save. Or use cdr config set CygnaDR:Email:SmtpPassword, which prompts for it.

    Passwords kept in settings are protected for the old machine, and ca recover carries over only the CA key. On the new machine the password reads as not set, and the Manager logs that it needs entering again. Until you enter it, recovery plans send no notifications and no report email.

  11. Select Administration > Agents, and wait for the agents to reconnect.

    Agents reconnect within a few minutes with no change on their side. An agent that enrolled after the snapshot is unknown to the restored database: the Manager refuses its session, and the agent enrolls again with its kept token and appears after one more reconnect. Check the Repository screen to confirm the storage agent has reconnected. If you use an Isolated Recovery Host, check that its panel on the Agents screen shows it connected and Ready; scheduled test restores resume once it is.

  12. Start a backup of every domain controller you back up.

  13. If there is any doubt about who has seen the recovery passphrase, set a new one on the Manager recovery card on the License screen, and replace the offline record in the same sitting.

    Copies written before the change still open with the old passphrase. See After you change the recovery passphrase.

If the new Manager has a different address

Agents trust the CA, not the address; the address is only where they look. A Manager on a new address with the recovered CA is trusted as soon as agents can reach it, so they need only the new address:

  • DC agents—Run cdr agent deploy with --repoint --force --manager-address https://<new address>:5211. This keeps their fingerprint. See Maintaining Agents.
  • The storage agent, the Isolated Recovery Host agent, and agents installed by hand—Reinstall with the new MANAGERADDRESS.
  • Operators—Use the new address in the browser and with cdr --manager. cdr asks you to confirm the fingerprint for the new address; it should be the one the agents trust.

After any Manager recovery

  • Take a fresh backup of every domain controller you back up. Backups that completed after the machine backup, configuration backup, or snapshot are still in the vault, but the restored Manager does not know about them and cannot restore from them. Fresh backups give it current restore points.
  • Check the Manager recovery card. It should show the passphrase as set and a snapshot written within the day. On a new machine the first snapshot is written a few minutes after the service starts.
  • Check the Configuration backups card. It should show a backup stored on the storage machine. On a new machine the first is sent a few minutes after the service starts.
  • Review the audit trail. Nothing that happened between the backup or snapshot and the failure is in it.
  • Review anything else changed since the backup or snapshot. Policies, settings, operators, and recovery plans are as they were when it was taken.
  • Update your records. Update the offline record if the address or the passphrase changed, and your recovery notes if the fingerprint changed.