Recovery Images

A full system recovery restores a domain controller's whole server, operating system included, from a full system backup. Windows cannot restore its own system volume while it is running from it, so the restore runs from a separate, small Windows environment loaded into memory: the recovery image.

What a recovery image is

A recovery image is a Windows PE environment built from the Windows Recovery Environment (Windows RE) image inside a Windows Server installation ISO that you supply. The storage agent adds the recovery agent to it, and any drivers you provide. It is not a copy of the domain controller's own operating system, and it holds no backup data. Its only job is to start, reach the vault, and restore the backup onto the machine's disks.

The recovery agent is not installed or enrolled separately. It ships with the storage agent and is copied into every recovery image the storage agent builds.

One image per operating system

A recovery image must come from the same Windows Server release as the domain controller it restores. The storage agent builds one image for each operating system variant your domain controllers run, as recorded in their newest backups. A variant is the combination of:

The Windows build.
The edition.
The install type: Desktop Experience or Server Core.

Each value must match exactly. The update build revision (UBR) is not part of the match, so an ISO older than your patched domain controllers still works. Architecture (x64 or ARM64) must match as well. On the OS Media screen, media in a language your domain controllers do not run is reported as Wrong language only; it installs and works, but the tool output an operator reads during recovery will be in that language.

How images are built

The storage agent builds images ahead of time, because a build takes minutes and needs the installation media, both of which are scarce during a disaster. It builds them with the DISM tool included in Windows; the Windows Assessment and Deployment Kit is not needed.

Images are built only when the storage agent's Recovery image folder is set. If it is empty, no images are built and no domain controller can use full system recovery. DC state recovery is not affected.
The installation media come from the storage agent's Media folder, which holds your Windows Server ISOs. Keep the recovery image folder separate from the media folder; Cygna Forest Recovery never writes to the media folder.
The storage agent checks on a schedule whether any required image is missing or out of date, and builds it.
When you upgrade the storage agent, it builds new images with the recovery agent of the new release. A new image is untested until a restore proves it. A new image is added beside the existing ones, and older ones are retired as described below.
When more than one image fits a variant, an image already proven by a completed restore is used ahead of a newer one that has not been proven yet.

Proven images and retirement

An image is Proven once a full system recovery that staged it reports a completed restore. A test restore counts the same as any other recovery. The Manager records the verdict and the storage agent stores it in the library's catalog. An image that has failed keeps its failed verdict. A new image is Untested until a restore proves it.

With an Isolated Recovery Host enrolled and the Prove new recovery images setting on, untested images are test-restored automatically, each with a small recent backup of a domain controller the image serves, or as part of a domain controller's scheduled test, so a new image is proven before you need it. Only the newest image of each operating system variant on each storage agent is tested this way, because that is the image a restore would start from. See Test Restores.

Every product build adds a new image per operating system variant, so the library would grow without limit. The Manager therefore decides which images each library keeps, and the storage agent deletes only the images it is told to. For each operating system variant that restore points in retention still need, the Manager keeps:

The newest image, whatever its verdict.
The newest proven image.
If the newest image has failed, the newest image that has not failed.
Any image a restore that has not ended is using. An image in use is never retired.

Everything else is retired: superseded images, failed images once a newer one exists, and images of a variant that no restore point in retention needs. Nothing is decided for a library while its last report did not succeed. No setting controls retirement. The Recovery images tab of the OS Media screen and cdr os-media list --images show what each library keeps, what it retires, and what it has retired.

For the setup steps, see Setting Up OS Media and Recovery Images.

How an image is staged and started

For a recovery in place or onto a candidate, a recovery image is not booted from removable media or the network. The DC agent stages it on the machine being recovered while that machine's own Windows is still running, and the machine then starts from it once:

The DC agent on the target machine fetches the image from the storage machine, checks its hash, and copies it to a folder on the system volume.
The DC agent sets the machine to start from the image for the next start only, with the loader that matches its firmware, UEFI or BIOS, and restarts it.
The whole image loads into a RAM disk, and Windows PE starts from memory. This is why the target needs enough memory for the image plus about 1 GB.
The recovery agent takes its network address, from DHCP unless the plan gives it one. It reaches the storage machine at the storage agent's advertised address—its IP address by default, so no DNS is needed unless the advertised host is a name. It reaches the Manager at the address the machine's DC agent uses, resolved to an IP address before the machine restarts. Its first contact goes out moments after the network is set, so it tries for 15 seconds; if the Manager still cannot be reached, it restores without reporting, since the vault credential is in the staged job.
It checks its own view of the machine's firmware and disks against the backup before anything is written, then reads the backup through its read lease and restores every critical volume. It reports each volume's progress to the Manager, using a token that is valid for this one recovery only.
After a successful restore, the recovery agent replaces the DC agent on the restored volume with the build of the agent that staged the restore, keeping the backup's copy beside it, so that the restored domain controller does not come back on an older agent from its backup.
When the Manager has recorded that the restore succeeded, the recovery environment restarts the machine into the restored system. The restored DC agent reconnects, checks that the domain controller works, and the recovery completes.

Because the start from the image is one-shot, a failure before anything is written is safe. If the image does not start, or the recovery environment refuses (for example, because it cannot reach the vault), the machine returns to its own Windows at the next start and the run reports what happened. If the restore fails after writing, or the Manager does not confirm the result, the recovery environment stays at its prompt for someone to look at.

The DC agent must be able to stage the image, so the machine's own Windows has to start one last time. A domain controller whose Windows no longer starts is recovered onto a candidate machine instead, where the candidate's own DC agent stages the image, or onto a new VM. See Restore Methods.

Booting from a recovery ISO onto a new VM

A full system recovery onto a new VM, and every test restore, needs no Windows or agent on the target. When the run is executed, the storage agent builds one ISO for that run from the installation media's boot files, the recovery image, and the job. The new VM starts from that ISO, and the recovery agent enrolls with the Manager using a single-use token the ISO carries; the vault credential is handed over only after it enrolls. Because the restore cannot start without the Manager, the recovery agent keeps trying its first contact for 90 seconds, which rides out a network that is still coming up. Before anything is written, the recovery agent checks the VM's firmware and disks against the backup.

In a test restore, the recovery agent powers the VM off after the restore instead of restarting it, so the Isolated Recovery Host can isolate the VM before its first start. See Full System Recovery onto a New VM and Test Restores.

Recovery image status

The Recovery image column on the OS Media screen states, for each operating system your domain controllers run, whether a domain controller on it can use full system recovery. An Available status means an image exists and will be used; image untested means no restore has proven it yet, which is normal for a new image until a full system recovery or test restore completes with it. An Unavailable status names what is missing, such as media to build from or an image folder.

Every unavailable status affects full system recovery only. A DC state recovery of a domain controller whose Windows still starts does not need a recovery image. For each status and what to do about it, see OS Media.