Alerts Reference

This topic lists every alert the Manager raises, what raises it, and what to do about it. Alerts appear on the Alerts screen and in the notifications menu in the top bar, and cdr alert list returns them. For the screen, see Alerts.

The Manager raises one alert per condition and names up to ten of the affected objects. Domain controllers recorded as removed from the directory appear in no alert, and neither do domain controllers that are rebuilt on recovery: those with no enabled backup policy and nothing in the vault. Objects that an operator has excluded raise no alerts; they are counted separately on the Alerts screen. The exceptions are the alerts about the backups themselves and the staging share, which are raised whether or not the domain controller is excluded: Vault content whose hash does not match, Committed vault content changed outside retention, and Unexpected SMB connections to the staging share. The code shown under each title is what cdr alert list and its --json output use.

Critical alerts

A critical alert means your ability to recover is impaired now.

Alert	Raised when	What to do
Backups and recovery operations stopped: no active license `license.stopped`	No license is installed, the installed key does not license Cygna Forest Recovery for Active Directory, or the license ended and its grace period is over. New backups, restores, recovery plans, and test restores are refused; work already under way finishes. The alert names the license's standing in one sentence.	Apply a current license key on the License screen or with `cdr license apply`. Contact Cygna for a renewal. See Licensing.
License ended: grace period running `license.grace`	The license's end date has passed. Everything still works until the grace period is over, then backups and recovery operations stop.	Apply a renewed license key on the License screen or with `cdr license apply` before the date named.
Domains that could not be recovered `domain.unrecoverable`	No writable domain controller in the domain holds a restorable backup, so a recovery has nothing to restore its first DC from.	Back up at least one writable domain controller in each domain with a full system backup. A read-only DC cannot start a recovery.
Forests that could not be recovered `forest.unrecoverable`	The forest root domain cannot be recovered, or a domain under it cannot.	Open the Forests screen, which says for each forest whether the root or a child domain is the problem, and fix the domains named.
Domains past the tombstone lifetime `domain.beyond-tombstone`	The newest restorable backup of every writable domain controller in the domain is older than the tombstone lifetime. Restore preflight refuses such backups.	Take a fresh backup of a writable DC in each domain, then find out why backups stopped.
Domains replicating SYSVOL by FRS `domain.sysvol-frs`	The newest backup reports that SYSVOL replicates with FRS rather than DFSR. Restore preflight refuses a domain on FRS.	Migrate SYSVOL replication to DFSR, then take fresh backups. Backups taken before the migration stay unusable.
Vault refusing backups `vault.full`	Free space on the vault volume is below the free space reserve, so the storage agent refuses every new backup.	Add capacity or reduce retention. Backups are failing now, and recovery capability degrades for as long as this lasts.
Domain controllers whose latest test restore failed `restore.verification-failed`	The newest test restore of the domain controller's backups, on an Isolated Recovery Host, did not come up as a working first DC: the restore failed, a check did not pass, or it never reported. Test restores that ended without a verdict are ignored.	Open the test restore on the Recovery Plans screen: its verdict names the step and the reason. Then test a newer backup. See Running a Test Restore.
Vault content whose hash does not match `vault.hash-mismatch`	At its last scheduled verification, a committed backup no longer hashed to the value recorded when it was committed: corruption or tampering. A version that could not be read, or has no recorded hash, is not in this alert, because the check was not made. Raised for as long as the version's latest verification result is a failure.	Do not restore from the versions named. Check the health of the vault volume and who has had write access to the storage machine, and take a fresh backup of each domain controller named. Each version's failing checks are on the Backup verification tab of the Repository screen; the content is left in place as evidence. See Vault Protection and Integrity.
Unexpected SMB connections to the staging share `storage.unexpected-smb`	The storage agent saw a use of staging that does not fit a backup job: a job's account used from an address that is not its domain controller's, before its credential was issued or after it was removed, or an account name of that form with no job; or any other account reaching the staging share, another share over the staging folder, or a file under the staging folder through any share, an administrative share such as `D$` included. Raised for seven days after it was last seen. Each finding names the account, the address, and what was wrong.	Treat the job's credential as in the wrong hands until shown otherwise. Find the machine at each address named, check who else has had access to the storage machine, and take a fresh backup of the domain controller whose job it was. The storage agent reports only: it closes no session and removes no account early. The audit trail records the first sighting of each finding as Unexpected use of the staging share; View audit trail opens it.
Committed vault content changed outside retention `storage.mass-delete`	Something other than retention deleted, renamed, wrote to, truncated, or changed the permissions or attributes of a committed backup. A single file is enough. Each version is named with what was done, how many paths, and over how long; `cdr repository changes` also says who did it, where the storage machine's audit policy recorded it. Raised until someone records a review.	Treat the storage machine as compromised until shown otherwise: keep it isolated and find out who has write access to the vault volume. Do not restore from the versions named. List the changes with `cdr repository changes --paths`, and once a version's changes are understood, record the review with `cdr repository changes review <version> --note <text>`, which clears the alert for that version. Nothing was blocked or reverted. See Vault Protection and Integrity.
Domain controllers silent past the tombstone lifetime `dc.replication-beyond-tombstone`	No DC agent's domain controller has replicated from this domain controller for longer than the tombstone lifetime, taken as 180 days, the Active Directory default. Its copy of the directory is too old to rejoin: reconnected, it brings back objects the rest of the forest has deleted, as lingering objects. The alert is raised only for a domain controller that some DC agent's domain controller replicates from; a machine nothing observes is not called silent. The Alerts screen shows when the longest-silent one last replicated.	Keep it off the network. If it is gone, remove it from the directory with `cdr cleanup dc <name>`, or `cdr cleanup domain` when it was its domain's last DC. If it is still needed, rebuild it—forced demotion, metadata cleanup, then promote it again—rather than repairing its replication.

Warning alerts

A warning means you can recover today, but for a reason that may not hold tomorrow.

Alert	Raised when	What to do
Mass removals from the directory held for review `topology.removal-held`	More than a quarter of a forest's domain controllers or domains, or of a domain's trusts, disappeared from the directory at once. Discovery records removals on its own, but not this many: it is what an attack, a mistaken cleanup, or a permissions change that hides objects from every agent looks like.	Confirm in the directory that each named object really is gone. If it is, record each one as removed with `cdr cleanup retire <name> --kind <kind> --reason <text>`, pasting the name as the alert lists it. If it is not, find out why the agents stopped seeing it.
Objects removed from the directory `topology.removed`	Discovery recorded a domain controller, domain, or trust as removed, because every agent with a current, complete view of the directory stopped seeing it. Raised for seven days after the removal.	Confirm that each removal was intended. An unexpected one is a security finding: the object is no longer counted, although recovery plans can still use restore points taken before it went.
Backup targets with nothing to restore from `dc.unprotected`	A domain controller with an enabled backup policy has no restorable backup: the first backup has not completed, every backup failed, or every version was pruned. A domain controller with no enabled policy and nothing in the vault is not a backup target and is in no alert, because recovery rebuilds it. Whether a missing backup costs the domain its recovery is decided by Domains that could not be recovered, which is raised on its own.	Open the domain controller's backup jobs to see why nothing is landing, then run a backup. See Backup Targets and Rebuilt Domain Controllers.
Global catalogs with no agent `dc.no-agent`	A global catalog has no enrolled agent, and its forest holds another domain this installation would recover. Recovering any other domain in the forest reconciles every global catalog through its agent, so that recovery's plan refuses to start until this one has an agent or stops being a global catalog. Domain controllers that are not global catalogs need no agent unless they are backed up.	Install the DC agent, or remove the machine's global catalog role.
Domain controllers whose last backup failed `backup.failing`	The most recent backup of a domain controller failed. An older restore point still exists, so no data is lost yet. A backup refused because the domain controller had no enrolled agent yet does not count as a failure.	Open the job on the Backup Jobs screen for the recorded error and transcript. See Backup Job Errors.
Backups past their freshness window `backup.stale`	A domain controller's newest restore point is older than twice its policy interval.	Check that the agent is connected, then run a backup.
Domain controllers with no backup schedule `backup.unscheduled`	A restore point exists but no enabled policy keeps it current, so it only gets older.	Give each one an enabled policy on the Backup Policies screen.
Domains with no full system recovery option `domain.mode-a-only`	Every restorable backup of a writable domain controller in the domain is a DC state backup, which cannot recover a machine whose operating system is gone.	Change the policy of at least one writable DC in each domain to Full system.
Topology stores that are stale `topology.stale`	No DC agent in the forest has reported the directory for longer than twice the topology report interval, so inventory figures are only as current as the last report.	Check that a DC agent is running and connected in each forest named.
Vaults with no room for another version `vault.tight`	The vault has free space, but not enough for another copy of the largest version it holds, so the largest domain controller's next backup will not fit.	Add capacity or reduce retention before the next backup of the largest DC.
Domain controllers with no recent verified backup `restore.unverified`	The domain controller's newest backup has not passed a test restore, and its backups have gone untested for longer than the Test each DC's new backups within period. Raised only while an Isolated Recovery Host is enrolled, and only for domain controllers with full system backups. Setting the period to 0 turns this alert off.	The schedule starts each test in time to pass before this deadline, so a domain controller named here was not reached in time: a host that is not ready, backups that keep it waiting, failing tests, or more domain controllers than the hosts can test in a period. Add a host or lengthen the period if it keeps happening; check the host's panel on the Recovery hosts tab of the Agents screen. You can also start one yourself from the Recovery Plans screen. See Running a Test Restore and Test Restores.
Exclusions due for review, or voided `exclusion.lapsed`	An exclusion reached its review date, or the object it was written about changed identity. An exclusion on a domain controller or domain since recorded as removed is left out.	Review each one: extend it with a fresh reason, or clear it and deal with what it was hiding.
Staging that could not be created `staging.mint-failed`	The storage agent could not give a domain controller's backup a staging area or a per-job write credential, and no later backup of it has been given one. Raised until a later backup of the domain controller is. Not raised for a domain controller that is excluded or no longer in the directory. A backup refused for lack of space raises Vault refusing backups instead.	Open the failed jobs on the Backup Jobs screen; each records the storage agent's reason. The cause is on the storage machine—the storage agent's rights, its password policy, the Cygna Forest Recovery local groups, or the staging volume—so retrying the backup alone does not clear it. See Backup Job Errors.
krbtgt second reset due `recovery.krbtgt-second-reset`	A domain in a running or halted recovery plan has had its first krbtgt password reset and not its second. Tickets forged with the old key stay usable until the second reset. Raised for the whole wait.	Let the plan run its second reset, or, on a halted plan, resolve the halt so that it can. The plan does not reconnect to production until the second reset is done. If a plan is aborted between the two resets, its recovery report lists the second reset as a follow-up.
Recovery protection override active `recovery.protection-override`	In a running or halted recovery plan, seizing the operations master roles set `Repl Perform Initial Synchronizations` to 0 on a domain controller, and the plan's step that sets it back to 1 has not run yet. The alert is read from the plan, not from the domain controller: a value changed by hand is not seen, and an aborted plan is not followed.	Let the plan's cleanup step run. If the plan will not reach it, set `HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\Repl Perform Initial Synchronizations` back to 1 on each domain controller named, once its replication partners are healthy.
Storage agents that are not the storage machine `storage.candidate`	A storage agent enrolled with the enrollment token while another is the storage machine. It is given no work: no backups, no restores, and no configuration backups.	If it is meant to replace the storage machine, replace it from its row menu on the Agents screen or with `cdr storage replace`; that ends every restore point and needs both `agents.manage` and `backup.manage`. If nobody installed it, uninstall the agent on that machine, delete its record with `cdr agent forget`, and treat the enrollment token as known to whoever did. See Moving the Storage Machine or an Isolated Recovery Host.
Requests waiting for a second operator `request.pending`	An action that needs two capabilities, such as replacing the storage machine, was asked for by an operator who holds only one of them. Nothing happens until an operator holding the other approves it, and it lapses after 24 hours.	An operator holding the capability named approves or declines it on the Repository screen, or with `cdr request approve` or `cdr request decline`. The requester can cancel it.
License ending soon `license.expiring`	The license ends within 30 days. A subscription then has a 14-day grace period; a trial stops the day after it ends. The alert names the end date.	Contact Cygna for a renewal and apply the new key on the License screen or with `cdr license apply`. See Licensing.

Could not find out

These alerts report something the Manager could not confirm. They are an absence of information, not a fault, and are counted as neither.

Alert	Raised when	What to do
Enrolled agents that are not connected `dc.agent-disconnected`	An agent has enrolled but is not connected now, so its figures are the last it reported.	Check that the agent service is running on each machine and that it can reach the Manager. See Agent Enrollment and Connectivity.
Forests never enumerated `topology.never-enumerated`	No DC agent has ever reported the forest's directory, so its inventory is only what machines claim.	Install a DC agent in the forest. It reports the directory when it connects.
Vault volumes that could not be measured `vault.unreadable`	The storage agent could not read the free space of its vault volume, or has never reported it. An unreadable volume is not a full one.	Check that the volume is present and that the storage agent can read it.
Storage machines not fully watching the staging share `storage.smb-watch-unavailable`	The storage agent's watch on its staging share is off or partial, or it has not reported on it for 30 minutes. The reasons: the Security log cannot be read; the audit policy does not record logons, share connections, or file access through shares; the poll of sessions and open files fails; or running jobs have no known domain controller address to check against. Each machine is named with its reason. While a machine is in this alert, the absence of Unexpected SMB connections to the staging share from it is not evidence that there were none.	Act on the reason given. Cygna Forest Recovery never changes the audit policy: for every connection to be seen, however short, turn on success auditing for Logon, File Share, and Detailed File Share in the storage machine's Advanced Audit Policy. See Turning on auditing on the storage machine below.
Vaults not being watched for changes `storage.vault-watch-unavailable`	A storage machine cannot tell whether its committed backups are being changed: the change journal of the vault volume cannot be read (a file system without one, a network share, or a storage agent that is not running elevated), the storage agent has not reported recently, or the journal lost records in the last seven days, so that stretch of time was checked by comparing the vault with its commit records rather than watched. Each machine is named with its reason.	Act on the reason given. Keep the storage root on a local NTFS or ReFS volume, where the storage agent can always read or create a change journal, and keep the storage agent service running as LocalSystem. For a journal gap, `cdr repository changes` lists what the comparison found. Scheduled hash verification continues either way.

Notices

A notice tells you about something that waits on an operator but does not affect recovery. Notices are listed under Notice on the Alerts screen and are not counted in the bell's badge.

Alert	Raised when	What to do
Isolated Recovery Hosts awaiting approval `recovery-host.awaiting-approval`	An Isolated Recovery Host agent enrolled and no operator has approved it, so test restores are not sent to it however ready it is.	If it is yours, approve it on the Agents screen or with `cdr recovery-host approve`. If it is not, uninstall the agent and delete its record. See Moving an Isolated Recovery Host.

Turning on auditing on the storage machine

Storage machines not fully watching the staging share names each machine with its reason, for example Audit File Share is off in this machine's audit policy. Cygna Forest Recovery never changes the audit policy, so turn the auditing on yourself. For full coverage of the staging share and the vault, turn on success auditing for Logon, File Share, Detailed File Share, and File System.

The storage machine is not joined to a domain, so set this in its local policy. You can use a command prompt or Local Security Policy.

To turn on the auditing from a command prompt:

On the storage machine, open a command prompt as an administrator.

Turn on success auditing for the four subcategories. The subcategories are given by their GUIDs, which work in every Windows language:

auditpol /set /subcategory:"{0CCE9215-69AE-11D9-BED3-505054503030}" /success:enable
auditpol /set /subcategory:"{0CCE9224-69AE-11D9-BED3-505054503030}" /success:enable
auditpol /set /subcategory:"{0CCE9244-69AE-11D9-BED3-505054503030}" /success:enable
auditpol /set /subcategory:"{0CCE921D-69AE-11D9-BED3-505054503030}" /success:enable

In order, these are Logon, File Share, Detailed File Share, and File System.

Check the result:

auditpol /get /subcategory:"{0CCE9215-69AE-11D9-BED3-505054503030},{0CCE9224-69AE-11D9-BED3-505054503030},{0CCE9244-69AE-11D9-BED3-505054503030},{0CCE921D-69AE-11D9-BED3-505054503030}"

Each of the four lines should show Success (or Success and Failure).

To turn on the auditing in Local Security Policy:

On the storage machine, run secpol.msc.
Go to Security Settings > Advanced Audit Policy Configuration > System Audit Policies - Local Group Policy Object.
Under Logon/Logoff, open Audit Logon, select Configure the following audit events and Success, and select OK.
Under Object Access, do the same for Audit File Share, Audit Detailed File Share, and Audit File System.

If the machine also has basic audit policies set under Local Policies > Audit Policy, check that Local Policies > Security Options > Audit: Force audit policy subcategory settings (Windows Vista or later) to override audit policy category settings is Enabled, which is the default. Otherwise Windows ignores the subcategory settings.

You do not need to add audit entries to any folder: the storage agent adds the one that File System auditing needs on each committed backup. Detailed File Share logs every access to every share on the machine, so increase the maximum size of the Security log to keep enough history.

The storage agent reads the audit policy every 10 minutes, so Storage machines not fully watching the staging share clears without a restart once it sees the change, unless the alert gives another reason. To have it read the policy at once, restart the storage agent service, CygnaDRStorageAgent, while no backup is running.

For what each subcategory adds, see Audit policy on the storage machine.

Alerts by email

The Manager emails alerts to the Recipients in the Alert notifications settings, through the mail server in the Email settings. With no recipients, alerts are not emailed. The email reads the same alerts as the Alerts screen, so it never names an alert the screen does not show.

Per object—Each object an alert names is reported on its own. A domain controller that joins an alert already raised for others is reported as raised; one that no longer matches is reported as cleared while the alert stays raised for the rest.
Raised, cleared, still raised—The Manager checks at the Check interval. An object is raised when a check first finds it and cleared when two checks in a row no longer find it. One email per check lists everything raised and cleared, and below that what is still raised. Nothing is sent when nothing changed and no reminder is due.
Reminders—An object still raised is reported again once the Reminder interval has passed since the last email that listed it.
Minimum severity—Alerts below the Minimum severity are not emailed. Raising the minimum stops reports about the alerts it leaves out without reporting them cleared; lowering it reports the alerts it brings in as raised.
Leaving an alert's scope—An object an alert no longer watches is dropped from email, not reported as cleared: a domain controller that stops being a backup target leaves Backup targets with nothing to restore from, and one that no recovery needs an agent on leaves Global catalogs with no agent. Leaving scope is not the same as being fixed.
After a restart—The first check waits five minutes, or the check interval if that is longer, so agents can reconnect first.
Failed sends—An email that could not be sent is held and sent with the next check, so a raised alert is not lost while the mail server is down. The Alerts screen then shows Email not working.

The subject line names the Manager machine, the worst severity listed, and the counts, for example Cygna Disaster Recovery alerts on <machine> (critical): 1 raised, 2 still raised. The email has HTML and plain-text parts and links to the Alerts screen. Times are in the Manager machine's local time. The email names that time zone once, beside when it was checked, with the offset in effect then, for example Times are the manager's local time: Eastern Daylight Time (UTC-04:00). The link uses the Manager machine's DNS name and HTTPS port, or the Console address in emails setting when recipients reach the console by another name. Each object is listed with the alert's title and, for raised objects, what to do; an alert lists at most 100 objects per section, then a count.

Every email sent is recorded in the audit trail as Alert email sent, and a failure as Alert email not sent when it starts and when its reason changes. For the settings and their defaults, see Settings Reference. To check that email works, use Send test email on the Alerts screen (see Alerts) or cdr alert notify test.

Monitoring alerts from other tools

To include Cygna Forest Recovery in your own monitoring, run cdr alert list on a schedule. It exits with code 4 when any alert is raised, and --json returns the alerts in machine-readable form. --critical limits the list to critical alerts. cdr repository changes also exits with code 4 while any change to a committed backup is open or any storage machine is not watching its vault. See Alert, Audit, and Report Commands.