Hi,

We have DPM 2016 running on WS2016 the major PG are VM running on hyper-v cluster 2012 R2, I install DPM, create PG and online protect, all run ok, for prower problems (Puerto Rico hurricane Maria), we need off the datacenter every 2 weekends, but the Monday when the datacenter power on, DPM cannot run any PG and this error is show for the jobs:

 The VHD containing the replica or one of it's snapshots could not be mounted or unmounted. (ID 40002)

 More information
Recommended action: Ensure that the storage containing the VHD is healthy, mounted, and the VHD is not in use by other processes, and retry the operation.

Isaias

Hi

I had receive error as below when backup my Hyper 2016 VM

pls advise.

either the servername needs reboot , is part of a cluster then some/all nodes in the cluster need t be rebooted. 

Hi..

we are running DPM 2016 RU6 to protect our clustered Hyper-V (2016) VMs.

Now I've tried to restore some of them and I'm wondering why the recovered VM lost all her Checkpoints...

Only one vhdx file was restored. I've tried also a network folder target. One vhdx and two config files (vmcx, vmrs) are the only one that were restored.

Does somebody know if this is the default behavior, that only the current selected/running checkpoint will be used for the backup?

Is it possible to enable DPM to backup any VM Source?

Use a file system backup to backup the VM folder are no option for us because the VMs are running on a S2D Cluster and so the storage location of the VM can dynamically changed...

Thanks

Alex

Hi,

I have newly deployed, 3 node, Windows 2016 (Build 2724) Hyper-Converged Hyper-V/S2D cluster which is operating normally. The intention is to use DPM 2016 LTSB UR6 to perform VM level backups hosted on the cluster.

After deploying the DPM agents (5.0.375.0) to the cluster nodes, i've started to see cluster errors generated when rebooting a node, CSV auto pause errors for each CSV in the cluster:

"CSVName has entered a pause state because of 'STATUS_UNEXPECTED_NETWORK_ERROR (c00000c4)"

When rebooting a node, there are no backup jobs running, if I remove the DPM agents the nodes reboot without error.

Also, it may be a red herring, but there are entries in the cluster logs generated during a node reboot that relate to an error closing a file named ' DpmFilterTrace.txt' which is stored on the CSVs. Not sure if this is normal or an indication of an issue.

Would appreciate any input from people protecting workloads on HCI clusters using DPM.

Thanks in advance,

James

Hello.

New day, new DPM problem:

Cannot protect one particular VM using host-level protection. It's the only VM having this problem. The error message (redacted names):

Affected area: RCT\<VMNAME>
Occurred since: 24.05.2019 15:52:46
Description: The replica of Microsoft Hyper-V RCT\<VMNAME> on <HVHOST> is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with
 consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent. 

For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106)
The DPM service was unable to communicate with the protection agent on <HVHOST>. (ID 52 Details: The remote computer refused the network connection (0x800704C9))
More information
Recommended action: 
1) Restart the DPM Replication Agent service on <HVHOST>.
2) If <HVHOST> is configured using certificates, ensure that the DPM CPWrapper service is running on DPM Server and aof0078.ad.apponfly.com. Also ensure that certificates used by both the computers are valid.
Synchronize with consistency check.
Run a synchronization job with consistency check...
Resolution: To dismiss the alert, click below

Can backup all other VMs on the Hyper-V host without issues. Hyper-V 2016 Server protected by DPM 2019. Previously protected by DPM 2016 UR7, where backup of this particular VM worked. HV hosts and DPM are in the same domain.

What I've tried:

 - CC job - no good

 - removed the protected DS and re-add it to DPM - no good. Actually the initial backup 'finishes' without a problem, but the backup has ~100MB (instead of ~40GB). RP count 0. CC job doesn't help.

 - migrated the VM to different host - same issue

 - 'reset' RCT (by shutting down the VM and deleting the .RCT .MRT files) - no good

 - tried monitoring the DPMRA log during the backup attempt: this error comes out, however doesn't makes it any clearer:

1250 16A4 
05/24 13:51:13.117
03 cbackupsubtask.cpp(653)
[000001F7BE491E40] C4FBC1EF-FB64-41D5-9F10-5A44C6B4D7B0
WARNING <Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2147023671" Reason="Error" CommandID="RAReadDatasetFixup" CommandInstanceID="044cde5b-e0d9-4fd3-8c37-a787048583e2"
 GuidWorkItem="e8436d7b-8204-4910-b7a6-d08637ec1f45" TETaskInstanceID="c4fbc1ef-fb64-41d5-9f10-5a44c6b4d7b0"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="998" DetailedCode="-2147023671"
 DetailedSource="2"/><RAStatus><RAReadDatasetFixup xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"><LWVStatus BytesTransferred="1272" NumberOfFilesTransferred="7" NumberOfFilesFailed="0"
 DataCorruptionDetected="false"/><FixupStatus BytesTransferred="0" NumberOfFilesTransferred="0" NumberOfFilesFailed="0" DataCorruptionDetected="false"/></RAReadDatasetFixup></RAStatus></Status>

Any ideas are welcome

Hi,

we've been (mostly) following this article to set up deduplication for DPM, adapted for

standalone machines. The article also contains instructions to set a few registry values:

Set-ItemProperty -Path HKLM:\Cluster\Dedup -Name DeepGCInterval -Value 0xFFFFFFFF

Set-ItemProperty -Path HKLM:\Cluster\Dedup -Name HashIndexFullKeyReservationPercent -Value 70

Set-ItemProperty -Path HKLM:\Cluster\Dedup -Name EnablePriorityOptimization -Value 1
Trouble is, "HKLM:\Cluster" doesn't exist when you don't have a cluster. Searching 
the web for the settings' names showed me a few people who seem  to have a similar 
problem, but no definitive solution, only a guess.
"DeepGCInterval" is the only setting that appears in a Microsoft document. The 
setting would be made in "HKLM\System\CurrentControlSet\Services\ddpsvc\Settings".
My question: can I assume that this is the right place to do ALL three of the 
above registry settings? Is this the place where the dedup service looks for 
settings when it's not running on a clustered machine? The path seems to imply that.
--
Cheers,
pk 

Hi Everyone!

I am kinda stuck on an issue and have not been able to find an answer to that solves the problem. I was wondering if anyone can point me in the right direction.

I currently have a Hyper-V 2016 server that I am backing up using DPM 2016, Now the Hyper-V host its self has roughly about 50 VM's and about 35 of them are backing up without issue however I have about 15 guests that are all in a "Replica is inconsistent" State with the following error in even log :

The replica of <Guest> on <Server> is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. (ID: 3106)
An unexpected error occurred while the job was running. (ID: 104)

Performing a consistency check does nothing and the alert pops back into the console. Any ideas or help ?

Stuart.

Hi,

I have a file server backed up using DPM 2019, that is in an inconsistent state. When I run the consistency check, after a while access to shares on the server is interrupted to the point that files cannot be opened. When I stop the check, access is fine again.

I have ran 2010 and 2019 and have had no issue with this before. 

The server that I am backing up is on a 2012 Hyper V Fail-over Cluster connected to iSCSI SAN. My backup server is on a dedicated server.

Any help with this would be much appreciated.

Thanks,

Mark

Hi,

I have been having a recent issue with DPM 2016 UR7 where my VM backups have been failing and leaving behind "checkpoints" that don't actually exist in Hyper-V.

If I then try to run the backup job again I get faced with Unknown error ID 104. I've seen a few posts regarding this however no fixes that have worked for me.

I have tried rebooting the virtual machines in question to find that they fail there merge. I have updated all protection agents to use UR7 agent (5.0.409.0). Some of the virtual machines will merge once shutdown and backup again successfully but a day or two later I will be faced with the same issue.

Has any experienced anything similar recently?

I would love some ideas on this one, in some cases if I shut down the VM to merge and try to start again I will be faced with power on error, virtual hard disk chain is corrupt of which I then run SET-VHD command to point back to the parent disk. It will then boot and backup.

Just seems like a really odd issue and has only been happening over the course of the past week or 2. I have also noticed that DPM has stopped deleting expired recovery points outside of the retention range but I believe that is a different issue.

Cheers.

Hi @ all,

[all servers involved are 2016, patched up to July CU]

trying to back up Hyper-V VMs running on three separate clusters. All VM configs and disks reside on the same SOFS living on an S2D cluster. The clusters are as follows:

Production: 10 nodes on HP Synergy blades, production VLAN and SMB VLAN sharing the same converged networking infrastructure.

DMZ: 2 nodes on HP rack servers, production VLAN is 4x1GB Cu, SMB VLAN is 10GB optic

Management: 2 nodes on HP rack servers, production VLAN is 4x1GB Cu, SMB VLAN is 10GB optic. This is where the DPM VM resides.

Shares on the SOFS, permissions, delegation and so on are configured in an identical way and differ only in regard to the machine accounts actually granted permissions. All permissions are granted via Active Directory groups.

Virtual machines can be created, managed and live migrated within their respective clusters without any issues.

DPM 2019 10.19.58.0 on a VM with a local SQL2016 instance.

DPM agent is installed on all hosts in all clusters, connectivity shows OK. Replicas of both Windows and Linux VMs can be created without warnings. And here's where it gets interesting:

• Any VMs on the Management Cluster (where the DPM VM is running) can be backed up without issue regardless of OS or power state.
• On the DMZ cluster, an initial replica can be created and shows OK. On attempt to create a recovery point, there is an error saying "Replica is inconsistent". Consistency check ends in error state saying ID 2033 (a required privilege is not held by the client (0x80070522). The UNC path and host shown in the error message are where the VM is stored and registered. This behaviour does not change with the power state of the VM.
• On the production cluster, an initial replica stays green for several seconds then goes red. From there on, the behaviour is identical to taht on the DMZ cluster.

All hosts have been rebooted at least once since DPM agent rollout.

What am I missing?

• added the machine account of the DPM server to Full Access on one of the shares, to no avail
• enabled File Server VSS writer on S2D cluster nodes, to no avail (backup of Management cluster worked before I did that)

Evgenij Smirnov

http://evgenij.smirnov.de

Dears,

can someone advise on how to configure hyperv replication between a cluster of 2 hyper-v in main site and one standalone hyperv in the DR.

many thanks

I have a 4 Node Cluster with 2 Shared Cluster Volumes on an ISCSI SAN.  Each node is up-to-date with Updates and is running Server 2012 DataCenter.  I have a seperate 2012 server with Microsoft DPM on it I use to backup my VMs.  The cluster has 24 diffrent VMs on it all diffrent sizes, OSs,  and configurations.  Recently DPM has been getting hung on one of the VMs.  When the backup gets hung, I get events

5217

Software snapshot creation on Cluster Shared Volume(s) ('\\?\Volume{f3c25587-f296-481e-b2fc-606c4eef1e92}\') with snapshot set id '{dd4a47b9-e645-4303-9df6-c978b29b3cd5}' failed with error 'HrError(0x00000001)(1)'. Please check the state of the CSV resources and the system events of the resource owner nodes.

and

5120

Cluster Shared Volume 'Volume1' ('Cluster Disk 1') is no longer available on this node because of 'STATUS_FILE_CLOSED(c0000128)'. All I/O will temporarily be queued until a path to the volume is reestablished.

And naturally I cannot backup the VM.  All of the other VMs backup fine.

Anyone got any Ideas?

I have been using 2 DPM 2016 servers to backup VMs in a 5 node Hyper-V cluster of 2012 R2 servers, using the Scale-Out Protection (setdpmserver –add –dpmservername serverX).  For the most part, it has been running well for a couple of years.

I recently brought up a new 3 node Hyper-V cluster of 2016 servers, and added the cluster to the same 2 DPM 2016 servers, installing the agent on each of the nodes from 1 server, and then used the setdpm -add -dpmserver serverX command to add the 2nd DPM server.  

Since I added this second Hyper-V cluster, most of the VM backups have been failing on the old 2012 R2 Hyper-V cluster.  I have been able to fix some of them temporarily (error 40003, by removing the last recovery point and error 30111, by temporarily disabling the Backup integration in the VM properties) but the errors are stacking up and I'm thinking I may have to rebuild the DPM environment, sticking to Scale-Out protection of one Hyper-V cluster at a time.

Is this supported - using a pair of DPM 2016 servers to backup multiple Hyper-V clusters using Scale-Out Protection?

Hi,

I have asked this question before however the same problem has occurred and is driving me crazy. Now we have a 2016 Hyper-V environment with Server 2016 VMs. We run daily backups using DPM 2016.

Now I came in after the weekend to find that all of my VM backups had failed and every VM that had failed had a .avhdx file left behind. I have tried shutting down the virtual machine but I get caught out with "Merge in Progress - Failed".

Sometimes when I shutdown the virtual machine it will fail to start and say that there is an issue with the virtual hard disk chain and it needs to be reconnected.

The only way I have found to fix it is to manually merge the virtual hard disks and then re-create the virtual machine. This works and then I can take another backup however a few days later the same issue will happen again.

I don't have time to constantly merge virtual machines every other day and of course it affects the production environment. There have been changes to our group policy and as it is affecting all of our hosts I wonder if this is the problem however I am not sure what to look at. I heard about the "Logon as a service" causing an issue however this made no change for me.

If anyone has any ideas I would love to hear them, currently going through it with Microsoft Support however they don't listen to the issue.

In the past I have been trying to protect a Hyper-V cluster with DPM 2012 and 2016 but the backup jobs were failing continuously. I suppose this could be due to live migration that has been executed from time to time. Then I read this article which states that in order for DPM to maintain protection within a cluster, VMM must be used. Is this correct? I don't have VMM but would like to protect the VMs somehow, what is the recommended method in this case?

We have a DPM 2016 Server that is configured for the VM Cluster Level Protection by installing the DPM Agents in all of the Nodes( Hyper-V Hosts) of the Cluster. And till now the DPM Backup jobs for all of the VM's are fine except only one Windows VM.

For that single VM it is showing the error as "Replica Inconsistent" I have tried running the Consistency Checks but again it is going back to the same error.

Below is the Details of the error

The replica of Microsoft Hyper-V RCT\VMNAME1 on SCVMM VMNAME1 Resources.cluster.domain.top is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent. 

For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106) 

An unexpected error occurred while the job was running. (ID 104 Details: Incorrect function (0x80070001))

 I have tried by disabling and enabling the  “Backup (Volume shadow copy)”  in the Hyper-V Integration Services of the VM by following the below steps. But No Luck.

Step1:- The trick is to first uncheck the Integration Service “Backup (Volume shadow copy)”

Step2:- Run a backup / consistency check

Step3:- And the recheck the backup service.

Please help me in this regard.!!!

Shaun

Any problem with protecting 1 Hyper-V VM using two separate DPM servers? For example, 1 DPM server takes folder level backups on the VM, while another DPM server takes a full VM-level snapshot? Assuming the two jobs run at separate times...

Thinking you can associate a DPMAgent with only 1 DPM server, but wanted to solicit other thoughts. 

Shaun

Hi DPM Admins,

We are running Native Hyper-V / RCT backups for our Virtual Machines

I began receiving DPM Unknown Error 104 error 0x8004100c

An unexpected error occurred while the job was running - unknown error

The backup to tape job failed for the following reason (ID 3311)

The reason I began receiving these errors was due to shared drives being presented to the VM

VM’s with Shared Drives can’t create checkpoints.

It took me a few days to figure out why I kept receiving Error 104 in DPM

Hope this helps someone along the way J

Thanks,

Brandy