Hi I am running a virtual DPM server version 2012 R2 UR9

The only workloads I backup are hyper-v workloads. A key feature which I depend on is the Item Level Recovery. This is because the majority of workloads I backup and need to restore are small files inside large volumes. Without ILR it would take a long time to restore the vhdx just to get a small file back.

I am having an issue where the ILR feature simply stops working randomly for a virtual machine. I get DPM error codes 958 and 0x80990F4E

What is strange is I can go back to an earlier recovery point and browse the vhdx with ILR. For instance a VM I am currently protecting I can browse with ILR all recovery points from Monday 22nd of February and earlier. All recovery points after this date cannot open with ILR and I get the before mentioned error.

I have checked the virtual machines which encounter this problem and there are no changes to their volume type, they are all just simple volumes formatted with NTFS.

Is there a solution to this issue? It is a real risk not being able to recovery items in a timely fashion.

Hi

I'm experiencing a problem trying to restore a hyperv backup from tape.
The vhdx file is 5T which utilizes 2 tapes to backup - the backup prompts for the second tape and the backup is successful.

It seems like the restore gets to the end of the first tape and then fails before prompting for the second tape. I have tried this multiple times and receive the same result each time.

I would like to know if anyone has successfully been able to restore a (HYPERV Backup) that spans multiple tapes  ?

error details below. 

Occurred since:                2/25/2016 5:09:45 PM

Description:       The recovery jobs for Microsoft Hyper-V XXXXL02 that started at Thursday, February 25, 2016 9:42:07 AM, with the destination of XXXXH03.thl.int, have completed. Most or all jobs failed to recover the requested data. (ID 3111)

               DPM has unexpectedly reached the end of the tape NAS HyperV Backup-00000003. This could be because the tape was tampered with. (ID 23043)

                More information

Recommended action:  If you have another copy of this tape, then retry recovery using the copy.

                On the Jobs tab in the Monitoring tasks area, group jobs by type to view details of the recovery jobs.

                Retry the recovery job...

Resolution:        DPM automatically changes this alert's status to inactive 10 days after it is issued.To dismiss the alert, click below

                Inactivate

I see many of these post all over google and none with any luck of solutions, so I may be shooting in the dark. Any assistance is much appreciated in advance.

I have several VMs from different 2012 R2 hosts being backed up with DPM 2012 R2. The VMs are Win 2008 R2.

Most of the VMs allow you to dig into the VHDx files without issue. A few of them give DPM Error ID 958 (0x80990F4E).

Basic Configuration:

System Center Data Protection Manager version 2012 R2

OS - MS Server 2012 R2

Storage - DAS - Dell MD3220

Hyper-V - OS MS Server 2012 R2 - Clustered

Virtual Machine - MS Server 2008 R2

The backup of one VM has ran fine for over a month, then two days ago it just stopped working.  The is approx. 350 GB in size with 100 GB free.  The Storage location that DPM uses has over 5 TB free.

This is the error that I keep getting: is 

The VSS application writer or the VSS provider is in a bad state. Either it was already in a bad state or it entered a bad state during the current operation. (ID 30111 Details: VssError:The specified object was not found.
 (0x80042308))

When I run vssadmin list writers on the host all writers show last error as No error, including Microsoft Hyper-V VSS Writer and Cluster Shared Volume VSS Writer.  

I am able to backup using DPM 3 other VM on the same Node.  

I Live Migrated this VM to the other node and got the same error.

I have rebooted the host same error.

I have deleted and recreated the vm same error.

I have verified that the integration services are up to date because the message comes up as This computer is already running the current version of integration services (version 6.3.9600.16384).

If I use Hyper-V Manager I can create manually a checkpoint I can also delete the check point.

Any help on this matter would be greatly appreciated.



Hi,

I've been having loads of issues with DPM lately, it doesn't seem to be backing up one of my HYPER-V server after its initial copy.

Today I managed to get a backup but it wont let me browse the content of the VHD. Its comes up with the error saying

Cannot browse because

1) VHD contains a dynamic disk (it doesn't)
2) VHD No volume (it does)
3) HYPER-v not installed (It is)
4) automatic mounting is disabled (it isn't)

Problem is the first backup mounts ok, the ones I did today don't. I am recovering to a file share to test if the recovered VHD are valid.

Anyone got any ideas why this is happening.

We are planning to upgrade our clustered hyper-v hosts from 2012 to 2012R2. There are many walkthroughs on how to do this online already. However something one of my fellow engineers brought up is our DPM backups.

Once the cluster name is changed to the new cluster. How do I change this in DPM so the backups continue as normal? 

The cluster name is shown as DCI-CLU-01 when the servers are upgraded and the new cluster is created it will be DCI-CLU-02 and DPM will no longer see the VMs as it is looking for the old cluster name.

How can I remedy this?

Hi

I have one server out of my 170 VM's that keeps failing no matter what I do.

The VM ran fine for a long time, then started making trouble.
I deleted the protection and even the data, and I got it working again for some time.

It started failing again, and for other reasons, the DPM got reinstalled, and everything setup again from scratch, and the VM still fails creating its replica, and consistency checks there after.

While typing this message, I moved the VM to another host in the cluster, and voila the consistency check succeded.

I have just been through the eventlogs and can't seam to find where I should look for errors related to this, do someon know which log things like this would land in?

Regards Lars.

Hi;

  I am trying to recover a VM to a network folder to a different Hyperv Server and have it run. Issue I am running into is I can copy the vm to a network folder and the vm drive is intact. However none of the network configuration comes over with it. Is there a way with Powershell commands to export the network configuration of this vm and then use powershell to import it?

Thanks

Bill

Hi. I have a SC DPM 2012 R2 CU9 and i have a big problem.

04/06 - I created a protectiongroup. First recovery point;

04/08 - DPM created second recovery point;

04/10 - DPM created third recovery point;

04/14 -A restore point 04/06 has disappeared.

In a protection group settings i have a 6 days of backups.

A restore point 04/06 is gone.

The free space on DPM Volume is present.

NO errors, no fails. Just gone. No alerts on the SCOM.

I don't understand this problem. Plz, help. Backups is very important for our work.

How to trust DPM if it deletes backups?

Good day

I'm running DPM 2012R2 as a POC in my environment. So far it's going great I can do the basics such as deploy agents, backup and restore files and folders and do complete BMR recoveries.

I'm having an issue backing up my hyper-V 2012 R2 cluster VMs. I've installed the agent on each host but I do not see the VMs underneath each host when I go to the Protection workspace to set up a new Protection group. All I see underneath a Hyper-V host is All Shares, All Volumes, System component and Hyper-V Host component. DPM does not list the virtual servers running on that particular host. What am I missing here?

It's DPM 2012R2; my Hyper-V test cluster is 2012 R2 using CSV disks running on a NetApp storage array.

Any pointers are appreciated.

Environment:

We have a Cluster of 3 VMHost, Host A,B,C.

We use DPM 2012 R2 to backup the VM host and the Cluster as a whole.

Issue: VMHost A failed, so we replaced with a different model machine (D). We change the name of VMHost D to the same name of VMHost A. DPM is now failing to backup our VM even though they are in different VMHost in the cluster.

Question: Why doesn't DPM register the new machine with the same name as part of the cluster?
How do I handle the current machines that are no longer being back up?

Hi all, I have a "True or False" question. If I have a six node Hyper-v 2012R2 cluster using a NetApp SMB share, defined in VMM, for virtual machine .vhdx file storage, will I be able to back up all of my virtual machines using a virtualized DPM2012R2 server?? I read where you have to install a DPM agent on the hyper-v nodes AND the smb file server. You can't install that agent on the NetApp.

One other question is: Can I define a DPM storage pool on a Netapp share if I define the share in VMM and then use VMM to create a disk for the DPM server on that share? 

Any feedback appreciated. Thanks.

Hi

Since a few days i got the following error when i try to back up VMs with DPM 2010.

"Triggering synchronization on \Backup Using Child Partition Snapshot\servername failed:
Error 30148: DPM was unable to access the cluster resource group hostname.domain for \Backup Using Child Partition Snapshot\servername of protection group abcde
Recommended action: Verify that the cluster resource group hostname.domain is accessible by using the Cluster Administration console."

The Protection Group protects seven VMs. Four of them fail with the error mentioned above, the other three work fine. there are other protection groups which protect VMs without any problems. The failed VMs run on different CSVs and on different Hyper-V.

Some Information about our Infrastrucutre:

- two windows server 2008 r2 hyper-v, configured as a cluster (all updates installed)

- three CSVs

- one DPM 2010 running on windows server 2008 r2 (all updates installed)

can anyone help? i saw there is another topic for this error, but the answers there didn't help.

regards from switzerland

sandro

Hi,

Disk space is being eaten up quite quickly on my CSVs and I can't see where its going.  I think it might be VSS but how do I tell?

My environment is Server 2012 cluster with iSCSI storage on Dell EqualLogic holding the CSVs.  It is being backed up by DPM2012 SP1 RU1.

I am getting diskspace warnings from the SAN that don't match up with whats on the disk.  On a 3TB CSV, Explorer tells me there is 1.81 TB in use, Cluster manager tells me there is 777GB free (2.22 in use) but if I do properties on the volume it says 329GB free space which matches what the SAN is reporting.

How can I see what space is being used by VSS, and how can I recover some of this space, assuming it is DPM / VSS that is using it?

Cheers,

James

Hi All,

First of all, I know it's a beta and these are the perils of being an early adopter, but I've got a serious problem.

I've upgraded our production Hyper-V cluster to Server 2012. The setup is a 4 node cluster running CSVs on an ISCSI SAN with MPIO via dual gigabit Ethernet networks. The SAN storage is provided byOpen-E DSS7 and replicated to another server in a different building.

Post the upgrade everything about the cluster seemed stable and to work as expected - live migrations etc all working. I then turned my attention to backups, and I discovered that Server 2012 wasn't supported by DPM. Fortunately there is a beta of DPM 2012 SP1 which adds support for Server 2012, unfortunately there is no upgrade path from the beta to RTM of SP1. Not wanting to upgrade our production DPM server to a beta, I installed a copy of DPM 2012 SP1 beta on a VM to provide a stopgap backup solution for VM level backups of certain machines that couldn't be backed up in other ways. I realise that running the backup server on the same cluster / SAN as the stuff that's being backed up is an odd thing to do, but this at least serves to provide snapshots, SAN replication provides resilience, and like I say, this is a stopgap.

Then I started noticing problems. First symptom was that on starting / rebooting VMs, sometimes other VMs would hang for perhaps 30s - 2m, people would start complaining that SharePoint had gone unresponsive etc. However, they would come back to life in a minute or two.On a couple of occasions we came in in the morning to find a number of VMs off or paused (backups ran overnight). Both of these problems occurred only when the DPM server was turned on. I thought the issue might be general load on the SAN, having both the backup server and the machines being backed up living on the same CSV / hardware. I moved the DPM server to a different ISCSI box and put on aggressive throttling (200Mbps) to try to reduce load, but the problem continues.

The event logs on the Hyper-V cluster suggest I/O timeouts to the SAN at the times of the backups. Lot's of event ID 1069, 1205, 1146, 1230,  (various cluster resources failed). The interesting one I think is 5120 Cluster Shared Volume 'Volume5' ('VOLUME NAME') is no longer available on this node because of 'STATUS_CLUSTER_CSV_AUTO_PAUSE_ERROR(c0130021)'. All I/O will temporarily be queued until a path to the volume is reestablished.

Is anyone else using SP1 beta to successfully backup a 2012 Hyper-V cluster? 

Is anyone seeing the same problem?

Is it likely that this is a problem with SP1 beta, will it be fixed at RTM?

Any suggestions for a stopgap solution?

I think I might try setting up a test physical DPM server to check the issue isn't in someway related to the fact that the DPM server sits on the same cluster it's backing up. I'm also happy to consider the problem could lie elsewhere i.e. with the SAN storage (this was upgraded from v6 to v7 at the same time as the 2012 upgrade, but as soon as I tell the vendor that the problem relates to running a beta of DPM they will be pointing fingers at that.

Thanks,

Tim

Does anyone have any tips for making my Azure Online Recovery Points take less time to complete?  I have some that take on average about 8 hours.  How can I make these shorter?  I imagine I need to create a better scheduling model for my protection groups, but I could use some advice.

Things to know about our env.:

• We have a fast site-to-site connection with Azure of 1 Gig.  Actual throughput is around 500 mbit up/down.
• We have our DPM servers virtualized in Hyper-V.  Each has a minimum of 8 GB of RAM, 4 CPUs, Win 2012 Std.
• We use Clustered Shared Volumes.
• Protection Groups like this one (shown below) creates an online recovery point every 24 hours. Some PGs are creating Hyper-V recovery points and others are creating SQL recovery points. I am not sure if that matters.  The below pic is filtered on one job that historically takes a long time.

Any advice is appreciated.

David Jenner IT Systems Engineer Colonial Williamsburg Foundation

Does anyone know if DPM will be dropping the requirement to have an agent installed on the SMB server presenting storage shares to Hyper-V servers?

We want to back up our Hyper-V VMs and we are using a storage technology that presents SMB to the Hyper-V host but is not running windows. Since it is not running windows I can't install the DPM agent on it.

I have tried installing TP5 and testing but DPM seems to still want this agent installed on the SMB server... Any ideas?

A+, N+, MCP

Hi,

I've had issues with ILR on my DPM box for a while now. Its comes up with the eroro stating about dynamic volumes, no volumes, DPM role or auto mounting not being enabled. The ID is 958 and the error code is 0x80990F4E.

What I have found is that it wont do a ILR mount on a server with more than one disc. I have a couple of servers where I've stuck the data on a different VHDX and these aren't viewable in the recovery console. Single VHDX servers go in fine.

When I mount a single VHDX server I get in the event log

I don't get any log entries on servers with the issue.

Looking at the DPM  logs when it fails I see

15:28:46.345 22 nativevhdprovider.cpp(174) [0000000000A26A60] 99303538-624D-4F25-AB30-45F6D758E77D WARNING Failed: Hr: = [0x80070013] : c:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\DpmVhds\bf56ca19-a372-4f2b-94fc-c0cfd3e861b5.diff.vhdx: lVal : pfnCreateVirtualDisk(&storageType, ssDiffVhdPath.PeekStr(), VIRTUAL_DISK_ACCESS_CREATE, NULL, CREATE_VIRTUAL_DISK_FLAG_NONE, 0, &createParam, NULL, &hDisk)
188C 2088 05/11 15:28:46.345 31 hypervvhdhelper.cpp(155) [00000000009C26A0] 99303538-624D-4F25-AB30-45F6D758E77D WARNING Failed: Hr: = [0x80070013] : m_pVHDProvider->CreateDifferencingVhd() path:[c:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\DpmVhds\bf56ca19-a372-4f2b-94fc-c0cfd3e861b5.diff.vhdx], parent path:[c:\Program Files\Microsoft System Center 2012 R2\DPM\DPM\DpmVhds\bf56ca19-a372-4f2b-94fc-c0cfd3e861b5.vhdx]
188C 2088 05/11 15:28:46.345 31 hypervvhdmanager.cpp(340) [0000000000A32A90] 99303538-624D-4F25-AB30-45F6D758E77D WARNING Failed: Hr: = [0x80990e52] : CreateDifferencingVhd: ssParentPath [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy389\76b21287-0f74-4580-bb7a-eb1f55c32a6c\Full\D-Vol\Email\Exchange2013\Skinner2_BEDFAB9E-DD04-417E-BE5C-8E042B21F78E.avhdx]
188C 2088 05/11 15:28:46.345 31 vhdmanager.cpp(270) [0000000000A32A90] 99303538-624D-4F25-AB30-45F6D758E77D WARNING Failed: Hr: = [0x80990e52] : Encountered Failure: : lVal : MountVhdInternal(ssVhdIdentifierInCache, ssParentPath, psErrorDetail)
188C 2088 05/11 15:28:46.345 61 vhdutils.cpp(78)  99303538-624D-4F25-AB30-45F6D758E77D NORMAL arrVhdChain[0] is [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy389\76b21287-0f74-4580-bb7a-eb1f55c32a6c\Full\D-Vol\Email\Exchange2013\Skinner2_BEDFAB9E-DD04-417E-BE5C-8E042B21F78E.avhdx]
188C 2088 05/11 15:28:46.345 61 vhdutils.cpp(78)  99303538-624D-4F25-AB30-45F6D758E77D NORMAL arrVhdChain[1] is [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy389\76b21287-0f74-4580-bb7a-eb1f55c32a6c\Full\D-Vol\Email\Exchange2013\Skinner2.vhdx]
188C 2088 05/11 15:28:46.345 61 inquirysubtask.cpp(1222) [0000000000A2C160] 99303538-624D-4F25-AB30-45F6D758E77D WARNING Failed: Hr: = [0x80990f4e] Ignoring processing error, processing continues for other nodes
188C 2088 05/11 15:28:46.345 61 inquirysubtask.cpp(1705) [0000000000A2C160] 99303538-624D-4F25-AB30-45F6D758E77D WARNING Failed: Hr: = [0x80990f4e] CInquirySubTask::CreateAndLogError::Adding inquiry error for component
188C 2088 05/11 15:28:46.345 61 inquirysubtask.cpp(1031) [0000000000A2C160] 99303538-624D-4F25-AB30-45F6D758E77D NORMAL <--CInquirySubTask::ExecuteInquiry
188C 13FC 05/11 15:28:46.345 61 inquirysubtask.cpp(458) [0000000000A2C160] 99303538-624D-4F25-AB30-45F6D758E77D NORMAL Sending final response with 1 records
188C 13FC 05/11 15:28:46.345 61 vhdutils.cpp(51)  99303538-624D-4F25-AB30-45F6D758E77D WARNING Failed: Hr: = [0x80990f4e] : \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy389\76b21287-0f74-4580-bb7a-eb1f55c32a6c\Full\D-Vol\Email\Exchange2013\Skinner2_BEDFAB9E-DD04-417E-BE5C-8E042B21F78E.avhdx: lVal : E_COORD_HYPERV_VHD_MOUNT_FAILED
188C 13FC 05/11 15:28:46.345 61 nevhd.cpp(124)  99303538-624D-4F25-AB30-45F6D758E77D WARNING Failed: Hr: = [0x80990f4e] : Encountered Failure: : lVal : VHDUtils::MountAndEnumerateVHD(ssId, arrVhdChain, arrVolLabel, arrVolGuid, arrVolId)
188C 13FC 05/11 15:28:46.345 61 nevhd.cpp(61) [0000000000A2BBA0] 99303538-624D-4F25-AB30-45F6D758E77D WARNING Failed: Hr: = [0x80990f4e] : Encountered Failure: : lVal : EnumerateVHDChildren(m_path, traversingReplica, callback)
188C 13FC 05/11 15:28:46.345 61 inquirysubtask.cpp(990) [0000000000A2C160] 99303538-624D-4F25-AB30-45F6D758E77D NORMAL CInquirySubTask::Inquiry finished with status [0000000000]
188C 13FC 05/11 15:28:46.345 03 workitem.cpp(296) [0000000000A2A880] 99303538-624D-4F25-AB30-45F6D758E77D ACTIVITY WorkItem stopping
188C 13FC 05/11 15:28:46.345 31 vsssnapshotrequestor.cpp(122) [0000000000A2C4A8] 99303538-624D-4F25-AB30-45F6D758E77D NORMAL CVssSnapshotRequestor::~CVssSnapshotRequestor [0000000000A2C4A8]
188C 13FC 05/11 15:28:46.345 31 vsssnapshotrequestor.cpp(2182) [0000000000A2C4A8] 99303538-624D-4F25-AB30-45F6D758E77D NORMAL CVssSnapshotRequestor::CleanUp [0000000000A2C4A8]
188C 13FC 05/11 15:28:46.345 31 vsssnapshotrequestor.cpp(1768) [0000000000A2C4A8] 99303538-624D-4F25-AB30-45F6D758E77D NORMAL CVssSnapshotRequestor::ReleaseVolumesForSnapshot [0000000000A2C4A8]
188C 13FC 05/11 15:28:46.345 31 vssbaserequestor.cpp(69) [0000000000A2C4A8] 99303538-624D-4F25-AB30-45F6D758E77D NORMAL CVssBaseRequestor: destructor [0000000000A2C4A8]
188C 110C 05/11 15:28:53.896 22 vdshelper.cpp(923) [0000000000A2D530]  WARNING Failed: Hr: = [0x80990fb0] Disk [\\?\PHYSICALDRIVE7] is not found as unknown
188C 110C 05/11 15:28:53.896 22 vdshelper.cpp(1892) [0000000000A2D530]  NORMAL Disk Name, fIsDiskFound = Port0Path0Target0Lun0, 0x0
188C 110C 05/11 15:28:53.896 22 vdshelper.cpp(1892) [0000000000A2D530]  NORMAL

It was working fine up until a couple of months ago.

Anyone got any ideas? I really hope it wasn't the new Anti-Virus I installed

Hi,

we are currently implementing DPM with the Hitachi VSS Hardware Provider. We have VSP Storage for our Hyper-V Cluster and AMS Storage for the Backup Data. The AMS Storage is presented as path-through disks to the DPM VM. When we run more than one backup in parallel we see the following error:

"DPM encountered a retryable VSS error. (ID 30112 Details: VssError:The writer's timeout expired between the Freeze and Thaw events. (0x800423F2))"

The backup will be started again and succeeds. Has anyone came accross this issue?

Regards Alex