wmc-dedupe: Remove duplicate recordings

Post Reply
mrsilver

Posts: 50
Joined: Mon Oct 20, 2014 5:01 pm
Location:

HTPC Specs: Show details

wmc-dedupe: Remove duplicate recordings

#1

Post by mrsilver » Thu Oct 23, 2014 2:32 pm

WMC-dedupe is a free program which will identify all duplicate Windows Media Center recorded television shows (in either WTV or DVR-MS formats) and then either move them into a folder for duplicates or delete them. Recordings that are sitting in the duplicates folder can be automatically deleted after a certain number of days.

Although the series link functionality within Windows Media Center is pretty good, due to poor programming and/or poor EPG data it means that setting up a series recording of a show can often result many duplicate recordings. This is because Windows Media Center often just ends up recording every single time the show is aired.

The solution is for Microsoft to review their code and also to review the quality of the EPG data provided by their third-parties. However since complaints directly to Microsoft about this issue go largely ignored, WMC-dedupe temporarily "fixes" the problem by deleting duplicate recordings and saves you from drowning in repeats of a show.

More information and download: http://www.fourteenminutes.com/code/WMC-dedupe/

If you have any questions, comments or bug reports, please use this thread to report them.

mrsilver

Posts: 50
Joined: Mon Oct 20, 2014 5:01 pm
Location:

HTPC Specs: Show details

#2

Post by mrsilver » Thu Oct 23, 2014 2:40 pm

WMC-dedupe version 1.5 released

This version has the following changes:
  • Added support for WTV files created in W8.1MC which have the description and subtitle of the show stored in a different location.
  • Cleaned up the formatting of the documentation.
  • Minor update to the copyright notices in the code and documentation.
  • New forum thread for questions, comments and bug reports.
Download it from http://www.fourteenminutes.com/code/WMC-dedupe/

pavlov70

Posts: 2
Joined: Sun Jul 13, 2014 3:18 pm
Location:

HTPC Specs: Show details

#3

Post by pavlov70 » Fri Nov 28, 2014 2:10 am

Just wanted to say thanks for this! I'm switching from extenders to Intel NUC's, and since each one will have DVR abilities, I'll be robocopying the recordings to the media server. My only concern was the possibility of a bunch of duplicate recordings. Looks like WMC-dedupe is just the solution.

dsg47

Posts: 1
Joined: Fri Dec 12, 2014 11:07 pm
Location:

HTPC Specs: Show details

#4

Post by dsg47 » Sat Dec 13, 2014 1:09 am

Thanks mrsilver for the creation of the deletion tool. It works great. I was wondering if I could add several paths to be deleted at one time. I tried "cscript WMC-dedupe-1.5.vbs /f /D "h:\recorded tv" /d "e:\recorded tv" /d "g:\recorded tv" I also tried cscript WMC-dedupe-1.5.vbs /f /D "h:\recorded tv" "e:\recorded tv" "g:\recorded tv" along with other trials, but whatever I tried, It would just delete H: and stop. Any thoughts? Thanks.

User avatar
apbms1

Posts: 29
Joined: Fri Jul 12, 2013 3:01 pm
Location:

HTPC Specs: Show details

#5

Post by apbms1 » Sat Dec 13, 2014 8:51 am

This works fantasic. Many thanks. Just cleaned up 200 gb of series linked my lottle pony. Over 300 episodes down to 60. Mega.8-)

mrsilver

Posts: 50
Joined: Mon Oct 20, 2014 5:01 pm
Location:

HTPC Specs: Show details

#6

Post by mrsilver » Tue Dec 16, 2014 9:13 am

dsg47 wrote:Thanks mrsilver for the creation of the deletion tool. It works great. I was wondering if I could add several paths to be deleted at one time. I tried "cscript WMC-dedupe-1.5.vbs /f /D "h:\recorded tv" /d "e:\recorded tv" /d "g:\recorded tv" I also tried cscript WMC-dedupe-1.5.vbs /f /D "h:\recorded tv" "e:\recorded tv" "g:\recorded tv" along with other trials, but whatever I tried, It would just delete H: and stop. Any thoughts? Thanks.
Sadly it doesn't support that right now (as it only supports one path in the command line) but can be easily added. In fact someone sent me a patch to do exactly that so I'll dig it out and implement it for you.

Give me a couple of days :)

tletourneau

Posts: 25
Joined: Tue Sep 03, 2013 6:31 pm
Location: SE Minnesota

HTPC Specs: Show details

#7

Post by tletourneau » Sun Apr 12, 2015 8:51 pm

Thanks for this, it works well!
Thanks,
Tom

dazza25

Posts: 1
Joined: Thu Apr 16, 2015 1:13 am
Location:

HTPC Specs: Show details

#8

Post by dazza25 » Thu Apr 16, 2015 10:38 am

This works very well, great program.

Only isssue I had was 2 files where I had to reboot Media Centre in middle of recording something, so the program was broken into two files, obviously with same metadata. It wanted to delete the smaller one of the two. Could you make it such that it ignores matches if one is within say 2 hours of the other to stop this eventuality?

mrsilver

Posts: 50
Joined: Mon Oct 20, 2014 5:01 pm
Location:

HTPC Specs: Show details

#9

Post by mrsilver » Thu Apr 23, 2015 10:08 pm

Sure I can add that to my "to do" list - only thing though is how often would such a scenario occur?

cjames9001

Posts: 8
Joined: Sun Aug 14, 2016 3:32 pm
Location:

HTPC Specs: Show details

#10

Post by cjames9001 » Sun Aug 14, 2016 3:50 pm

I've got a pet project that I'm working on getting more polished for others to use that will prevent duplicate shows from even being recorded.

I've been using it for about two years now and I absolutely love it and think it is finally to a point where I can start working on making this easier to setup.

I see this post is a bit old but I do wonder if there is any interest in this out there?

For anyone that is code literate this is my project up on github: https://github.com/cjames9001/WMCDuplicateRemover

There are a few steps to get it setup and isn't super intuitive yet. I am working on it though!

Anyways, if anyone is curious or wants to know more I'd be happy to share anything and everything about this setup.

mrsilver

Posts: 50
Joined: Mon Oct 20, 2014 5:01 pm
Location:

HTPC Specs: Show details

#11

Post by mrsilver » Sun Aug 14, 2016 4:05 pm

So basically you do what I would have loved to have done, which is proactively go into future scheduled recordings and cancel them if they have already been recorded?

If so, nice, very nice and I'd be interested in it. Well done!

Space

Posts: 1552
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#12

Post by Space » Sun Aug 14, 2016 8:57 pm

Interesting...

Just a warning... if WMC is scheduled to record the same episode multiple times (due to a known bug), it does not consider those episodes to be the same episode, so if you tell it to not record all but one of these episodes and there is an issue while recording that one instance of the episode (power failure, conflict, etc.), it will not reschedule to record the other instances of that episode that is doesn't think are the same episode (although there may be other instances of that episode in the guide that it DOES consider to be the same episode that it may schedule to record).

So if your recordings are unreliable, then you may want to just leave it recording episodes multiple times, and just deal with deleting the duplicates.

cjames9001

Posts: 8
Joined: Sun Aug 14, 2016 3:32 pm
Location:

HTPC Specs: Show details

#13

Post by cjames9001 » Mon Aug 15, 2016 10:24 pm

mrsilver wrote:So basically you do what I would have loved to have done, which is proactively go into future scheduled recordings and cancel them if they have already been recorded?

If so, nice, very nice and I'd be interested in it. Well done!
I only have a couple of minutes but I hope to post either later tonight or tomorrow with some actual details.
Space wrote:Interesting...

Just a warning... if WMC is scheduled to record the same episode multiple times (due to a known bug), it does not consider those episodes to be the same episode, so if you tell it to not record all but one of these episodes and there is an issue while recording that one instance of the episode (power failure, conflict, etc.), it will not reschedule to record the other instances of that episode that is doesn't think are the same episode (although there may be other instances of that episode in the guide that it DOES consider to be the same episode that it may schedule to record).

So if your recordings are unreliable, then you may want to just leave it recording episodes multiple times, and just deal with deleting the duplicates.
If I am understanding you correctly this should be a non-issue. The program I have written will not consider cancelling a recording unless there has been a confirmed successful recording in the event log or if it has previously been cancelled (Another item on the todo list is to make it configurable on which events count as previously recorded as not everyone will consider a previously cancelled show as something to cancel). It will not cancel anything that has not had an event tied to it yet, only after a first instance of a success or cancellation. In the event of a malfunction or power loss WMC reports that there was some error and my project will not pick it up as something to be cancelled.

Space

Posts: 1552
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#14

Post by Space » Tue Aug 16, 2016 12:44 am

cjames9001 wrote:...

If I am understanding you correctly this should be a non-issue. The program I have written will not consider cancelling a recording unless there has been a confirmed successful recording in the event log or if it has previously been cancelled (Another item on the todo list is to make it configurable on which events count as previously recorded as not everyone will consider a previously cancelled show as something to cancel). It will not cancel anything that has not had an event tied to it yet, only after a first instance of a success or cancellation. In the event of a malfunction or power loss WMC reports that there was some error and my project will not pick it up as something to be cancelled.
Excellent!

cjames9001

Posts: 8
Joined: Sun Aug 14, 2016 3:32 pm
Location:

HTPC Specs: Show details

#15

Post by cjames9001 » Wed Aug 17, 2016 2:33 am

So here are the steps that you'd have to take to use my program:

Setup The Event Logs:

1. Open the Event Viewer
2. Open the "Applications and Services Logs"
3. Right click "Media Center"
4. Click "Properties"
5. In "Maximum log size ( KB ):" enter 1000000 (This is so your log file can get large so it remembers your recordings 2 years worth is about 38MB for me so ~100 MB will last for quite sometime I am working on making steps 1-6 obsolete but I'm not there yet.)
6. Close the event viewer

Setup the programs files:

7. Create this folder "C:\ProgramData\WMCDuplicateRemover"
8. Download mc2xml.exe (http://mc2xml.hosterbox.net/) and place it in "C:\ProgramData\WMCDuplicateRemover"
9. Create this folder "C:\ProgramData\WMCDuplicateRemover\WMCDuplicateRemoverDriver"
10. Place my program's files in the "C:\ProgramData\WMCDuplicateRemover\WMCDuplicateRemoverDriver" folder

11. Run mc2xml.exe the first time it runs it will ask you what service you want to use, there are a few options here. I used to use microsoft legacy but about a year ago they stopped updating it :( so my recordings weren't being cancelled anymore. Using any of these other services will work so long as their data is accurate but you'll have to get accounts or api access keys for yourself. I currently use schedules direct which is $25 a year. Really reasonable if you ask me.
12. You can leave all the checkbox options blank.
13. Once it has finished running for the first time it should have created mc2xml.dat and xmltv.xml. If you need to change your service or put the wrong zip etc. Delete the mc2xml.dat file and run it again it will prompt you with the questions again.

Setup the scheduled task:
I've setup a scheduled task to run twice a day at 3:20 and 15:20 mostly because these are times that I'm not likely to be using the dvr it used to put a pretty big toll on resources but I think I've mitigated a lot of the performance issues. I suppose I'd still recommend doing it on off hour times just in case there is some collision of some kind that could cause any errors, I haven't seen this happen in the 2 years I've been using it but you never know.

14. Open the Task Scheduler.
15. Right click the "Task Scheduler Library"
16. Click "Create Task..."
17. I named mine "WMCDuplicateRemover" but it can be named whatever
18. I created a user to run as and picked "Run whether user is logged on or not"
19. You should be able to just use any account provided it has read/write access to "C:\ProgramData\WMCDuplicateRemover"
20. On the "Triggers" tab I added a Daily trigger to start at 3:20PM and recur every 1 day and repeat the task every 12 hours and to stop the task if it runs longer than 4 hours. (It used to take a long time to run but now it takes less than 10 minutes so a smaller number could go here, I'd say if it ran for an hour its probably hung.)
21. On the "Actions" tab select "Start a program"
22. In the "Program/script" box put C:\ProgramData\WMCDuplicateRemover\WMCDuplicateRemoverDriver\WMCDuplicateRemoverDriver.exe or just browse to the file.
23. Then sit back and wait for it to start at whatever time you scheduled it for.

I don't have releases setup on my github quite yet so I could email you a zip of my program. I'm aiming to get releases setup this weekend on my github.

Anyways if this doesn't seem to daunting send me a PM and I can send you what you'll need.

Also I'd be more that happy to help out with any of the setup steps or if you have any other questions.

cjames9001

Posts: 8
Joined: Sun Aug 14, 2016 3:32 pm
Location:

HTPC Specs: Show details

#16

Post by cjames9001 » Mon Aug 29, 2016 10:24 pm

Just an update, I've managed to make steps 1-6 no longer necessary. I've also created a release on my github page https://github.com/cjames9001/WMCDuplic ... r/releases. If you get it downloaded and need any help let me know!

mrsilver

Posts: 50
Joined: Mon Oct 20, 2014 5:01 pm
Location:

HTPC Specs: Show details

#17

Post by mrsilver » Mon Oct 31, 2016 8:23 pm

Daft question I'm sure, but what is the point of the XML bit?

If you can read the meta-data out of the recorded TV files (easy enough) and you can read the meta-data for the scheduled recordings, wouldn't you just loop through all scheduled recordings, check if the meta-data associated with it matches a WTV file and (if so) cancels the schedule?

cjames9001

Posts: 8
Joined: Sun Aug 14, 2016 3:32 pm
Location:

HTPC Specs: Show details

#18

Post by cjames9001 » Sun Dec 04, 2016 4:29 pm

mrsilver wrote:Daft question I'm sure, but what is the point of the XML bit?

If you can read the meta-data out of the recorded TV files (easy enough) and you can read the meta-data for the scheduled recordings, wouldn't you just loop through all scheduled recordings, check if the meta-data associated with it matches a WTV file and (if so) cancels the schedule?
Sorry for the late reply, I think you're asking why I'm using mc2xml instead of just relying on the WMC meta data for scheduled recordings and meta data for previously recorded shows?

Originally I had queried windows media center for the scheduled recording info but a couple things happened thay led me to change the design.
1. When Windows changed their epg source a while back I found the data from windows media center less reliable than the information I could get from an external source. The crappier reliability resulted in shows not getting cancelled that should have been.
2. Windows media center only allows you a maximum of 1200 queries per day and then it just gives you errors back. That made it a pain to test and just felt like a bad idea.

Now the reason I don't rely on the meta data of the recordings is because they are non permanent. As in when I finish watching a show I'll delete he recording. I suppose I could've gone that route for getting historical information but then you still run into the problem of if I watched and deleted a file between the program's runs then it wouldn't be counted in the istorical data. I'd probably have to rewrite the program as a service instead of a scheduled task and have it watch the recorded tv directory instead of running x times a day. I really felt that a service would be a little heavy handed for what I wanted to do.

mrsilver

Posts: 50
Joined: Mon Oct 20, 2014 5:01 pm
Location:

HTPC Specs: Show details

#19

Post by mrsilver » Sat Aug 10, 2019 5:51 pm

In case anyone is interested, the old domain is now dead so I've moved WMC-dedupe over to github.

New link is https://github.com/mrsilver76/WMC-dedupe

Post Reply