Created a new MXF Importer

[glugglug]

It's using MXFImporter.Import like LoadMXF is, but first it replaces a few methods in the ScheduleEntries, ProgramMerger, and ProgramMatcher classes to ensure all the updates are imported.

For example, it picks up the recent change to the HD flag for shows on STZESP (FIOS channel 363) while the regular LoadMXF doesn't.

The code is a mess for something so small, but I figure it may still be useful. Somewhat as a proof of concept to see that I can successfully replace methods in-process before attempting to do it to other processes

https://www.dropbox.com/sh/9cgz6sfabxvv ... 4dTfa?dl=0

Current binary is for Win7 x64, it would need mcepg and mcstore references replaced on a 8.1 machine and to be rebuilt to make a Win8.1 binary.

Other gotchas:

Only works as a Release build. The code injection stuff has separate cases for debug vs. release, and I guess since mcepg.dll with the target methods to replace is release, the app containing the new versions of those methods must also be built as release. Also, the classes are declared public to avoid said methods from being inlined (could also prevent this using [MethodImpl(MethodImplOptions.NoInlining)] on each method.)

[Space]

I assume this can only be used to import clear-text MXF files (like the one generated by EPG123), and not the one that is downloaded from Microsoft (which appears to be encrypted).

[glugglug]

Yes, it works on the file created by epg123, i.e.

mxfloader -i c:\epg123\epg123.mxf

It doesn't launch the reindexing task, but that can either be launched separately or as part of the next run of epg123

[garyan2]

This is soooo above my skill level, I don't even yet know how cool it is ... and I have no doubt it is. I can understand overrides somewhat, which is what I was imagining trying to do, but the relationships going on here is just beyond me. Like you noted before, the possibilities are just crazy now.

[audinutt]

Ten thumbs up for perseverance and keeping wmc alive

Sent from my SM-N900V using Tapatalk

[glugglug]

Added Win8.1 x64 binary:
https://www.dropbox.com/s/2jlx0emc535e9 ... .rar?dl=0a

[Sammy2]

Added Win8.1 x64 binary:
https://www.dropbox.com/s/2jlx0emc535e9 ... .rar?dl=0a

Anything new with this endeavor?

[glugglug]

I'm working on a version that wraps MergeScheduleEntries to catch the many things it misses even with the AlwaysReplace set on the ProgramMerger and MatchTitles cleared on the ProgramMatcher.

Running into a lot of complications with it, inclusing corrupting the guide db if I try to remove all the duplicate/overlapping schedule entries it finds. Going to be refactoring that and see if some of those dupe entries are "special" causing this. See the thread about the BrainDead issue which inspired this effort.

[glugglug]

New version is here! (source, Win7 x64, and Win8.1 x64): https://www.dropbox.com/sh/9cgz6sfabxvv ... 4dTfa?dl=0

Both debug and release builds now work. Got rid of the debug vs release check in the reflection based code injection library, the code that was on stackexchange for release works in debug mode as well.

I looked through what ApplyMergedScheduleEntriesToStore does for deleted schedule entries -- they are never actually deleted, it just sets their service and program ids to null so they don't show up in the guide or searches. I guess this is where I was going wrong attempting to delete the dupes.

Added a "safe mode" that will import just like the normal LoadMXF but still check to see what is different after each ScheduleEntries block is imported and can display this to the console and/or the objectstore tracelog for the process.

This tool should fix issues similar to the BrainDead one, with one caveat:

I found a bunch of instances where the guide data has gone the opposite direction you would expect -- it had specific episodes for a series listed and the current SchedulesDirect data now has generic episodes of the same series. For these I have it leaving it with the specific episode rather than making it generic. Biggest example is Forensic Files on HLN from 8/22 onwards has generic entries now, but had more specific episodes earlier. Would this be an issue with the SD feed?

Sample usage:

Code: Select all

MXFLoader -i C:\EPG123\epg123.mxf -c 3 --reindex

Command line flags:

Code: Select all

  -i, --input_mxf               Required. Path to input MXF file

  -s, --safe_mode               (Default: False) Only make changes that the
                                regular loadMXF would.  Things it misses may
                                still be output.

  --skip_listing_consistency    (Default: False) Don't make listings
                                assignments of Channels and MergedChannels
                                consistent. Making them consistent avoids
                                failed update attempts from becoming infinitely
                                recursive and crashing the import.  Ignored if
                                -safe_mode is true

  -c, --console_loglevel        (Default: 2) Trace level severity to output to
                                console.  0=none, 1=error, 2=warning, 3=info,
                                4=verbose.  Logging to file through ObjectStore
                                tracing is controlled separately by registry
                                value.

  --reindex                     (Default: False) Kick off the ReindexSearchRoot
                                task when import is completed.

  --help                        Display this help screen.

[glugglug]

I just tried this on my main machine -- without --safe_mode it crashes with a PVR rescheduling thread getting an access violation after trying to update a TVRating flag. Clearly still need to debug this more

[jbf154]

Is forcing AlwaysReplace=True not working? The simpler, the better probably.

[glugglug]

Setting AlwaysReplace=true catches some things but still misses a lot.

The problem seems to be that the background thread that wakes up when the schedule is updated to see if recordings or guide filters need to be updated is using ObjectStore.DefaultSingleton instead of an administrative ObjectStore instance. I guess in normal media center processes maybe the default ObjectStore instance is administrative?

I have a fix that replaces SchedulerWorker.ThreadSetup to use an administrative ObjectStore that I am testing now. Will probably update it to this new version later.

[glugglug]

Updated the files on the dropbox again. It still exits before finishing on my main machine that has 200+ series set to record, with no stack trace to give me a hint as to why. Gets a little farther each time, but I think updates to any show set to record are triggering it, still looking into it. For now, I recommend always using --safe_mode for this.

Replacing ScheduleWorker.ThreadSetup didn't actually work, I think because it is virtual. But I looked into how the ObjectStore.DefaultSingleton property works and as long as it is the first call, it uses the FriendlyName and DisplayName properties for the username and password, so I could make the default objectstore administrative.

Edit: Finally got it running all the way through on my main machine with the 200+ scheduled series, and updated the files on DropBox again. I would still recommend using --safe_mode unless one of the missing updates it lists is very important to you on a machine that can be easily restored if something goes wrong. This is still pretty beta. Seeing how the earlier crashes were in a background thread, and looking at the normal LoadMXF in .net Reflector again, I noticed it is set to use an apartment threading model for COM interop, and set my app to do the same, in case that was somehow causing the background thread crashes. Seems to have worked, so far....

[garyan2]

I found a bunch of instances where the guide data has gone the opposite direction you would expect -- it had specific episodes for a series listed and the current SchedulesDirect data now has generic episodes of the same series.

This is what I saw earlier that led me to believe my test with isMergedService and isWmisOnly flags somehow cause bad things to happen... the trace file was showing it was trying to overwrite specific episodes with generic episodes.

[glugglug]

Added unhandled exception handling to hopefully catch a stack trace of any future crashes.

https://www.dropbox.com/sh/9cgz6sfabxvv ... 4dTfa?dl=0

[glugglug]

Got another crash -- the background thread doing resheduling of scheduled recordings throws that access violation exception when the field bindings of a ScheduleEntry don't match those of the ScheduleEntriesEnumerator.

I don't know why they wouldn't always match -- I never change them.... but the normal MergeScheduleEntries code also has a special case dealing with recordings associated with a ScheduleEntry being updated, which doesn't yet make much sense to me. Going to be digging into that.

[glugglug]

A new version is now uploaded to the dropbox link.

I added an option to wait for background threads to complete for debugging after each ScheduleEntries block is updated, in order to tell which updates were triggering the background thread to crash -- with this enabled, I couldn't get it to crash, so maybe it's a race condition with schedule updates occurring while the background thread is doing recording conflict resolution. I expected this waiting for the background thread repeatedly to slow down the import a lot, but it was actually not a noticeable difference in the speed, so I made the wait just always there.

Also, the MediaCenter.Store.OleDB.StoredObjectsEnumerator.Update method that was previously throwing an access violation if the field bindings passed in don't match the ones in the enumerator is now replaced during the import -- if it would previously have crashed, it should detect it, skip the update for that particular record, and output an error message to tell you about it.

https://www.dropbox.com/sh/9cgz6sfabxvv ... 4dTfa?dl=0

[glugglug]

Updated the binary archives to include ALL files the compiler copies to the bin directory, including those I didn't think were needed.

[glugglug]

Not currently working in 8.1. For some reason RuntimeHelpers.PrepareMethod is failing on the replacement MergeScheduleEntries method, no idea why. It is working on several other methods, even in 8.1...

[glugglug]

Updated to work on Windows 8.1 again (hopefully).

There were 2 issues:
1. PrepareMethod was failing on the replacement MergeScheduleEntries. This is supposed to pre-JIT compile it so the compiled code can be copied over that for the built in MergeScheduleEntries method. Maybe it is too big for that in 8.1? Wrapped this function, so the wrapper itself, which is smaller, could be prepared and copied in.
2. ScheduleEntries constructor replacement that set the AlwaysReplace=true, NeverReplace=false, and ProgramMatcher.MatchTitleEpisode=false was causing net runtime failures before said constructor even got called in 8.1. I have no idea why. I guess the kludge I had in place back when I hadn't figured out how to successfully replace a constructor even in Windows 7 was better for 8.1.... Anyways, the changes these flags try to make happen within the MS MXF importing code are done by the replacement for MergeScheduleEntries for any that it misses so this is ultimately redundant. This constructor replacement is now disabled if not running on Windows 7.

https://www.dropbox.com/sh/9cgz6sfabxvv ... 4dTfa?dl=0