Strange "BrainDead" issue

An evolving, supported alternative to Rovi
Forum rules
★ Download the latest EPG123 here: https://garyan2.github.io/ <> Setup guide here: https://garyan2.github.io/install.html
jbf154

Posts: 55
Joined: Mon Jan 14, 2013 7:41 pm
Location:

HTPC Specs: Show details

#21

Post by jbf154 » Mon Jul 25, 2016 3:42 pm

One thing worth considering. Garyan2, you mentioned working on a method of loading the MXF without using loadmxf.exe. If you are able to do this, you can simply use your own ProgramMerger instance where you set alwaysReplace = True.

I looked at this a few months back, but couldn't get the alwaysReplace to read in from the MXF. I did find a work around where I would create a "dummy" schedule entry block where the duration is the full day with a dummy program assigned to it. Then immediately after, it would import the correct schedule entries that overwrote that block of time. It worked, BUT I had an issue where previously scheduled one-time recordings were not recorded if they aired during one of these blocks of time.

glugglug

Posts: 391
Joined: Thu Jun 09, 2011 1:34 am
Location:

HTPC Specs: Show details

#22

Post by glugglug » Tue Jul 26, 2016 1:09 am

jbf154 wrote:One thing worth considering. Garyan2, you mentioned working on a method of loading the MXF without using loadmxf.exe. If you are able to do this, you can simply use your own ProgramMerger instance where you set alwaysReplace = True.

I looked at this a few months back, but couldn't get the alwaysReplace to read in from the MXF. I did find a work around where I would create a "dummy" schedule entry block where the duration is the full day with a dummy program assigned to it. Then immediately after, it would import the correct schedule entries that overwrote that block of time. It worked, BUT I had an issue where previously scheduled one-time recordings were not recorded if they aired during one of these blocks of time.
It's still using the same underlying library that loadMXF is, and the ScheduleEntries objects which own the ProgramMergers are deserialized within that, so not so simple.
One thing I've been thinking could be possible would be that you could descend from the various types of StoredObject in mcepg, i.e. make an AlwaysReplaceScheduleEntries that descends from ScheduleEntries and updates the ProgramMerger flags in the constructor, or maybe override MergeScheduleEntries to be much smarter about the comparison (I don't think it's checking any of the settings like isHdtv, isCC, isLive etc for updates, as long as the program id and start/end time match). The MXF would then have <AlwaysReplaceScheduleEntries/> elements instead of the base type, and the library with your custom classes would need to be added to the Assemblies element in the MXF header. Curious what other affects this would have if it worked. I'd be afraid it would create an object of the new type in the DB that WMC wouldn't know how to read.

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#23

Post by garyan2 » Tue Jul 26, 2016 3:46 am

Correct. I was basically doing exactly what loadmxf.exe was doing to call the MxfImport.Import() in the mcstore.dll. I still don't have any real control over the import at that point and I don't plan on going any further than this. To try and manually perform all the imports I think is way too complicated and way too risky for my limited skills.

As much as I hate workarounds, I think setting the isOnlyWmis and isMergedService flags is the way to go. Just need to flesh out any side effects, if any.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

glugglug

Posts: 391
Joined: Thu Jun 09, 2011 1:34 am
Location:

HTPC Specs: Show details

#24

Post by glugglug » Wed Jul 27, 2016 3:10 am

Looking through Reflector, I noticed an internal class Microsoft.MediaCenter.Store.Explorer.StoreExplorerForm that could maybe be some kind of GUI for debugging the ObjectStore db.

I remembered reading that it was possible to use Reflection to call internal and private methods, so I did some searching and yeah, I think it would be possible to trigger that form. But even better (or worse, depending how you look at it), it is possible to use Reflection to REPLACE existing IL methods at runtime!!! So in theory jbf154 had it right, it would be possible for instance to replace the ScheduleEntries constructor to have the desired flags on the ProgramMerger, and lots of other craziness. The security implications are mind-boggling. I will have to play around with this when I get the time. See the answer by LogMan here: http://stackoverflow.com/questions/7299 ... arp-method

So many other methods that are tempting to try to replace.... <cough>DRM</cough>.

schottjy

Posts: 45
Joined: Wed May 16, 2012 3:02 am
Location:

HTPC Specs: Show details

#25

Post by schottjy » Thu Aug 04, 2016 1:44 am

I'm not sure if this is the same issue or a different one, but it seems that some of my guide information isn't consistent with the egp123 cache & schedule direct data.

Here's an example. Tonight on (Provider:Time Warner Cable, Type:CableDigital, Zip:27612, Lineup:NC32421:X Device:Digital‏) on channel 264 (Cartoon Network) at 11:30PM EST my guide lists "Aqua Teen Hunger Force / Creature From Plague Lagoon" as showing. I have no epg123 cache file with that episode information on my system. According to the SchedulesDirect feed (thanks rkugalow) and adultswim.com, I should be seeing:

Code: Select all

{
"startTime": "2016-08-04T04:30Z",
"endTime": "2016-08-04T04:45Z",
"duration": 15,
"title": "Squidbillies",
"episodeTitle": "Flight of the Deep Fried Pine Booby",
}
and indeed I have epg123 cache data that says:

Code: Select all

{"programID":"EP007008180035",
"titles":[{"title120":"Squidbillies"}],
"episodeTitle150":"Flight of the Deep Fried Pine Booby",
}
and

Code: Select all

{"programID":"EP007008180035"
"airDateTime":"2016-08-04T04:30:00Z"
Is this another symptom of MXF import failing? I've already run the epg123 1.01beta1 you posted.

-jeff

glugglug

Posts: 391
Joined: Thu Jun 09, 2011 1:34 am
Location:

HTPC Specs: Show details

#26

Post by glugglug » Thu Aug 04, 2016 2:54 am

I'm seeing the same, even with my alternate MXFLoader tool. (channels 257 and 757 on FIOS)

Running it with tracing on to get a better idea what's going on.

Gary: I figured out the location it writes to can be controlled with a "TraceFile" registry value, to reduce SSD wear with the verbose setting:

Code: Select all

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Debug\mcstore]
"TraceLevel"=dword:00000004
"TraceFile"="e:\\temp\\mcstore.trace.log"
Relevant tracelog entries:

Code: Select all

MXFLoader.16956 Information: 0 : [LM] ScheduleEntries.MergeScheduleEntries() - *** Comparing SE to merge (Wmis 8/4/2016 4:30:00 AM - 8/4/2016 4:45:00 AM on TOON -> Squidbillies - Flight of the Deep Fried Pine Booby Id:0xE0000019BA553 (3940649700926803)) with SE in store (Merged 8/4/2016 4:30:00 AM - 8/4/2016 4:45:00 AM on TOON -> Aqua Teen Hunger Force - Creature From Plaque Lagoon Id:0xE0000019B9E85 (3940649700925061)) ***
    DateTime=2016-08-04T02:58:35.0201112Z
MXFLoader.16956 Information: 0 : [LM] Entering ProgramMatcher.CheckIfProgramsMatch(8/4/2016 4:30:00 AM - 8/4/2016 4:45:00 AM on TOON -> Squidbillies - Flight of the Deep Fried Pine Booby Id:0xE0000019BA553 (3940649700926803), 8/4/2016 4:30:00 AM - 8/4/2016 4:45:00 AM on TOON -> Aqua Teen Hunger Force - Creature From Plaque Lagoon Id:0xE0000019B9E85 (3940649700925061), 8/4/2016 4:45:00 AM - 8/4/2016 5:00:00 AM on TOON -> Squidbillies - School Days, Fool Days Id:0xE0000019B9E8B (3940649700925067))
    DateTime=2016-08-04T02:58:35.0201112Z
MXFLoader.16956 Information: 0 : [LM] ProgramMatcher.CheckIfExactMatch() - Failed all comparisons, NOT AN EXACT MATCH
    DateTime=2016-08-04T02:58:35.0201112Z
MXFLoader.16956 Information: 0 : [LM] ProgramMatcher.CheckIfProgramsMatch() - MatchTitleEpisode is TRUE, comparing titles...
    DateTime=2016-08-04T02:58:35.0201112Z
MXFLoader.16956 Information: 0 : [LM] ProgramMatcher.CheckIfTitlesMatch() - Titles DO NOT MATCH
    DateTime=2016-08-04T02:58:35.0201112Z
MXFLoader.16956 Information: 0 : [LM] ProgramMatcher.CheckIfProgramsMatch() - Failed all comparisons, so programs DO NOT MATCH
    DateTime=2016-08-04T02:58:35.0201112Z
MXFLoader.16956 Information: 0 : [LM] ScheduleEntries.MergeScheduleEntries() - Checking SE in store (8/4/2016 4:30:00 AM - 8/4/2016 4:45:00 AM on TOON -> Aqua Teen Hunger Force - Creature From Plaque Lagoon Id:0xE0000019B9E85 (3940649700925061)) for overlaps
    DateTime=2016-08-04T02:58:35.0201112Z
Last edited by glugglug on Thu Aug 04, 2016 3:30 am, edited 1 time in total.

schottjy

Posts: 45
Joined: Wed May 16, 2012 3:02 am
Location:

HTPC Specs: Show details

#27

Post by schottjy » Thu Aug 04, 2016 3:21 am

Thanks for digging in glugglug. It looks like it's happening on that channel quite a bit always around the same time. That AdultSwim lineup is constantly moving around and I've (sadly) been in the habit of checking it quite a bit since the original Rovi transition.

glugglug

Posts: 391
Joined: Thu Jun 09, 2011 1:34 am
Location:

HTPC Specs: Show details

#28

Post by glugglug » Thu Aug 04, 2016 4:19 am

Stepping through MergeScheduleEntries with breakpoints set when the CheckIfProgramsMatch returns false, I actually see a lot of these on many different channels. Some of the branches it takes make no sense, maybe just because of the optimizer? They mostly seem to be getting added to the "adds" list instead of the "updates" list for the call to ApplyMergedScheduleEntriesToStore later.

It's temping to make my MXFLoader completely replace MergeScheduleEntries instead of just setting AlwaysReplace and checking program flags like it does now. P.S. another thing I figured out from all this: The class used to merge keyword groups for display is MergedKeywordGroup, which is not a stored object. I could probably change this to ignore KeywordGroups which are subgroups of others at the top level for a multi-level keyword hierarchy :-)

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#29

Post by garyan2 » Thu Aug 04, 2016 4:51 am

Well, durnit ... not a lineup I have on my testbench, but wouldn't matter too much since I had turned the tracing off. :oops:

glugglug, was there nothing after the last ScheduleEntries.MergeScheduleEntries() to check for overlaps? I agree about the logic ... just doesn't make too much sense sometimes. I haven't done a deep dive into the code yet so I've just been assuming I was missing something. I too am starting to get tempted to just replace chunks of the code based on what you've discovered but like I said, way beyond my skills.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

glugglug

Posts: 391
Joined: Thu Jun 09, 2011 1:34 am
Location:

HTPC Specs: Show details

#30

Post by glugglug » Thu Aug 04, 2016 5:10 am

Not sure what you mean by 'last" MergeScheduleEntries. I only see one. Although I have noticed copying code from reflector into a text editor through the clipboard often creates a double copy.

There is code both within MergeScheduleEntries and within the program matcher that checks for overlap, but a lot of what it does when it sees that overlap doesn't really make sense to me.

Another example to look into that may be in your lineup: the breakpoints I set for detecting programs not matching and seeing it not replace them (otherwise they would have matched after multiple runs!) stopped a lot of times on the upcoming block of Blue Bloods episodes Thursday night on WGN. (do you call that today or tomorrow given the time LOL?) It's still a block of Blue Bloods episodes but which episodes has been shuffled around and it's creating adds instead of updates to pass to ApplyMergedScheduleEntriesToStore which aren't taking.

glugglug

Posts: 391
Joined: Thu Jun 09, 2011 1:34 am
Location:

HTPC Specs: Show details

#31

Post by glugglug » Thu Aug 04, 2016 5:41 am

There are still a lot of conditions which don't make sense to me, but I think I figured out the main flaw in the logic:

The cases where I see an "update" created in MergeScheduleEntries can only be reached if the time slot is changed, among other conditions I don't really get. (but overlapping the old time slot).
Edit: Not all of them. I misread some of the conditions. Updates are possible.

Shows that replace an existing show in the same timeslot are done as "adds".

The logic for handling the adds in ApplyMergedScheduleEntriesToStore really treats them as what you would intuitively expect for an "update" if the OriginalStoredObjectId property is set on the new ScheduleEntry. (It will generate a delete for the entry specified by OriginalStoredObjectId). But the MergeScheduleEntries logic never sets this, in fact, I don't see anything that does!!.

Maybe this means that the updated ScheduleEntrys are getting added to the DB, but since the old ones in the same timeslot are not removed, we don't always end up seeing them. Worse, it may be adding duplicates of them on every import.
Edit: I doubt this though.... the code path it seems to take in the debugger doesn't match what the variables/parameters are. After multiple imports of the same MXF with a conditional breakpoint in ApplyMergedScheduleEntriesToStore if the add, update or delete list is non-empty, it never hits that breakpoint. Thinking since I am not close to really understanding the MergeScheduleEntries logic and am afraid of screwing up cases with recordings, to just wrap it and have extra logic after the wrapped call to just handle the common straightforward case of an exact timeslot match with a non-matching program not set to record.

schottjy

Posts: 45
Joined: Wed May 16, 2012 3:02 am
Location:

HTPC Specs: Show details

#32

Post by schottjy » Thu Aug 04, 2016 12:35 pm

Is there any simple workaround you guys can think of? Or is the only fix to delete eHome and rebuild it from scratch? The guide is wrong for some new shows tomorrow night.

glugglug

Posts: 391
Joined: Thu Jun 09, 2011 1:34 am
Location:

HTPC Specs: Show details

#33

Post by glugglug » Fri Aug 05, 2016 12:01 pm

Redoing TV setup should do it without having to wipe out eHome.

I think I am very close to fixing this with a replacement for MergeScheduleEntries. I am not sure what will happen if one of the scheduleentries to be updated/removed has an associated recording scheduled. Also, a couple issues remain that I need to make workarounds for:

1. The private method of ScheduleEntries for finding the last entry at it before the start key seems to have a bug where it sometimes returns the entry before the one it should, or sometimes updates the enumerator to point to the entry you want but somehow the return value is in a screwy state where it doesn't test as null or not null. I think the former issue is a problem with the underlying SQLite seek.

2. For a few cases the adds of missing ScheduleEntrys are triggering calls to MergeScheduleEntries, which in turn trigger the adds again, etc, infinitely recursively. This may be an artifact of incorrect modifications being made due to #1, or maybe I need to move this fix code out to ScheduleEntries.Add(List<ScheduleEntry>) instead of MergeScheduleEntries. I'm kind of wondering if this infinite recursion being triggered is why the existing code is so conservative in which updates it applies.

The guide on my test machine is down to about 2 dozen entries not matching the MXF from probably 1000. The data getting updated at the last minute and ending up not updating in WMC seems to disproportionately affect Spanish and PBS-like channels, which explains why those have had a lot more repeats recorded with generic metadata and/or shows recorded just not matching what they are supposed to be over the years, even when the standard WMC guide was still using zap2it.

schottjy

Posts: 45
Joined: Wed May 16, 2012 3:02 am
Location:

HTPC Specs: Show details

#34

Post by schottjy » Fri Aug 05, 2016 1:15 pm

Redoing TV setup will still require recreating series recordings, etc, yes? I'll give it a try tonight I guess.

Thanks again for digging into the issue glugglug!

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#35

Post by garyan2 » Fri Aug 05, 2016 6:15 pm

garyan2 wrote:Correct. I was basically doing exactly what loadmxf.exe was doing to call the MxfImport.Import() in the mcstore.dll. I still don't have any real control over the import at that point and I don't plan on going any further than this. To try and manually perform all the imports I think is way too complicated and way too risky for my limited skills.

As much as I hate workarounds, I think setting the isOnlyWmis and isMergedService flags is the way to go. Just need to flesh out any side effects, if any.
Okay, very bad side effect on doing the above ... scratch that out of possibilities.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

schottjy

Posts: 45
Joined: Wed May 16, 2012 3:02 am
Location:

HTPC Specs: Show details

#36

Post by schottjy » Fri Aug 05, 2016 6:16 pm

garyan2 wrote:
garyan2 wrote:Correct. I was basically doing exactly what loadmxf.exe was doing to call the MxfImport.Import() in the mcstore.dll. I still don't have any real control over the import at that point and I don't plan on going any further than this. To try and manually perform all the imports I think is way too complicated and way too risky for my limited skills.

As much as I hate workarounds, I think setting the isOnlyWmis and isMergedService flags is the way to go. Just need to flesh out any side effects, if any.
Okay, very bad side effect on doing the above ... scratch that out of possibilities.
Gary does this mean I should revert back to epg123 1.0?

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#37

Post by garyan2 » Fri Aug 05, 2016 6:24 pm

I would, absolutely. It looks like in some cases, rather than changing what program the schedule entry references, it overwrites the program. In my case, I saw a generic program get overwritten with a specific program. Results was all generic schedule entries for that series was listed as that specific episode.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

schottjy

Posts: 45
Joined: Wed May 16, 2012 3:02 am
Location:

HTPC Specs: Show details

#38

Post by schottjy » Fri Aug 05, 2016 6:51 pm

garyan2 wrote:I would, absolutely. It looks like in some cases, rather than changing what program the schedule entry references, it overwrites the program. In my case, I saw a generic program get overwritten with a specific program. Results was all generic schedule entries for that series was listed as that specific episode.
OK, and you agree with glugglug that re-running Setup TV is the best way to reset the guide? I had previously used ObjectStoreRecoveryTask in the olden days...

webminster

Posts: 657
Joined: Tue Dec 20, 2011 11:05 pm
Location:

HTPC Specs: Show details

#39

Post by webminster » Fri Aug 05, 2016 7:03 pm

garyan2 wrote:I would, absolutely. It looks like in some cases, rather than changing what program the schedule entry references, it overwrites the program. In my case, I saw a generic program get overwritten with a specific program. Results was all generic schedule entries for that series was listed as that specific episode.
Can you revert to 1.0.0 from 1.0.1b1 just by replacing the bits and rerunning EPG123 setup? Or is the damage done and I really need to run a LiveTV setup as well?
-Alan

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#40

Post by garyan2 » Fri Aug 05, 2016 7:14 pm

I honestly don't know yet. I suspect replacing the epg123.exe will repair most of the damage, but can't guarantee all. I think what is left will probably die out. I'm running an update right now but won't be able to see the results for quite a bit later today.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

Post Reply