Strange "BrainDead" issue
Forum rules
★ Download the latest EPG123 here: https://garyan2.github.io/ <> Setup guide here: https://garyan2.github.io/install.html ★
★ Download the latest EPG123 here: https://garyan2.github.io/ <> Setup guide here: https://garyan2.github.io/install.html ★
-
- Posts: 55
- Joined: Mon Jan 14, 2013 7:41 pm
- Location:
- HTPC Specs:
One thing worth considering. Garyan2, you mentioned working on a method of loading the MXF without using loadmxf.exe. If you are able to do this, you can simply use your own ProgramMerger instance where you set alwaysReplace = True.
I looked at this a few months back, but couldn't get the alwaysReplace to read in from the MXF. I did find a work around where I would create a "dummy" schedule entry block where the duration is the full day with a dummy program assigned to it. Then immediately after, it would import the correct schedule entries that overwrote that block of time. It worked, BUT I had an issue where previously scheduled one-time recordings were not recorded if they aired during one of these blocks of time.
I looked at this a few months back, but couldn't get the alwaysReplace to read in from the MXF. I did find a work around where I would create a "dummy" schedule entry block where the duration is the full day with a dummy program assigned to it. Then immediately after, it would import the correct schedule entries that overwrote that block of time. It worked, BUT I had an issue where previously scheduled one-time recordings were not recorded if they aired during one of these blocks of time.
-
- Posts: 391
- Joined: Thu Jun 09, 2011 1:34 am
- Location:
- HTPC Specs:
It's still using the same underlying library that loadMXF is, and the ScheduleEntries objects which own the ProgramMergers are deserialized within that, so not so simple.jbf154 wrote:One thing worth considering. Garyan2, you mentioned working on a method of loading the MXF without using loadmxf.exe. If you are able to do this, you can simply use your own ProgramMerger instance where you set alwaysReplace = True.
I looked at this a few months back, but couldn't get the alwaysReplace to read in from the MXF. I did find a work around where I would create a "dummy" schedule entry block where the duration is the full day with a dummy program assigned to it. Then immediately after, it would import the correct schedule entries that overwrote that block of time. It worked, BUT I had an issue where previously scheduled one-time recordings were not recorded if they aired during one of these blocks of time.
One thing I've been thinking could be possible would be that you could descend from the various types of StoredObject in mcepg, i.e. make an AlwaysReplaceScheduleEntries that descends from ScheduleEntries and updates the ProgramMerger flags in the constructor, or maybe override MergeScheduleEntries to be much smarter about the comparison (I don't think it's checking any of the settings like isHdtv, isCC, isLive etc for updates, as long as the program id and start/end time match). The MXF would then have <AlwaysReplaceScheduleEntries/> elements instead of the base type, and the library with your custom classes would need to be added to the Assemblies element in the MXF header. Curious what other affects this would have if it worked. I'd be afraid it would create an object of the new type in the DB that WMC wouldn't know how to read.
- garyan2
- Posts: 7474
- Joined: Fri Nov 27, 2015 7:23 pm
- Location:
- HTPC Specs:
Correct. I was basically doing exactly what loadmxf.exe was doing to call the MxfImport.Import() in the mcstore.dll. I still don't have any real control over the import at that point and I don't plan on going any further than this. To try and manually perform all the imports I think is way too complicated and way too risky for my limited skills.
As much as I hate workarounds, I think setting the isOnlyWmis and isMergedService flags is the way to go. Just need to flesh out any side effects, if any.
As much as I hate workarounds, I think setting the isOnlyWmis and isMergedService flags is the way to go. Just need to flesh out any side effects, if any.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io
Keeping WMC alive beyond January 2020. https://garyan2.github.io
-
- Posts: 391
- Joined: Thu Jun 09, 2011 1:34 am
- Location:
- HTPC Specs:
Looking through Reflector, I noticed an internal class Microsoft.MediaCenter.Store.Explorer.StoreExplorerForm that could maybe be some kind of GUI for debugging the ObjectStore db.
I remembered reading that it was possible to use Reflection to call internal and private methods, so I did some searching and yeah, I think it would be possible to trigger that form. But even better (or worse, depending how you look at it), it is possible to use Reflection to REPLACE existing IL methods at runtime!!! So in theory jbf154 had it right, it would be possible for instance to replace the ScheduleEntries constructor to have the desired flags on the ProgramMerger, and lots of other craziness. The security implications are mind-boggling. I will have to play around with this when I get the time. See the answer by LogMan here: http://stackoverflow.com/questions/7299 ... arp-method
So many other methods that are tempting to try to replace.... <cough>DRM</cough>.
I remembered reading that it was possible to use Reflection to call internal and private methods, so I did some searching and yeah, I think it would be possible to trigger that form. But even better (or worse, depending how you look at it), it is possible to use Reflection to REPLACE existing IL methods at runtime!!! So in theory jbf154 had it right, it would be possible for instance to replace the ScheduleEntries constructor to have the desired flags on the ProgramMerger, and lots of other craziness. The security implications are mind-boggling. I will have to play around with this when I get the time. See the answer by LogMan here: http://stackoverflow.com/questions/7299 ... arp-method
So many other methods that are tempting to try to replace.... <cough>DRM</cough>.
-
- Posts: 45
- Joined: Wed May 16, 2012 3:02 am
- Location:
- HTPC Specs:
I'm not sure if this is the same issue or a different one, but it seems that some of my guide information isn't consistent with the egp123 cache & schedule direct data.
Here's an example. Tonight on (Provider:Time Warner Cable, Type:CableDigital, Zip:27612, Lineup:NC32421:X Device:Digital) on channel 264 (Cartoon Network) at 11:30PM EST my guide lists "Aqua Teen Hunger Force / Creature From Plague Lagoon" as showing. I have no epg123 cache file with that episode information on my system. According to the SchedulesDirect feed (thanks rkugalow) and adultswim.com, I should be seeing:
and indeed I have epg123 cache data that says:
and
Is this another symptom of MXF import failing? I've already run the epg123 1.01beta1 you posted.
-jeff
Here's an example. Tonight on (Provider:Time Warner Cable, Type:CableDigital, Zip:27612, Lineup:NC32421:X Device:Digital) on channel 264 (Cartoon Network) at 11:30PM EST my guide lists "Aqua Teen Hunger Force / Creature From Plague Lagoon" as showing. I have no epg123 cache file with that episode information on my system. According to the SchedulesDirect feed (thanks rkugalow) and adultswim.com, I should be seeing:
Code: Select all
{
"startTime": "2016-08-04T04:30Z",
"endTime": "2016-08-04T04:45Z",
"duration": 15,
"title": "Squidbillies",
"episodeTitle": "Flight of the Deep Fried Pine Booby",
}
Code: Select all
{"programID":"EP007008180035",
"titles":[{"title120":"Squidbillies"}],
"episodeTitle150":"Flight of the Deep Fried Pine Booby",
}
Code: Select all
{"programID":"EP007008180035"
"airDateTime":"2016-08-04T04:30:00Z"
-jeff
-
- Posts: 391
- Joined: Thu Jun 09, 2011 1:34 am
- Location:
- HTPC Specs:
I'm seeing the same, even with my alternate MXFLoader tool. (channels 257 and 757 on FIOS)
Running it with tracing on to get a better idea what's going on.
Gary: I figured out the location it writes to can be controlled with a "TraceFile" registry value, to reduce SSD wear with the verbose setting:
Relevant tracelog entries:
Running it with tracing on to get a better idea what's going on.
Gary: I figured out the location it writes to can be controlled with a "TraceFile" registry value, to reduce SSD wear with the verbose setting:
Code: Select all
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Debug\mcstore]
"TraceLevel"=dword:00000004
"TraceFile"="e:\\temp\\mcstore.trace.log"
Code: Select all
MXFLoader.16956 Information: 0 : [LM] ScheduleEntries.MergeScheduleEntries() - *** Comparing SE to merge (Wmis 8/4/2016 4:30:00 AM - 8/4/2016 4:45:00 AM on TOON -> Squidbillies - Flight of the Deep Fried Pine Booby Id:0xE0000019BA553 (3940649700926803)) with SE in store (Merged 8/4/2016 4:30:00 AM - 8/4/2016 4:45:00 AM on TOON -> Aqua Teen Hunger Force - Creature From Plaque Lagoon Id:0xE0000019B9E85 (3940649700925061)) ***
DateTime=2016-08-04T02:58:35.0201112Z
MXFLoader.16956 Information: 0 : [LM] Entering ProgramMatcher.CheckIfProgramsMatch(8/4/2016 4:30:00 AM - 8/4/2016 4:45:00 AM on TOON -> Squidbillies - Flight of the Deep Fried Pine Booby Id:0xE0000019BA553 (3940649700926803), 8/4/2016 4:30:00 AM - 8/4/2016 4:45:00 AM on TOON -> Aqua Teen Hunger Force - Creature From Plaque Lagoon Id:0xE0000019B9E85 (3940649700925061), 8/4/2016 4:45:00 AM - 8/4/2016 5:00:00 AM on TOON -> Squidbillies - School Days, Fool Days Id:0xE0000019B9E8B (3940649700925067))
DateTime=2016-08-04T02:58:35.0201112Z
MXFLoader.16956 Information: 0 : [LM] ProgramMatcher.CheckIfExactMatch() - Failed all comparisons, NOT AN EXACT MATCH
DateTime=2016-08-04T02:58:35.0201112Z
MXFLoader.16956 Information: 0 : [LM] ProgramMatcher.CheckIfProgramsMatch() - MatchTitleEpisode is TRUE, comparing titles...
DateTime=2016-08-04T02:58:35.0201112Z
MXFLoader.16956 Information: 0 : [LM] ProgramMatcher.CheckIfTitlesMatch() - Titles DO NOT MATCH
DateTime=2016-08-04T02:58:35.0201112Z
MXFLoader.16956 Information: 0 : [LM] ProgramMatcher.CheckIfProgramsMatch() - Failed all comparisons, so programs DO NOT MATCH
DateTime=2016-08-04T02:58:35.0201112Z
MXFLoader.16956 Information: 0 : [LM] ScheduleEntries.MergeScheduleEntries() - Checking SE in store (8/4/2016 4:30:00 AM - 8/4/2016 4:45:00 AM on TOON -> Aqua Teen Hunger Force - Creature From Plaque Lagoon Id:0xE0000019B9E85 (3940649700925061)) for overlaps
DateTime=2016-08-04T02:58:35.0201112Z
Last edited by glugglug on Thu Aug 04, 2016 3:30 am, edited 1 time in total.
-
- Posts: 45
- Joined: Wed May 16, 2012 3:02 am
- Location:
- HTPC Specs:
Thanks for digging in glugglug. It looks like it's happening on that channel quite a bit always around the same time. That AdultSwim lineup is constantly moving around and I've (sadly) been in the habit of checking it quite a bit since the original Rovi transition.
-
- Posts: 391
- Joined: Thu Jun 09, 2011 1:34 am
- Location:
- HTPC Specs:
Stepping through MergeScheduleEntries with breakpoints set when the CheckIfProgramsMatch returns false, I actually see a lot of these on many different channels. Some of the branches it takes make no sense, maybe just because of the optimizer? They mostly seem to be getting added to the "adds" list instead of the "updates" list for the call to ApplyMergedScheduleEntriesToStore later.
It's temping to make my MXFLoader completely replace MergeScheduleEntries instead of just setting AlwaysReplace and checking program flags like it does now. P.S. another thing I figured out from all this: The class used to merge keyword groups for display is MergedKeywordGroup, which is not a stored object. I could probably change this to ignore KeywordGroups which are subgroups of others at the top level for a multi-level keyword hierarchy
It's temping to make my MXFLoader completely replace MergeScheduleEntries instead of just setting AlwaysReplace and checking program flags like it does now. P.S. another thing I figured out from all this: The class used to merge keyword groups for display is MergedKeywordGroup, which is not a stored object. I could probably change this to ignore KeywordGroups which are subgroups of others at the top level for a multi-level keyword hierarchy
- garyan2
- Posts: 7474
- Joined: Fri Nov 27, 2015 7:23 pm
- Location:
- HTPC Specs:
Well, durnit ... not a lineup I have on my testbench, but wouldn't matter too much since I had turned the tracing off.
glugglug, was there nothing after the last ScheduleEntries.MergeScheduleEntries() to check for overlaps? I agree about the logic ... just doesn't make too much sense sometimes. I haven't done a deep dive into the code yet so I've just been assuming I was missing something. I too am starting to get tempted to just replace chunks of the code based on what you've discovered but like I said, way beyond my skills.
glugglug, was there nothing after the last ScheduleEntries.MergeScheduleEntries() to check for overlaps? I agree about the logic ... just doesn't make too much sense sometimes. I haven't done a deep dive into the code yet so I've just been assuming I was missing something. I too am starting to get tempted to just replace chunks of the code based on what you've discovered but like I said, way beyond my skills.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io
Keeping WMC alive beyond January 2020. https://garyan2.github.io
-
- Posts: 391
- Joined: Thu Jun 09, 2011 1:34 am
- Location:
- HTPC Specs:
Not sure what you mean by 'last" MergeScheduleEntries. I only see one. Although I have noticed copying code from reflector into a text editor through the clipboard often creates a double copy.
There is code both within MergeScheduleEntries and within the program matcher that checks for overlap, but a lot of what it does when it sees that overlap doesn't really make sense to me.
Another example to look into that may be in your lineup: the breakpoints I set for detecting programs not matching and seeing it not replace them (otherwise they would have matched after multiple runs!) stopped a lot of times on the upcoming block of Blue Bloods episodes Thursday night on WGN. (do you call that today or tomorrow given the time LOL?) It's still a block of Blue Bloods episodes but which episodes has been shuffled around and it's creating adds instead of updates to pass to ApplyMergedScheduleEntriesToStore which aren't taking.
There is code both within MergeScheduleEntries and within the program matcher that checks for overlap, but a lot of what it does when it sees that overlap doesn't really make sense to me.
Another example to look into that may be in your lineup: the breakpoints I set for detecting programs not matching and seeing it not replace them (otherwise they would have matched after multiple runs!) stopped a lot of times on the upcoming block of Blue Bloods episodes Thursday night on WGN. (do you call that today or tomorrow given the time LOL?) It's still a block of Blue Bloods episodes but which episodes has been shuffled around and it's creating adds instead of updates to pass to ApplyMergedScheduleEntriesToStore which aren't taking.
-
- Posts: 391
- Joined: Thu Jun 09, 2011 1:34 am
- Location:
- HTPC Specs:
There are still a lot of conditions which don't make sense to me, but I think I figured out the main flaw in the logic:
The cases where I see an "update" created in MergeScheduleEntries can only be reached if the time slot is changed, among other conditions I don't really get. (but overlapping the old time slot).
Edit: Not all of them. I misread some of the conditions. Updates are possible.
Shows that replace an existing show in the same timeslot are done as "adds".
The logic for handling the adds in ApplyMergedScheduleEntriesToStore really treats them as what you would intuitively expect for an "update" if the OriginalStoredObjectId property is set on the new ScheduleEntry. (It will generate a delete for the entry specified by OriginalStoredObjectId). But the MergeScheduleEntries logic never sets this, in fact, I don't see anything that does!!.
Maybe this means that the updated ScheduleEntrys are getting added to the DB, but since the old ones in the same timeslot are not removed, we don't always end up seeing them. Worse, it may be adding duplicates of them on every import.
Edit: I doubt this though.... the code path it seems to take in the debugger doesn't match what the variables/parameters are. After multiple imports of the same MXF with a conditional breakpoint in ApplyMergedScheduleEntriesToStore if the add, update or delete list is non-empty, it never hits that breakpoint. Thinking since I am not close to really understanding the MergeScheduleEntries logic and am afraid of screwing up cases with recordings, to just wrap it and have extra logic after the wrapped call to just handle the common straightforward case of an exact timeslot match with a non-matching program not set to record.
The cases where I see an "update" created in MergeScheduleEntries can only be reached if the time slot is changed, among other conditions I don't really get. (but overlapping the old time slot).
Edit: Not all of them. I misread some of the conditions. Updates are possible.
Shows that replace an existing show in the same timeslot are done as "adds".
The logic for handling the adds in ApplyMergedScheduleEntriesToStore really treats them as what you would intuitively expect for an "update" if the OriginalStoredObjectId property is set on the new ScheduleEntry. (It will generate a delete for the entry specified by OriginalStoredObjectId). But the MergeScheduleEntries logic never sets this, in fact, I don't see anything that does!!.
Maybe this means that the updated ScheduleEntrys are getting added to the DB, but since the old ones in the same timeslot are not removed, we don't always end up seeing them. Worse, it may be adding duplicates of them on every import.
Edit: I doubt this though.... the code path it seems to take in the debugger doesn't match what the variables/parameters are. After multiple imports of the same MXF with a conditional breakpoint in ApplyMergedScheduleEntriesToStore if the add, update or delete list is non-empty, it never hits that breakpoint. Thinking since I am not close to really understanding the MergeScheduleEntries logic and am afraid of screwing up cases with recordings, to just wrap it and have extra logic after the wrapped call to just handle the common straightforward case of an exact timeslot match with a non-matching program not set to record.
-
- Posts: 45
- Joined: Wed May 16, 2012 3:02 am
- Location:
- HTPC Specs:
Is there any simple workaround you guys can think of? Or is the only fix to delete eHome and rebuild it from scratch? The guide is wrong for some new shows tomorrow night.
-
- Posts: 391
- Joined: Thu Jun 09, 2011 1:34 am
- Location:
- HTPC Specs:
Redoing TV setup should do it without having to wipe out eHome.
I think I am very close to fixing this with a replacement for MergeScheduleEntries. I am not sure what will happen if one of the scheduleentries to be updated/removed has an associated recording scheduled. Also, a couple issues remain that I need to make workarounds for:
1. The private method of ScheduleEntries for finding the last entry at it before the start key seems to have a bug where it sometimes returns the entry before the one it should, or sometimes updates the enumerator to point to the entry you want but somehow the return value is in a screwy state where it doesn't test as null or not null. I think the former issue is a problem with the underlying SQLite seek.
2. For a few cases the adds of missing ScheduleEntrys are triggering calls to MergeScheduleEntries, which in turn trigger the adds again, etc, infinitely recursively. This may be an artifact of incorrect modifications being made due to #1, or maybe I need to move this fix code out to ScheduleEntries.Add(List<ScheduleEntry>) instead of MergeScheduleEntries. I'm kind of wondering if this infinite recursion being triggered is why the existing code is so conservative in which updates it applies.
The guide on my test machine is down to about 2 dozen entries not matching the MXF from probably 1000. The data getting updated at the last minute and ending up not updating in WMC seems to disproportionately affect Spanish and PBS-like channels, which explains why those have had a lot more repeats recorded with generic metadata and/or shows recorded just not matching what they are supposed to be over the years, even when the standard WMC guide was still using zap2it.
I think I am very close to fixing this with a replacement for MergeScheduleEntries. I am not sure what will happen if one of the scheduleentries to be updated/removed has an associated recording scheduled. Also, a couple issues remain that I need to make workarounds for:
1. The private method of ScheduleEntries for finding the last entry at it before the start key seems to have a bug where it sometimes returns the entry before the one it should, or sometimes updates the enumerator to point to the entry you want but somehow the return value is in a screwy state where it doesn't test as null or not null. I think the former issue is a problem with the underlying SQLite seek.
2. For a few cases the adds of missing ScheduleEntrys are triggering calls to MergeScheduleEntries, which in turn trigger the adds again, etc, infinitely recursively. This may be an artifact of incorrect modifications being made due to #1, or maybe I need to move this fix code out to ScheduleEntries.Add(List<ScheduleEntry>) instead of MergeScheduleEntries. I'm kind of wondering if this infinite recursion being triggered is why the existing code is so conservative in which updates it applies.
The guide on my test machine is down to about 2 dozen entries not matching the MXF from probably 1000. The data getting updated at the last minute and ending up not updating in WMC seems to disproportionately affect Spanish and PBS-like channels, which explains why those have had a lot more repeats recorded with generic metadata and/or shows recorded just not matching what they are supposed to be over the years, even when the standard WMC guide was still using zap2it.
-
- Posts: 45
- Joined: Wed May 16, 2012 3:02 am
- Location:
- HTPC Specs:
Redoing TV setup will still require recreating series recordings, etc, yes? I'll give it a try tonight I guess.
Thanks again for digging into the issue glugglug!
Thanks again for digging into the issue glugglug!
- garyan2
- Posts: 7474
- Joined: Fri Nov 27, 2015 7:23 pm
- Location:
- HTPC Specs:
Okay, very bad side effect on doing the above ... scratch that out of possibilities.garyan2 wrote:Correct. I was basically doing exactly what loadmxf.exe was doing to call the MxfImport.Import() in the mcstore.dll. I still don't have any real control over the import at that point and I don't plan on going any further than this. To try and manually perform all the imports I think is way too complicated and way too risky for my limited skills.
As much as I hate workarounds, I think setting the isOnlyWmis and isMergedService flags is the way to go. Just need to flesh out any side effects, if any.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io
Keeping WMC alive beyond January 2020. https://garyan2.github.io
-
- Posts: 45
- Joined: Wed May 16, 2012 3:02 am
- Location:
- HTPC Specs:
Gary does this mean I should revert back to epg123 1.0?garyan2 wrote:Okay, very bad side effect on doing the above ... scratch that out of possibilities.garyan2 wrote:Correct. I was basically doing exactly what loadmxf.exe was doing to call the MxfImport.Import() in the mcstore.dll. I still don't have any real control over the import at that point and I don't plan on going any further than this. To try and manually perform all the imports I think is way too complicated and way too risky for my limited skills.
As much as I hate workarounds, I think setting the isOnlyWmis and isMergedService flags is the way to go. Just need to flesh out any side effects, if any.
- garyan2
- Posts: 7474
- Joined: Fri Nov 27, 2015 7:23 pm
- Location:
- HTPC Specs:
I would, absolutely. It looks like in some cases, rather than changing what program the schedule entry references, it overwrites the program. In my case, I saw a generic program get overwritten with a specific program. Results was all generic schedule entries for that series was listed as that specific episode.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io
Keeping WMC alive beyond January 2020. https://garyan2.github.io
-
- Posts: 45
- Joined: Wed May 16, 2012 3:02 am
- Location:
- HTPC Specs:
OK, and you agree with glugglug that re-running Setup TV is the best way to reset the guide? I had previously used ObjectStoreRecoveryTask in the olden days...garyan2 wrote:I would, absolutely. It looks like in some cases, rather than changing what program the schedule entry references, it overwrites the program. In my case, I saw a generic program get overwritten with a specific program. Results was all generic schedule entries for that series was listed as that specific episode.
-
- Posts: 657
- Joined: Tue Dec 20, 2011 11:05 pm
- Location:
- HTPC Specs:
Can you revert to 1.0.0 from 1.0.1b1 just by replacing the bits and rerunning EPG123 setup? Or is the damage done and I really need to run a LiveTV setup as well?garyan2 wrote:I would, absolutely. It looks like in some cases, rather than changing what program the schedule entry references, it overwrites the program. In my case, I saw a generic program get overwritten with a specific program. Results was all generic schedule entries for that series was listed as that specific episode.
-Alan
- garyan2
- Posts: 7474
- Joined: Fri Nov 27, 2015 7:23 pm
- Location:
- HTPC Specs:
I honestly don't know yet. I suspect replacing the epg123.exe will repair most of the damage, but can't guarantee all. I think what is left will probably die out. I'm running an update right now but won't be able to see the results for quite a bit later today.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io
Keeping WMC alive beyond January 2020. https://garyan2.github.io