1.5.0 - Verifying MXF...

An evolving, supported alternative to Rovi
Forum rules
★ Download the latest EPG123 here: http://epg123.garyan2.net <> Setup guide here: http://epg123.garyan2.net/downloads/epg123_Guide.pdf
Space

Posts: 2100
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

Re: 1.5.0 - Verifying MXF...

#61

Post by Space » Sat Oct 31, 2020 12:06 am

After having no issues for the past few days, today's run failed (last log entry):

Code: Select all

[10/30/2020 7:50:14 PM] Entering VerifyLoad()
I'm guessing that there were a large number of "fixes" that needed to be done and it took too long, so the process was killed. I will run it again later to see if it is able to continue where it left off, but can't right now since prime-time recordings are taking place...

EDIT:
After waiting for the recordings to finish, the client kicked off the MXF import, however the MXF import failed. It also seemed to cause the WMC GUI to crash for some reason.

I then ran the client command line manually (with verbose option) to only import the MXF file and it succeeded, but it also caused the WMC GUI to crash.

So I'm not sure why WMC crashed (and auto-restarted). I mean the first run failed the MXF import, the second one succeeded (as did the VerifyLoad), but still caused WMC to crash. WMC was just running, not doing anything (no playback, recording, etc.).

Note also that the VerifyLoad found many shows that it Removed/Replaced/Added/Changed. It reported that it correct 37 of them, but there were at least double that amount of INFO logs indicating one of those "fixes" being made.

I've added the "-verbose" option to the Task Scheduler run of the client so I'll be able to see how many fixes it made in the initial run and then the second run if it happens to be killed during the first one.
Last edited by Space on Sat Oct 31, 2020 10:09 pm, edited 1 time in total.

jamestx10

Posts: 9
Joined: Tue Jul 05, 2011 6:56 pm
Location:

HTPC Specs: Show details

#62

Post by jamestx10 » Sat Oct 31, 2020 2:08 pm

Last two runs have also failed for me again. Let me know what if any info you need from me to help out with this.

User avatar
garyan2

Posts: 5522
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#63

Post by garyan2 » Sat Oct 31, 2020 4:05 pm

I just released v1.5.1.0 to make the verification process optional. Please update as soon as possible. If you still want to run the verification, you will need to add "-verify" in the command line for the epg123Client.exe.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

Space

Posts: 2100
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#64

Post by Space » Sat Oct 31, 2020 10:12 pm

Does 1.5.1.0 include the change to make all verification fixes take place after they are all detected and not as they are detected?

User avatar
garyan2

Posts: 5522
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#65

Post by garyan2 » Sat Oct 31, 2020 10:37 pm

Space wrote:
Sat Oct 31, 2020 10:12 pm
Does 1.5.1.0 include the change to make all verification fixes take place after they are all detected and not as they are detected?
No. It is strictly 1.5.0.0 with the verify being optional. I'll get to trying to apply the fixes all at once at the end soon... I was a bit disappointed that the MXF changes had no effect on the results.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

Sancho

Posts: 145
Joined: Wed Jun 13, 2012 9:52 pm
Location:

HTPC Specs: Show details

#66

Post by Sancho » Sun Nov 01, 2020 2:23 am

garyan2 wrote:
Sat Oct 31, 2020 4:05 pm
I just released v1.5.1.0 to make the verification process optional. Please update as soon as possible. If you still want to run the verification, you will need to add "-verify" in the command line for the epg123Client.exe.
Maybe a stupid question, but where exactly does one add "-verify"? I mean such that it's included with each run of the scheduled task. Running it manually, I guess this would work:

"c:\program files (x86)\epg123\epg123client.exe -verify" -i c:\programdata\garyan2\epg123\output\epg123.mxf -verbose

but that doesn't seem to be the point; or is it?

Apart from that, I had some strange discontinuities between the PC that was having the problem, wherein I cleared a few channels' guide info in the Client as part of the troubleshooting, and my main HTPC that never experienced the problem, and has been running 1.5.0.0. The latter guide data includes channel 876, as it always has, but the problem PC suddenly lacked that channel in the guide. That made sense, as Comcast had moved it into the 940s somewhere. When I looked at the main HTPC via the Client GUI, I saw 876 on the left side, but not on the right side. I then manually deleted on the left side.

What's weird is that I never did that on the problem PC. Somewhere in all the troubleshooting and manually running the client to import the MXF file, that channel got removed (from the left side, and thus the displayed guide).

User avatar
garyan2

Posts: 5522
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#67

Post by garyan2 » Sun Nov 01, 2020 2:45 am

Yes, the -verify will need to be manually added to the task, but it should be added in the arguments field... not sure if what you have there will work. It should show

Code: Select all

"c:\program files (x86)\epg123\epg123client.exe" -i c:\programdata\garyan2\epg123\output\epg123.mxf -verify -verbose
With automatch enabled, any movement of stations amongst the channels will be automatic. It should have unsubscribed and disabled the station from 876 and subscribed and enabled the station on 940 something.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

Sancho

Posts: 145
Joined: Wed Jun 13, 2012 9:52 pm
Location:

HTPC Specs: Show details

#68

Post by Sancho » Sun Nov 01, 2020 3:04 am

Ah, my mistake. Edit the scheduled task in the Task Scheduler. Got it. I was thinking it was somewhere in an epg123 GUI.

Thanks for the command line fix.

Comcast is kinda strange these days, but yes, I would have thought that if SD didn't list it anymore (i.e., 876), then it should have been removed from the guide. It was already correctly listed as 940whatever, as well as in the "new" channel block above 999. Maybe I don't have the config GUI set correctly.

Sancho

Posts: 145
Joined: Wed Jun 13, 2012 9:52 pm
Location:

HTPC Specs: Show details

#69

Post by Sancho » Mon Nov 02, 2020 4:24 pm

Just a quick update: I modified the scheduled task on all three HTPCs to run the verification. So far so good, but it's only been a couple three runs. The verification takes between 22-38 seconds with varying numbers of corrections. Also, got rid of the -verbose option.

User avatar
Gripweed

Posts: 68
Joined: Sun Feb 03, 2013 6:35 pm
Location: Scranton,PA_USA

HTPC Specs: Show details

#70

Post by Gripweed » Tue Nov 03, 2020 3:37 am

Gary,
On the EPG123 website you wrote...
The verification will probably work on small databases, those that download hundreds of stations and maximum days are at risk of having problems.
I took it to heart and did what I should have done a long time ago. I went through the list of channels that I process daily and got rid of the ones that I don't actually receive in my Comcast TV tier or watch (goodbye home shopping networks and other useless stations) and all the SD (standard definition, not Schedules Direct) stations. I went from 293 channels down to 108. Being a Senior with no young grandchildren I can probably rid myself of all the kids/cartoon shows in the future if I need to and get it under 100. I also switched from receiving 14 days in advance down to 8. It's early yet but so far so good. The one run that had a lot of verifications only ran 7 seconds. I think I'm going to be alright. Thank you for working so hard on this.

User avatar
garyan2

Posts: 5522
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#71

Post by garyan2 » Tue Nov 03, 2020 7:55 am

Sometime in the future you won't have to worry about that. I seem to have ventured into the rabbit hole and am deep in re-writing the client code... and boy did it need it! So far, I've cut the GUI load times in half, reduced the memory footprint by 75%, have a better sorting function, viewing enabled channels only is near instantaneous, and there is one really cool feature that pretty much no one will see. Filtering by sources has increased in time, but I don't expect folks to use that very often. Will work on that, maybe.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

Space

Posts: 2100
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#72

Post by Space » Tue Nov 03, 2020 8:36 am

Sounds like some great improvements. You've got me curious about the cool feature that no one will see.

Space

Posts: 2100
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#73

Post by Space » Fri Nov 06, 2020 1:27 am

So there were LOTS of verify fixes today, and it got to a certain point and then just died...

So it does seem that if there are a lot of changes to be made to the WMD DB that it will eventually die...

I feel that unless a way to avoid the process being killed is found, the verification needs to be separated from the "fix" portion of the code, so that the fixes can be done as quickly as possible. But I also think that that may not be enough if there are very many fixes to be done.

You may also have to somehow break up a long list of fixes in to separate updates, although I'm not sure if that is possible.

User avatar
Gripweed

Posts: 68
Joined: Sun Feb 03, 2013 6:35 pm
Location: Scranton,PA_USA

HTPC Specs: Show details

#74

Post by Gripweed » Fri Nov 06, 2020 1:55 am

Just out of curiosity, how many verify fixes is LOTS? Also, how many channels are you processing? I had 78 verify fixes today at 12:30 PM (Eastern time) on my recently reduced total of 103 channels (from 293). That took 8 seconds.

User avatar
garyan2

Posts: 5522
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#75

Post by garyan2 » Fri Nov 06, 2020 2:16 am

It's not really about how many changes there are, it is about how much time it takes to complete them all after the very first change. I believe on that very first schedule entry change, WMC will kick off a PVR task and when it is complete, it does a DisposeAll on the database which kill epg123 while it is still accessing the database.

I may be wrong on the technical, but that is what my observations are telling me right now. That why one of the methods that may work is to gather all the needed changes and don't execute until I am done scanning all the services.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

Space

Posts: 2100
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#76

Post by Space » Fri Nov 06, 2020 2:26 am

Gripweed wrote:
Fri Nov 06, 2020 1:55 am
Just out of curiosity, how many verify fixes is LOTS? Also, how many channels are you processing? I had 78 verify fixes today at 12:30 PM (Eastern time) on my recently reduced total of 103 channels (from 293). That took 8 seconds.
I have 362 channels for which I download listings.

There were 351 "fixes" logged in the log before the process was killed.

I then ran an import of the same MXF file again and it continued the fixes at the point where it left off and "fixed" an additional 99 items. This run did not get killed and it successfully went on to run the reindexing, etc.

It would be good if there were an option to the client to just run the fixes and kick off the reindexing (in other words, skipped the import of the MXF and just did everything that comes after that).

Here are the last few lines of the log from the first run that resulted in the process being killed:

Code: Select all

[11/5/2020 7:55:02 PM] [ INFO] Service MTVLIVE at 11/12/2020 5:00:00 PM: Replacing [SH00821001_0000 - [MTV Live Special]-[]] with [EP00236074_0186 - [MTV Live]-[#186 John Legend]]
[11/5/2020 7:55:02 PM] [ INFO] Verifying !Service!EPG123_60046: VH1HD...
[11/5/2020 7:55:02 PM] [ INFO] Verifying !Service!EPG123_63220: BHERHD...
[11/5/2020 7:55:03 PM] [ INFO] Verifying !Service!EPG123_59440: CMTVHD...
Here is the portion of the log starting around where the first fixes were started from the second run:

Code: Select all

[11/5/2020 9:05:19 PM] [ INFO] Verifying !Service!EPG123_60046: VH1HD...
[11/5/2020 9:05:19 PM] [ INFO] Verifying !Service!EPG123_63220: BHERHD...
[11/5/2020 9:05:19 PM] [ INFO] Verifying !Service!EPG123_59440: CMTVHD...
[11/5/2020 9:05:20 PM] [ INFO] Service CMTVHD at 11/9/2020 4:00:00 PM: Replacing [EP01419581_0082 - [Last Man Standing]-[s04e16 Three Sundays]] with [EP01419581_0037 - [Last Man Standing]-[s02e11 Mike's Pole]]
You can see that it started at CMTVHD where it left off on the first run.

User avatar
garyan2

Posts: 5522
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#77

Post by garyan2 » Fri Nov 06, 2020 6:06 am

Space wrote:
Fri Nov 06, 2020 2:26 am
It would be good if there were an option to the client to just run the fixes and kick off the reindexing (in other words, skipped the import of the MXF and just did everything that comes after that).
garyan2 wrote:
Wed Oct 28, 2020 4:29 pm
For those having problems with the verification, could you please download epg123client.exe v1.5.0.40. Extract it to a temporary folder and run the following command from a command prompt. The example is with the exe in the c:\temp folder and using the default MXF file location... replace folder paths as necessary.

Code: Select all

c:\temp\epg123client -i c:\programdata\garyan2\epg123\output\epg123.mxf -nogc -verbose
This version does not perform an import, it just does the verification so it should take <30 seconds to complete. Please send the trace.log file (c:\temp\trace.log) to support@garyan2.net.

Thank you,
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

Space

Posts: 2100
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#78

Post by Space » Fri Nov 06, 2020 6:16 am

Thanks Gary, I was aware of that version of the client that does not do the MXF load.

I was just suggesting that having the option to do the same thing with the regular client might also be a good idea just in case it ever needs to be done with a future version of the client which may have a modified process for handling the verification.

Of course if you fireproof it so that it never gets killed part way through, then it may be moot.

User avatar
IT Troll

Posts: 824
Joined: Sun Nov 27, 2011 9:42 am
Location: Edinburgh, UK

HTPC Specs: Show details

#79

Post by IT Troll » Wed Nov 11, 2020 12:05 am

I have been watching this thread with interest but haven't been brave enough to try out verification on my live system.

I am intrigued to know what it might find with my EPG. Is there a way to run it in "log only" mode so it goes through the motions, logs what it would fix, but doesn't actually make any changes?
Are you a Recorded TV HD user or want to give it a try? Check out the new community-made update; Recorded TV HD v2.1.1

User avatar
garyan2

Posts: 5522
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#80

Post by garyan2 » Wed Nov 11, 2020 12:22 am

IT Troll wrote:
Wed Nov 11, 2020 12:05 am
I have been watching this thread with interest but haven't been brave enough to try out verification on my live system.

I am intrigued to know what it might find with my EPG. Is there a way to run it in "log only" mode so it goes through the motions, logs what it would fix, but doesn't actually make any changes?
No, the software corrects the differences as it finds them. I thought about that but have been focusing my attention on a pretty big rewrite of the client code and working my way to make this work with less/no risk.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

Post Reply