1.5.0 - Verifying MXF...

An evolving, supported alternative to Rovi
Forum rules
★ Download the latest EPG123 here: http://epg123.garyan2.net <> Setup guide here: http://epg123.garyan2.net/downloads/epg123_Guide.pdf
Space

Posts: 2100
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

Re: 1.5.0 - Verifying MXF...

#81

Post by Space » Wed Nov 11, 2020 1:04 am

I've been running the latest client with the -verify option enabled in the task scheduler.

I haven't noticed any issues running this way. Occasionally the Verify step fails to complete, which results in all steps that normally occur after that step to also fail to execute. However, this is really not that big a deal, as when the next run occurs, it will run Verify again, and it will just continue where it left off in the first run (as well as fixing any new issues). While I can't be sure there is no lasting issues, I have not noticed any signs of database corruption when the Verify step fails to complete.

Note that when the Verify fails to complete, the database has still been updated, just like it is with prior clients that do not have this new feature.

One of the steps that is skipped when Verify ends prematurely is the reindex task. This can cause changes that were made to the database to not be reflected in your scheduled recordings. However I think that reindexing is eventually kicked off anyway as I've seen changes to programming properly reflected in the recording schedule even after a run of the client that resulted in the Verify not completing. I think that reindexing occurs in the background when there is not much activity on your system, so it will eventually occur anyway if your system remains idle for a short time. But even if reindexing does not occur, it just means that your recording schedule remains the same as it was, so you should not miss any recordings unless there was a last minute change to shows you are recording within the next 24 hours.

There are other things that will not occur after an incomplete Verify such as automatching for new channels or Series modification of any new Series you may have created since the last run of EPG123, but these are minor things and only affect new channels that were added to your guide and new Series, and will be picked up the next time the client runs successfully.

I have hundreds of channels in my guide, if you have a smaller amount, you may never see the issue with the Verify section since it is related to the time it takes to scan all the channels, and less channels means less time.

User avatar
garyan2

Posts: 5522
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#82

Post by garyan2 » Thu Nov 12, 2020 6:38 am

I think I found the solution to this by just changing the way I open the WMC database. The PVR Recovery still runs, but it doesn't kill the client now. I'm going to run some real world tests on it to see how it does.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

Space

Posts: 2100
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#83

Post by Space » Thu Nov 12, 2020 6:55 am

Sounds good, hopefully it doesn't cause DB corruption :D

User avatar
garyan2

Posts: 5522
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#84

Post by garyan2 » Thu Nov 19, 2020 9:44 pm

Anybody want to do some beta testing of the next release? I haven't been able to make it "fail" with the new version and the option to verify is by default enabled now. I do have a "-noverify" command switch in case it is still happening, but I believe my new method of opening/using/closing the database will prevent the PVR task from killing the client.

I still have a couple things to work on prior to releasing it, but that is centered around the client setup.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

Space

Posts: 2100
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#85

Post by Space » Fri Nov 20, 2020 3:32 am

I should probably not, but I can try it.

Just yesterday I had an issue where WMC was not recording shows even though it indicated it was. Looking in the event viewer, there were two process that had issues, ehShell.exe and SearchProtocolHost.exe (which seemed to crash).

I think that ehShell is the WMC GUI, so not sure how that could cause problems with recording...

It ran client maintenance yesterday at 4pm until around 5pm, then went to sleep, woke up at 6pm to record a show for 30 minutes, went back to sleep, then woke up at 7:45pm to run EPG123.

It seemed to run to completion without any problems, although I saw this in the log:

Code: Select all

[11/18/2020 7:52:38 PM] [ INFO] Removing FLKTV from channel 229 in lineup EPG123 Verizon Fios Freehold - Digital (Freehold).
Which would not normally be unusual, except for today's run (see below).

After EPG123 ran, when 8pm came around, the two shows scheduled to record, did not record, even though it indicated that they were (I noticed it around 30 minutes in to the show). No errors were logged, they just weren't recording. I tried to stop/restart the recording, but even though it said it was stopped/started it did not actually start recording anything. I then rebooted the machine and when it came up, the recordings started fine. I am wondering if I can run something to fix this without rebooting, maybe one of the tasks in task scheduler?

Anyway...

Today the load of the MXF file took longer than usual and ended up finishing 2 minutes past prime-time (show started recording 2 minutes earlier). Then it did it's Verify thing successfully, however then I see this:

Code: Select all

[11/19/2020 8:04:13 PM] [ INFO] Verifying !Service!EPG123_DUMMY: DUMMY...
[11/19/2020 8:04:13 PM] [ INFO] Checked 152810 entries and corrected 176 of them.
[11/19/2020 8:04:13 PM] Exiting VerifyLoad()
[11/19/2020 8:04:15 PM] [ INFO] Removing FLKTV from channel 229 in lineup EPG123 Verizon Fios Freehold - Digital (Freehold).
And that's it, no more log entries. I assume the process crashed. I have never seen it crash after printing the "Exiting VerifyLoad()" log entry (usually it happens somewhere DURING the Verify).

Also, why did it remove FLKTV again, when it should have already done it the prior day? Perhaps that is what caused ehShell to crash yesterday?
Looking in the guide, I think that FLKTV is gone now, so it did remove it, at least after it's second try...

As for today's recording, they seem to be recording fine...

I'm going to run EPG123 again to load the MXF file to see if it crashes again or completes successfully., but I have to wait until the recordings are done after 11pm ET...

Space

Posts: 2100
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#86

Post by Space » Fri Nov 20, 2020 5:07 am

I ran the client via CMD line just to load the MXF and do the other stuff and it ran fine. Although it took a long time again to "load" this MXF (about 14 minutes). I don't know much about how loadMXF works, but since it already had loaded this MXF file, it shouldn't actually have changed anything in the database. The EPG123 Verify also said that it fixed 0 entries (as expected). So not sure if this is something specific to the MXF or if there are other issues that caused the load to take a bit longer (normally it takes about 5 to 12 minutes, so it was not THAT much longer than usual).

User avatar
garyan2

Posts: 5522
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#87

Post by garyan2 » Fri Nov 20, 2020 5:04 pm

I think the ehshell problem caused a database recovery which uses a local backup of your tuners and lineups. If the recovery happened after the first time epg123 removed FLKTV, it probably restored a backup that still had that station present so epg123 would have to remove it again.

So with the new beta version, epg123 does checks a little different than what it did before. To avoid accessing the database prior to a garbage cleanup or mxf import, I look at the scheduled tasks to determine if a recording is in progress and I look at the registry to see when the next scheduled recording is. Now, garbage cleanup will not start unless there are no recordings in progress AND there are no future recordings scheduled within the next 60 minutes. The MXF import will be the same as garbage cleanup except any future recording just needs to be more than 15 minutes in the future.

Speaking of garbage cleanup... You are not doing yourself any favors by updating the guide multiple times a day. Though the lineups, services, and programs from the new MXF file won't take additional space if they already exist in the database, that is not true for the schedule entries. So using your numbers above, every time you import a MXF file (even the same MXF file), you are adding 152810+ objects in the database. The unused/dead schedule entries will not be removed until garbage cleanup. The more dead entries there are, the larger your database is and the longer garbage cleanup will take to complete.

As far as the recording issue you had... I have no clue on that one yet.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

Space

Posts: 2100
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#88

Post by Space » Fri Nov 20, 2020 7:11 pm

I'm not sure if a DB recovery was done. Is there an easy way to tell? I think one way is to check the database folder to see if there is an active one and another one with recent dates (from a couple days ago). Is there another way as well?

I don't update the guide multiple times per day, just once per day. The only reason I loaded the MXF file that second time was because the first time the client ran, it died after logging the "Removing FLKTV..." entry. I wanted to see if running it again would allow it to get past that spot. Did you find it strange that the client stopped/crashed at this point?

I didn't know that loading the same MXF file duplicated the objects. I'll have to keep that in mind when if I decide to do this again (hopefully the new client will not exit prematurely anymore, so I may not need to).

User avatar
garyan2

Posts: 5522
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#89

Post by garyan2 » Fri Nov 20, 2020 7:33 pm

Right, the easiest way would be to look at the database folder and see if there is an "old" one with a date a couple days ago. There are a registry I think that will have the last time a recovery was done, and possibly the recovery task will show, but just take a look at the database folder.

Sorry, I thought you said earlier that you updated your guide twice a day... my mistake.

The process killing of the client can happen at any time, not necessarily during the verification process. I'm not too surprised it happened in that very small timeframe after the verification during the automatch and indexing. That PVR task that kills the client is initiated on the very first schedule entry change and will kill the client whenever it completes regardless of what the client is doing.

I didn't realize the bloat of schedule entries until I started working on the verification process. I would be importing/verifying the same MXF file over and over and noticed that database size would keep increasing. When I noticed that, it was around 1.2GB and running a garbage cleanup took it back down to around 300MB. So a lot of bloat the more imports there are between garbage collections.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

Space

Posts: 2100
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#90

Post by Space » Sat Nov 21, 2020 8:24 am

I checked the DB files/folders and beside the current one, there was only one from February of this year, so there was no DB recovery.

I suppose if you log what is about to happen before it happens, it could have killed the process right after logging that message and before it actually removed the channel.

When the new EPG123 checks to see when the next recording is starting, does it see when the program actually starts, when the recording actually will start (perhaps earlier due to pre-padding) or when the computer is set to wake up to start the recording (5 minutes before the recording is supposed to start). I want to schedule EPG123 to run at a time where I am sure it will not be delayed due to recordings starting.

User avatar
garyan2

Posts: 5522
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#91

Post by garyan2 » Sat Nov 21, 2020 4:09 pm

Yes, the "next" recording time I am looking at in the registry looks like it has a 5 minute buffer on top of any pre-padding. So technically my 15 minute buffer is actually 20 minutes.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

Space

Posts: 2100
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#92

Post by Space » Sat Nov 21, 2020 5:01 pm

Could you maybe change this to 10 minutes so that with the 5-min extra it is actually 15 minutes?

If I have to make the EPG123 start time earlier, it may result in the PC being idle for too long after EPG123 finishes and it may go to sleep. I want to avoid the PC going to sleep and then waking up again within 5-10 minutes.

Or perhaps you can even have EPG123 not tell the PC it is OK to go to sleep if it is within 10 or so minutes of a recording starting after it has completed.

User avatar
garyan2

Posts: 5522
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#93

Post by garyan2 » Sat Nov 21, 2020 8:07 pm

I may change the time. I am being pretty conservative here because everyone has a different setup, fast and slow, and I am also thinking about the reindexing taking place after the import, verify, and automatch. No, I am not going to add a dial for everyone to tweak on this. There is an option " -force" that will ignore whether a recording is active or upcoming. That option will also happen to disable garbage collection by the way.

I'm not going to mess with how I handle sleep settings. To do what you propose, I would have to have the client running up until the next recording starts so I can restore the sleep settings.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

Space

Posts: 2100
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#94

Post by Space » Sat Nov 21, 2020 9:05 pm

I already have a watchdog process that prevents the PC from going to sleep when the indexing process is running so this extends the insomnia after EPG123 exits, but even that can end too soon before the recording is due to start and the PC will go to sleep.

It's a bit strange because I had thought (based on posts in the forum) that if the PC is awake 10 minutes before a recording is due to start, the PC will not go to sleep in that time (WMC tells the PC to not go to sleep starting 10 minutes before the recording is to start), but in my experience this does not (always?) work.

EDIT:

As far as the client continuing to run until the recording starts (or 5 minutes before it starts), yes, that is what I was proposing. It would only be in effect if you are running the client in background mode, not if it was initiated via the GUI. It would only do this if it was within 10-15 minutes of the recording starting, otherwise it would exit as normal.

I can understand why you wouldn't want to do this, so no big deal, just a suggestion. I can probably come up with something with my watchdog process to handle it if I need to. What registry value are you using to determine when the next recording is starting?

User avatar
garyan2

Posts: 5522
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#95

Post by garyan2 » Sat Nov 21, 2020 9:31 pm

The registry is [\\HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Media Center\Service\Recording] "NextRecordingAt"
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

Post Reply