1.5.0 - Verifying MXF...

[garyan2]

For those having problems with the verification, could you please download epg123client.exe v1.5.0.40. Extract it to a temporary folder and run the following command from a command prompt. The example is with the exe in the c:\temp folder and using the default MXF file location... replace folder paths as necessary.

Code: Select all

c:\temp\epg123client -i c:\programdata\garyan2\epg123\output\epg123.mxf -nogc -verbose

This version does not perform an import, it just does the verification so it should take <30 seconds to complete. Please send the trace.log file (c:\temp\trace.log) to support@garyan2.net.

Thank you,

[The Mac]

tracelog sent.

either a compile error, or my runtimes are hosed.

lol

[Space]

Just as an FYI and to clarify my previous posts...

The initial run of 1.5.0 resulted in the client exiting when trying to do the VerifyLoad().

I updated the client to the 1.5.0.20 version and it ran clean (with the -verbose option, although it did trigger an initial garbage cleanup due to not using the -nogc option).

So either that version fixed the problem, the garbage cleanup fixed the problem, or running the client twice resulted in it working for some reason.

I currently have the 1.5.0.20 client installed and have run a "save & execute" one time and it worked fine (the second time it ran clean), the VerifyLoad() completed successfully. I'm waiting to see if today's normal scheduled update works without any issue...

I assume there is no issue with leaving the 1.5.0.20 client in place and using it normally until an official fix is released (and I assume using 1.5.0.40 would not be a good idea for this, as it does not do the import)?

[garyan2]

For those having problems with the verification, could you please download epg123client.exe v1.5.0.40. Extract it to a temporary folder and run the following command from a command prompt. The example is with the exe in the c:\temp folder and using the default MXF file location... replace folder paths as necessary.

Code: Select all

c:\temp\epg123client -i c:\programdata\garyan2\epg123\output\epg123.mxf -nogc -verbose

This version does not perform an import, it just does the verification so it should take <30 seconds to complete. Please send the trace.log file (c:\temp\trace.log) to support@garyan2.net.

Thank you,

All, I have replaced the zip file for v1.5.0.40 to include a .config file for .NET. I forgot that without the installer setting things up for you, a "portable" install needs that file for Win7.

[The Mac]

as space did above, i also replace the live epgclient with 1.5.0.20 and got a successful import.

on a side, how do you know when a garbage cleanup has been run, or needs to be run, and how to kick one off?

[garyan2]

I might have to abandon this. Even having all my interactions with the database inside a try/catch, embedded in a try/catch, which is embedded in a try/try catch, the process can still be terminated. The only thing I can think of is what I see in the client when I add or undelete a channel. Ten seconds after I do that, WMC fires off some PVR task which kills all threads connecting to the database and that is pretty much what is happening here.

I'm going to do a little more research, but I will probably post a new version with the verification process being optional (user will have to modify the task in order to enable it). I don't want to completely get rid of it yet. At least get a version out there that includes the fixes for problems with 1.4.x.x without the concern of the verification process causing epg123client not to complete.

[garyan2]

on a side, how do you know when a garbage cleanup has been run, or needs to be run, and how to kick one off?

When epg123 performs a garbage cleanup, it writes a date/time in the registry 4 days and 12 hours in the future. So for any imports, if the registry value is in the past, it will kickoff a garbage collection. The command is "c:\windows\ehome\mcupdate.exe -dbgc -updatetrigger" (without quotes). The -UpdateTrigger just prevents mcupdate from trying to download/update packages from the MS servers which are turned off.

[The Mac]

Thanks

[SwissCheese]

garyran2 wrote:

"I'm going to do a little more research, but I will probably post a new version with the verification process being optional (user will have to modify the task in order to enable it). I don't want to completely get rid of it yet."

Or perhaps add a GUI checkbox for creating the Task to enable the verification process, next to the GUI checkbox
that enables garbage collection. =:-)

[Space]

I might have to abandon this. Even having all my interactions with the database inside a try/catch, embedded in a try/catch, which is embedded in a try/try catch, the process can still be terminated. The only thing I can think of is what I see in the client when I add or undelete a channel. Ten seconds after I do that, WMC fires off some PVR task which kills all threads connecting to the database and that is pretty much what is happening here.

I'm going to do a little more research, but I will probably post a new version with the verification process being optional (user will have to modify the task in order to enable it). I don't want to completely get rid of it yet. At least get a version out there that includes the fixes for problems with 1.4.x.x without the concern of the verification process causing epg123client not to complete.

1.5.0.20 seems to have fixed the problem for at least two people in this thread (myself and The Mac), I know you said you made one small change (other than the extra logging), so that may have fixed at least one of the problems causing this (although I think at least one person still had the issue with 1.5.0.20).

It would be a shame to get rid of this feature all together. Is there no way to detect the thread being killed? At least then you can just skip that update and continue (or abandon all "fix" updates, if necessary) and continue with the rest of the processing.

I know when a process is hard killed in Unix, it first sends a soft kill so that the process has the opportunity to exit gracefully, but if it does not die within a certain time-frame, it is force killed. Maybe you can monitor for any signals being sent and disconnect from the DB before the process is killed, maybe that will prevent the force kill. Another option might be a watchdog process that detects when the client process ungracefully exits and restarts it with the verification disabled.

I think it might be a good idea to disable the "fix" feature, as you described, so that testing can still be done by those who want to help diagnose the problem. If you do eventually fix it, then it can be enabled for everyone by default.

You've only been working on this problem for a short time, so abandoning it seems premature. Disable it by default for now, and you will have a longer time to work on it. There is no rush. It's a great thing to have (and I will continue to use it, since it is working for me), but most people will not even know what they are missing.

[garyan2]

There's still hope. I think I found a way in the MXF file to change some of the WMC logic so that it won't reject as many changes (not all). I can also change how epg123 does its' verification in that rather than make the updates as I find them, I just collect all the adds/updates/removes and then do them all at once. This will give me more time to complete all the import actions.

I got some sleep last night (I think) and realized the reason why my development systems aren't affected. The PVR task that kills the client will only trigger if WMC has ever had a recording request... doesn't matter if the request is active or not. My development systems are clean and never had a recording request so the PVR task never fired. So for those that are having this problem, their systems have had a recording request and as soon as epg123 makes a change to the schedule entries it has around 10 seconds (plus/minus) to complete before it gets killed.

This is also why multiple runs can appear to "fix" the problem since the counter starts on the first change and previous changes may have already been committed. Now understanding what is going on, I can work on a better solution.

[Space]

Are you saying that the problem is that the changes you are making to the DB are taking more than ~10 seconds to complete, so the process is being killed?

Is each change being committed immediately, or is there just one commit at the end of all changes (I assume there is not just one commit, as then multiple runs would not "fix" the problem)?

You seem to imply that there are multiple commits, since you say there could have been commits that succeeded in a prior run.

Do separate commits restart the timer, or is it started when you make the first change, regardless of the number of commits?

Is the timer started when you connect to the DB and then make a change? Does the timer stop when you commit or when you disconnect from the DB?

Doing the updates all at once seems like a good idea, but might it still result in > 10 seconds if there are many changes? Perhaps if there are more than a certain number of changes you can split them in to batches and apply them separately, assuming that there is a way to reset the 10 second timer in between.

I assume that you are thinking that the reason I and others had this problem is because when you run the verify routine for the first time, it is most likely going to find a lot of issues (since there are most likely an accumulation of issues in the DB from the past several updates rather than just from the current update), whereas running it subsequently will most likely find less issues, since it will only find discrepancies that were created with the latest update (but it is still possible for it to time out if there are many discrepancies that need to be fixed caused by the current update).

As for changing the WMC logic in the MXF, I'd be interested in knowing some more about that.

[garyan2]

As soon as I make a change, the clock starts and I have a limited amount of time to complete all my actions. My imprecise 1 Mississippi - 2 Mississippi method when dealing with adding channels was a pretty consistent 9/10 Mississippi count. The log file I saw yesterday was around 13 seconds from first change to terminated process. That is why making all the changes at the end will mitigate the risk. After the verification, the client does the automatch routine and then kicks off the reindexing which combined shouldn't take long at all.

Each change I am currently doing is to each individual schedule entry when I find a difference. Whether they actually get committed at that time or sometime later doesn't matter, but the change is made. Again, the trace.log file I saw yesterday using v1.5.0.40 showed after the initial incomplete run, the next run's first change was after the last change implying they were all committed as the changes were made.

For the MXF change that may alter how WMC performs its logic, I'm doing some test runs now. I'll know later today when SD updates their cache so I can compare how many changes were required between systems with and without the MXF change. I'm hoping this method will take care of all those generic/specific/generic changes that we've seen the last 2 weeks. I believe the changes in start times, end times, and some overlaps will still exist.

[Gripweed]

Gary,
I don't know if you remember this but I did a little Googling and it appears that this is not the first time you've been around the block with this 10 second issue.

viewtopic.php?p=137356#p137356

[garyan2]

Yah, the nightmares keep coming back.

Well, my first go at having the MXF file change the WMC logic didn't work. No differences between the 2 setups. I'm going to try another way, but if it doesn't work then I will be just guessing and never knowing whether it is even possible or not. Just need to wait on the next round of updates from SD.

[Space]

Once you make the first change to the DB and the timer starts, does that doom the entire process to death within ~10 seconds? I mean if you disconnect from the DB, does it still get killed?

If it does get killed before the DB changes have completed, are you able to kick off the process again automatically with an option to just do the verify, automatch, and reindexing? This is a messy way to do things, but if there is no way to prevent the 10 second kill, then there may be no other way (if you can be 100% sure that all processing after the first DB update will take less than 10 seconds, then I guess you don't really need this, but you can never be 100%).

[WMCLives]

Hi, got this today, you care or just try again?

Beginning epg123 client execution. version 1.4.10.0

[10/30/2020 4:41:03 PM] [ INFO] Successfully started the ReindexSearchRoot task. Exit code: 0
[10/30/2020 4:41:03 PM] Exiting runWmcTask(ReindexSearchRoot). SUCCESS.
[10/30/2020 4:41:03 PM] [ INFO] MXF file was created on 10/30/2020 4:23:00 PM
[10/30/2020 4:41:03 PM] [WARNG] The imported MXF file contained a WARNING in its status field.
[10/30/2020 4:41:03 PM] [ INFO] Completed EPG123 client execution.
[10/30/2020 4:41:03 PM] [ INFO] EPG123 client execution time was 00:17:38.6848679.

[garyan2]

If you look further up in your log file, you will see the source of the warning during downloading of the guide data from Schedules Direct. It will probably be either a time discontinuity on a station or your SD membership is about to expire.

[WMCLives]

This it?

[10/30/2020 4:23:21 PM] [ INFO] TMDb catalog search for "Victory at Sea" from 1954 found 1 results.
[10/30/2020 4:23:21 PM] [WARNG] Problem occurred during getAllMoviePosters(). Did not process all movie image responses.

[garyan2]

That is the source. It didn't check/verify all the movie posters. Your guide is fine, just there may be a number of movies that didn't get the opportunity to find a poster on TMDb if SD/Gracenote didn't have one available. Your guide is fine.