Olympics preparation

[Space]

Gary,

I usually record all the Olympics coverage on all the broadcast/cable networks. Just in case you had any doubts that I was crazy.

In any case, the Olympics schedule changes quite rapidly, so I would like a way to keep the listings as up to date as I can (although I know it will never be able to be 100% accurate due to propagation delays).

One problem with EPG123 is that it will not update the guide if there is a recording in progress. Since Olympic coverage is 24/7, there will pretty much never be a time when there is no recording in progress, so the plan is to do manual updates at least once a day and force it to update the WMC database while I watch it to resolve any potential issues with the in-progress recordings.

My plan is to disable the normal automatic update in the Task Scheduler and set up a batch file that I can run manually to do the update, but I want to be sure I am using the correct options in order to force the client to update the WMC database. I also believe you've said in the past that -nogc is auto-enabled if you use the "-f" option, but I included it anyway just to be sure (and I assume it will do no harm).

Here is what I plan to use, please let me know if you see anything wrong:

Code: Select all

"C:\Program Files (x86)\epg123\epg123.exe" -update
"C:\Program Files (x86)\epg123\epg123Client.exe" -i "C:\ProgramData\GaRyan2\epg123\output\epg123.mxf" -match -nogc -verbose -f 

Thanks!

[garyan2]

Looks good. That will work.

Sent from my SM-G965U using Tapatalk

[Space]

Can I use %ERRORLEVEL% from "epg123.exe -update" to determine if a new MXF file was created?

[garyan2]

You should be able to, I've never tried it. It will return a zero (0) on success, -1 on failure.

[garyan2]

I take that back, it will always return 0 for an update run regardless if the mxf file was created or not.

EDIT: I'll fix that in the next (pre-)release.

[Space]

Thanks!

The SD server is down for maintenance and I don't want it to import the same MXF file every time I run the batch script to see if the maintenance is over.

Code: Select all

[7/24/2021 7:15:36 PM] [ERROR] Failed token request. code: 3000 , message: Offline for maintenance. , datetime: 2021-07-24T16:30:00Z

Does this message mean it's been down since 11:30 am ET (16:30 UTC) today?

Returning "0" if it created a new MXF and some other values for the various other errors that may have caused it to not create the file would be cool, although just 0 and -1 is fine for my purposes and I'm not sure if anyone else cares about this.

I looked in to checking the modified date of the MXF to determine if it was new, and while it's possible in a batch file, it's not that straight forward to do date math, and I need to be watching the Olympics, not figuring that out

[garyan2]

Yes, it has been down for maintenance since 1630Z. SD is updating the API to handle the images a little differently and it looks like it is taking a bit longer than expected.

[rswsr]

Is there a command you can add to the beginning of this script that will stop a running guide update that is waiting for the current recording to end?

[Space]

Gary will have the ultimate answer, but I am guessing "no".

But you should be able to kill the "epg123Client.exe" process that is running to stop it.

Just as a side note, I've been running the epg123Client.exe process to update the WMC database with the "-f" option to force it to run while WMC was recording multiple channels (up to 3) and have had no problems with the recordings or the update.

I believe some people have reported that updating while recording causes the recordings to fail, but this may only be under certain circumstances or on certain systems. Note that it is also possible to lower the process priority of the epg123Client process, although I don't know if that would help those having issues (probably only if the issue is with a slow CPU).

[garyan2]

Is there a command you can add to the beginning of this script that will stop a running guide update that is waiting for the current recording to end?

I would have to ask why? The client will eventually time-out and close itself, but only after 23 hours. So basically it will close so that it can run a new update with the latest listings.

The client, while waiting, barely uses any processor cycles. It will check your recording status once a minute until all recordings end and then proceed with the import.

[garyan2]

I take that back, it will always return 0 for an update run regardless if the mxf file was created or not.

EDIT: I'll fix that in the next (pre-)release.

The latest release (v1.6.7.0) will return a -1 from epg123.exe if it fails to create the mxf file.

[Space]

I take that back, it will always return 0 for an update run regardless if the mxf file was created or not.

EDIT: I'll fix that in the next (pre-)release.

The latest release (v1.6.7.0) will return a -1 from epg123.exe if it fails to create the mxf file.

Nice! Thanks for adding this.

I've been renaming/deleting the existing MXF file before running epg123 to create a new one, then checking if the MXF exists afterwards. This was much easier then checking the modification date of the MXF file in a batch file.

[Space]

Just a follow-up to using the "force" option when updating the WMC guide with EPG123Client.exe...

I hadn't updated in over 24 hours, so I decided to do it (while two Olympics recordings were in progress). This turned out to be a mistake...

It resulted in the database getting corrupted (I assume) and an automatic rebuild...

The recordings that were recording, of course, stopped and I had to reload the guide data with EPG123Client.exe since it was completely empty (and a popup saying I had to re-download it).

I noticed that the programs that were recording at the time of the initial update were no longer in the guide and were replaced with new programs. I can only assume this is what caused the problem.

It appears that WMC DOES update the schedule entries for programs that are currently in progress, or at least can cause catastrophic issues if the guide entries for those programs change (assuming that was the cause).

Another thing I noticed is that the "Favorite Lineup" that I created for the Olympics was gone, even though older "Favorite Lineups" I had created were still there. I can only assume that the backup that was restored after the automatic recovery was not the latest one. I also noticed several "Matching XXX to channel XXX" log entries in the EPG123 log that weren't there when I updated the guide prior to the DB corruption. I assume this is also evidence that the backup that was restored was an older one...

One weird thing is that the Series I created for the Olympics was still there, even though that was set up around the same time I set up the Olympics "Favorite Lineup", so that is a bit of a mystery, unless those are not stored in the same place...

But maybe my understanding of what happened here is incomplete...

EDIT:

I looked at the filesystem and see that the DB was moved from mcepg2-0 to mcepg2-1.

These are the dates of the latest backup file in the backups folder under the old mcepg2-0 folder:

lineup - April 24th, 2021
subscriptions - April 24th, 2021
recordings - August,1 2021

So this would explain why the Series seemed to be up to date, but the channels (and favorite lineups?) did not. The lineup was last backed up in April. I definitely have had new channels added to the lineup since April. It looks like WMC does not backup the lineup after it is updated...

[garyan2]

You've got the understanding right, and thanks for filling in some blanks for me with this post. I know from past observation that whenever you add/change recording requests, WMC will create a backup for the requests almost immediately. Without any critical thinking, I assumed WMC also performed a backup of the lineup as well, but from your data that is obviously false. This now answers why I see some folks reporting a database recovery and then their tuners are missing... old data.

I am going to add an automatic backup run in the client that will run the day prior to the automatic garbage collection. Of course, the same rules for the backup will be as with the GC; If the -nogc or -f switches are used, then no backup. I am also giving the backups a 10 minute buffer to any upcoming recordings rather than the 60 minutes for the GC.

To hypothesize on why your lineup was old... if you haven't changed your tuners or scanned for new channels, then you haven't really changed the lineup. You have changed guide mapping, but not the tuners/channels themselves. I think maybe guide mapping is not to a sufficient level of change for WMC to create a backup.

[Space]

I wonder exactly what changes in WMC would trigger a backup?

From my experience here, it looks like the following do NOT trigger a backup:
Enable/disable a channel
Create a new "Favorite Lineup"

I wonder if unchecking a tuner for a channel will force a backup?

I also wonder what exactly is saved in these backups?


As for your changes to add an automatic backup run. Will this put the backup in the location that WMC uses to restore the backup after a DB automatic rebuild? Or will it save it to a zip file like the "backup" button does in the WMC client? To be honest I don't really understand how these are related...

If it is not putting it in the place it needs to be for the auto-rebuild to use it, then can you provide a procedure for using that backup to restore the DB after an automatic rebuild is done? These EPG123 backups of the WMC DB would be a lot less useful if they are not used by the automatic rebuild initiated by WMC.

Also, I'm not sure why you are linking this backup to the GC. I do my own separate GC and always run with the -nogc option, so that would mean I would never get a backup (unless you add an option to force the backup).

Also, doing backups every 5 days seems like overkill, it would be much better if you were able to tell if there was a change to the lineup (and any other thing that is backed up, such as Favorite lineups) and then only make a backup then (perhaps it would be easier to compare the backup to the previous backup and don't keep the new backup if the old one is the same).

[garyan2]

I am tying it to GC because it makes it periodic. The backups will only be for WMC, I am not going to create a zip file from it so in case there is a database recovery by WMC, it will be from a pretty recent backup. The [Backup] feature in the client to create a zip file will always be a user directed action. I am not going to fill the users backup folder with automatic backups.

I am not going to make it an option or separate it from the GC. This is how I recommend the user to operate in order to keep WMC as tip-top shape as we can make it. If you don't want epg123 to handle the GC, and created your own task to do it when you want to do it, then you can also set up a task to perform the WMC database backup. The command is "mcupdate.exe -b -nogc". You may even be able to combine the backup (-b) with the GC but I'm not sure what it will do in that case. Database backups can take up to a minute for large databases that I have seen, but no data to prove that. My backups take a few seconds.

I am not going to turn EPG123 into a HAL. We all know how that turned out. What you are suggesting is on a level of AI the epg123 will almost have to observe your every action and decide what you would want, or what I think you would want.

[Space]

What is the difference between the backups done by "mcupdate.exe -b -nogc" and "your backups"? Why is there a difference in the time taken?

The reason to only backup when things change is so that you have a backup for each of the prior three changes to the database. It appears that WMC keeps the last three backups and if you backup every 5 days, I'm assuming that the last three backups will most likely all be the same and not reflect the state of the database after the three last changes.

[garyan2]

The client [Backup] will perform "mcupdate.exe -b -nogc", wait for it to complete, and then copy those new WMC backup files into a zip file. Because it does the -b option, all 3 files will be created at the same time. The lineup backup file is for all your tuners, channels, station mapping, favorite lineups, custom channel numbers and callsigns. The recordings backup is all your current and past recording requests. The subscriptions backup is pretty much nothing nowadays; it was basically a list of packages installed (ClientUpdate, MCEClientMaintenance, MCEClientUX, NetworkMetadata, Broadband, EPGListings, SportsSchedule, and SportsV2) that are no longer supported by MS.

You're right. For most, the last 3 backups will probably be identical except for maybe some changes by the user for recording requests. But that is way better than restoring a backup that is a year old. When was the last time you ventured into the ehome backup folder and restored a file? If you restore a lineup while WMC is already setup you will find that all your tuners and possibly channels are doubled up. Same with the recordings. So the backups are only there for a database recovery, which by the way, only selects the newest file. I'm not seeing what we are missing out on here.

If you want a backup for every change to the database, open the client and click the [Backup] button whenever you make a change.

EPG123 is not resident in memory, embedded in WMC, or monitors your activity. How would it know when there was a change in the database that you might feel was important enough to make a backup for?

[Space]

I was suggesting that it make a backup every X days, but only retain that backup if it is not identical to the previous backup (which may be the one performed X days earlier, or perhaps initiated by WMC when you created a new Series or whatever other action triggers a new backup). Just doing a file compare of each XML file to the prior one and if there are any differences, keep the new backup, otherwise why keep the backup if it is just a duplicate of the one prior?

I know very little about these backup files, but I do see two "lineup" backups in my old DB folder and they were taken 2 minutes apart and they are identical (I used the DOS command FC.exe to check), so it does appear that the backup files will be identical if no changes were made.

This can be done on a per-backup file basis, meaning you will probably never retain multiple copies of the "subscriptions" backup, as it will probably never change. However the "recordings" backup will probably change quite often, so will probably be backed up and retained every time.

I just did an "FC" check of my two latest "recordings" backups and it displayed one line that changed, I had changed an existing recording from "Keep until: I watch" to "Keep until: I delete", and that was the only change. So even minor changes such as this cause a new backup file to be created, although it seems to only have retained 5 previous backups (all from today), so I'm not sure when it deletes older backups.

I agree that retaining prior backups for "lineup" may not be useful in many situations. But it can be a lifesaver in cases of DB corruption (if the latest backup is of a corrupted DB). But also, if someone makes major changes to the lineup and then wants to go back to a prior one, perhaps several weeks later, (and they didn't know to do a manual backup in EPG123 before making the change) they would be out of luck if the backup was gone. There would be more of a chance of the old backup they needed still being around if you only retained the backup file if it had changed (assuming WMC deletes older backup files once the number it has retained has reached a certain amount). Having 5 backup files, all identical, is not very helpful.

On another note... When you said WMC DB backups can take up to 1 minute, but yours (thinking you meant the process you use to do the backup) takes a few seconds, I thought that meant you were not using mcupdate.exe to do the backup, but I think I misunderstood and you were actually saying it only takes a few seconds to back up your personal WMC DB, but others DB (bigger and/or on a spinning hard drive) can take longer to back up.

[Space]

I added the "-b" to my separate garbage cleanup task and it appears to have worked ("mcupdate.exe -b -dbgc"). It appears that the first thing it did was create the three backup files before it started garbage cleanup.

The lineups backup contains a lot of timestamps and it appears that it will change every time a new day of data is loaded in to the guide for any channel. Both the recordings and the subscriptions backup file did not change from when they were last backed up, so I would say that they probably don't need to be backed up by the separate process, however it's probably not possible to only back up the lineups.

I wonder what the schedule is to clean up (delete) the backup files, I assume WMC deletes them at some point, as I've never seen more than a handful of them at any one time.