Big change coming

[garyan2]

I'm working on a pretty big addition to the EPG123 server. There will be included a service that will provide TCP services to include:

• download epg123.mxf and epg123.xmltv file using http
• view server trace.log file from web browser
• cache images downloaded from SD to provide to all clients
• detect when token access is no longer available and automatically login to SD

The first bullet means that a client only setup will no longer need to change any user settings of the scheduled task. Any program that uses the xmltv file will just need to use "http://<SERVERNAME>:<PORT>/output/epg123.xmltv" to import the file.

I am trying to optimize everything right now, but it is looking pretty good. Thinking of other scenarios I need to account for. I'm not sure about the portable installs since I now have to create the service, add firewall exceptions, and create a registry key for epg123. Maybe include a batch file to run as administrator, but not sure yet.

[Space]

Interesting... Pretty cool surprise!

Now that you have an HTTP server, maybe you can look in to simulating the Microsoft extended guide data server? This would act like a local proxy that takes the queries from WMC for extended guide data and then queries the needed data from an external site (not sure which sites would be best to provide this info).

You could put an entry in the local host file to override the original Microsoft domain that was used for this server and point it to localhost (the loopback address). This technique is similar to what was used for WiRNS to keep the ReplayTV alive when the official service went away (although it required emulating a DNS server to get the standalone ReplayTV to use the alternate IP address, so a little more complicated than this would be).

I know this is something you rejected in the past, but maybe you would like to take another look at it with fresh eyes. You'd probably have to reverse engineer the protocol that was used to get this data (it may be simple, I have no idea). Just thought I'd throw this out there again in case you were looking for something cool to do for your next challenge since you now have the HTTP server which is a large part of it.

As with all my suggestions, feel free to ignore it, just throwing them out there in case you have any interest...

[Gripweed]

Can we stay with what we have? EPG123 is working perfectly for me. As I get older I'm turning more, "if it ain't broke, don't fix it."

[alhaunts]

Will this fix the missing guide images from SD issue that some of us have been dealing with??

[garyan2]

Will this fix the missing guide images from SD issue that some of us have been dealing with??

It will address it by automatically requesting a new token, but it won't fix it. SD is going to have to find the cause on their end and do that.

[garyan2]

Can we stay with what we have? EPG123 is working perfectly for me. As I get older I'm turning more, "if it ain't broke, don't fix it."

I will say the current versions are time-limited. In the future, SD will require tokens in the image requests in order to download them. The image caching, as well as the auto-login are optional, but the service will be necessary to add tokens to image requests. So it may not be broke now, but when the requirements change it will be.

The update to the new method should be seamless and all I am doing is changing the image links from a Schedules Direct server to the service running on your machine where it will route the request to SD. Basically a middle man between SD and WMC for images. There will be no need to change anything on the user end to make it work.

[Space]

Can we stay with what we have? EPG123 is working perfectly for me. As I get older I'm turning more, "if it ain't broke, don't fix it."

I will say the current versions are time-limited. In the future, SD will require tokens in the image requests in order to download them. The image caching, as well as the auto-login are optional, but the service will be necessary to add tokens to image requests. So it may not be broke now, but when the requirements change it will be.

The update to the new method should be seamless and all I am doing is changing the image links from a Schedules Direct server to the service running on your machine where it will route the request to SD. Basically a middle man between SD and WMC for images. There will be no need to change anything on the user end to make it work.

Interesting, so using the new HTTP server as a proxy seems to have been the stimulus for these additional features. Will a new token be needed every X number of minutes/hours in order to get a requested image?

[garyan2]

The image access will be tied to your normal 24-hour token, but since the images are "separate" from your data downloads SD needs to track valid tokens/IP addresses. It is the transfer of that information to the gatekeeper for images that doesn't appear to working as intended right now and users are finding they no longer have access to images even though their token may only be an hour old. That is why I added the auto-login feature if when requesting an image we are denied with a specific error code.

[Gripweed]

I will say the current versions are time-limited. In the future, SD will require tokens in the image requests in order to download them. The image caching, as well as the auto-login are optional, but the service will be necessary to add tokens to image requests. So it may not be broke now, but when the requirements change it will be.

So this is totally about the images? It's not something that I really have any concern about. I use WMC purely as a PVR and not as a media server. I never actually play any of the shows on WMC and so never see the images there. I schedule the shows to record and then take the resulting files (after being processed into MP4s) and save them to either Emby, where they pick up some images, or process them with a program called TV Rename (https://www.tvrename.com/) that provides images for my wife's WDTV box (along with acting as her TV/Movie database). If it wasn't for my wife's total love and reliance on that old WDTV box and the system I built around it I'd just throw away WMC and stick with Emby. I guess I will do what I should have done a while ago and turn off the images retrieval in EPG123 and stick with the version of EPG123 (1.6.7.0 or 1.6.7.3) that I have now.

[garyan2]

Then you definitely won't see a change either way. EPG123 and WMC won't download any images if you never pull up the guide to look at the listings. The image downloads are "on demand".

EPG123 will provide the links to the images to WMC, and WMC will only download the images if you look at the guide and then only for any program that has the focus. Once the image is downloaded, then it is cached locally and not downloaded again until the image expires and you again look at a program requiring the same image. Either way, do what you're gonna do.

[Gripweed]

I guess I'm going to have to go along with your new version when it comes around. I just (thought I) turned off all images retrieval by unchecking all the checkboxes in EPG123 Config that had to do with images and logos. I also cleaned the cache. When I looked at the log after saving and executing I see that the caches for posters, logos, and other images are empty but EPG123 then retrieved them all again. It seems that you have it rigged that you can't turn off images retrieval even if you choose to do so.

Code: Select all

[8/24/2021 1:16:43 AM] Entering GetAllMoviePosters() for 918 movies.
[8/24/2021 1:16:44 AM] [ INFO] Found 0 cached/unavailable movie poster links.
[8/24/2021 1:16:48 AM] [ INFO] Successfully retrieved artwork info for 418 programs. (0:00:00:04.2472429 /     6.203 MB)
[8/24/2021 1:16:49 AM] [ INFO] Successfully retrieved artwork info for 500 programs. (0:00:00:05.5103151 /     7.953 MB)
[8/24/2021 1:16:50 AM] [ INFO] Processed 918 movie poster links.
[8/24/2021 1:16:50 AM] Exiting GetAllMoviePosters(). SUCCESS.
[8/24/2021 1:16:50 AM] Entering GetAllSeriesImages() for 2950 series.
[8/24/2021 1:16:50 AM] [ INFO] Found 0 cached/unavailable series image links.
[8/24/2021 1:16:54 AM] [ INFO] Successfully retrieved artwork info for 500 programs. (0:00:00:03.5322020 /     4.867 MB)
[8/24/2021 1:16:54 AM] [ INFO] Successfully retrieved artwork info for 500 programs. (0:00:00:03.8762217 /     5.110 MB)
[8/24/2021 1:16:54 AM] [ INFO] Successfully retrieved artwork info for 500 programs. (0:00:00:04.0612323 /     5.156 MB)
[8/24/2021 1:16:55 AM] [ INFO] Successfully retrieved artwork info for 500 programs. (0:00:00:04.3442485 /     4.781 MB)
[8/24/2021 1:16:58 AM] [ INFO] Successfully retrieved artwork info for 450 programs. (0:00:00:03.1591807 /     4.013 MB)
[8/24/2021 1:16:58 AM] [ INFO] Successfully retrieved artwork info for 500 programs. (0:00:00:03.6832107 /     4.925 MB)
[8/24/2021 1:17:00 AM] [ INFO] Processed 2950 series image links.
[8/24/2021 1:17:00 AM] Exiting GetAllSeriesImages(). SUCCESS.
[8/24/2021 1:17:00 AM] Entering GetAllSeasonImages() for 3468 seasons.
[8/24/2021 1:17:00 AM] [ INFO] Found 2522 cached/unavailable season image links.
[8/24/2021 1:17:04 AM] [ INFO] Successfully retrieved artwork info for 446 programs. (0:00:00:04.1602380 /     5.405 MB)
[8/24/2021 1:17:04 AM] [ INFO] Successfully retrieved artwork info for 500 programs. (0:00:00:04.4552548 /     6.073 MB)
[8/24/2021 1:17:05 AM] [ INFO] Processed 3468 season image links.
[8/24/2021 1:17:05 AM] Exiting GetAllSeasonImages(). SUCCESS.
[8/24/2021 1:17:05 AM] Entering GetAllSportsImages() for 0 sports events.
[8/24/2021 1:17:05 AM] [ INFO] Found 0 cached/unavailable sport event image links.
[8/24/2021 1:17:05 AM] [ INFO] Processed 0 sport event image links.
[8/24/2021 1:17:05 AM] Exiting GetAllSportsImages(). SUCCESS.

Since I rarely use the Guide, just to set up a recording schedule for a new show or series, I guess it will be alright if it is setup on an "as needed" basis as you say it will be.

[garyan2]

I actually don't know what to say. I don't know what options would lead you to believe it was including/excluding program images. If you look at the log, it says "retrieved artwork info", and "processed ... image links". No artwork/images are being downloaded.

I never thought of actually not including the links to artwork. It could provide a slightly smaller MXF file size, but could make a nice dent in the XMLTV file size... but, why? particularly for WMC? I'll have to think about that.

[StinkyImp]

Can we stay with what we have? EPG123 is working perfectly for me. As I get older I'm turning more, "if it ain't broke, don't fix it."

I will say the current versions are time-limited. In the future, SD will require tokens...

Probably a stoopid question but, I'm going to ask it anyway.

Way back in yesteryear I downloaded the portable version of EPG123 to test it out. Needless to say it's been the core of my setup ever since (Thank you so much!). I've never changed anything and have only downloaded the portable updates and overwritten the files in my "portable" directory. It's also only used on a standalone system.

Can I simply continue to update the portable version like I always have?

[garyan2]

I haven't decided on the portable package yet. Since I am adding the service, there are some registry entries to create and the service to add to the service controller. I may add a command script to do that for the portable version to be a one-time-and-done that would need to be run as administrator.

The thing is, if you run the setup program and point it to your existing portable install, then all of that will be taken care of and it will continue on as normal. It will not break your portable install and will become a real install.

With adding the service, a true portable version doesn't really make sense anymore.

[IT Troll]

Any program that uses the xmltv file will just need to use "http://<SERVERNAME>:<PORT>/output/epg123.xmltv" to import the file.
Thinking of other scenarios I need to account for.

Will the port be user-configurable? I suspect some users will be running other webservices on their machines; My Media Center, Remote Potato, Plex, etc.

[IT Troll]

Now that you have an HTTP server, maybe you can look in to simulating the Microsoft extended guide data server? This would act like a local proxy that takes the queries from WMC for extended guide data and then queries the needed data from an external site (not sure which sites would be best to provide this info).

I think ModernMedia UI+ uses a technique similar to this to serve additional metadata into Media Center; e.g. movie review ratings, sports and weather. I think it occasionally works for brief periods.

[garyan2]

Any program that uses the xmltv file will just need to use "http://<SERVERNAME>:<PORT>/output/epg123.xmltv" to import the file.
Thinking of other scenarios I need to account for.

Will the port be user-configurable? I suspect some users will be running other webservices on their machines; My Media Center, Remote Potato, Plex, etc.

I'm looking into how I would do that, and expected that desire from the group. Right now I have the TCP port at 9009 and a UDP port at 9010 for discovery. I would like to leave the UDP port alone, but changing the TCP port should be simple to implement a user configurable option.

[StinkyImp]

The thing is, if you run the setup program and point it to your existing portable install, then all of that will be taken care of and it will continue on as normal. It will not break your portable install and will become a real install.

With adding the service, a true portable version doesn't really make sense anymore.

Fair enough. Thank you, and thank you for your tireless work on this project!

[garyan2]

Will this fix the missing guide images from SD issue that some of us have been dealing with??

Learned something new yesterday. It appears that when WMC caches artwork, it caches the entire url string of the redirect to include the query options. Since the redirect we get from SD will supply a new authorization code for each request, WMC will ignore what was already downloaded and download the image again. Because of this, when you are no longer allowed access to the images from SD, there will be no images in WMC.

Also, because of this, I am going to make the cache server default to be enabled.

Good news is the server is light-weight and is running pretty solid. It handles refreshing your token when it detects an image being denied without issue... and is very responsive.

Code: Select all

[8/25/2021 5:03:40 PM] /image/aaee140d5bdc0e5c1314b159862ab862727d659ee268505f0bd3c6b577f09e71.jpg:
{"response":"UNKNOWN_USER","code":5004,"serverID":"20141201.web.1","message":"Download request from unknown account. Post message to http:\/\/forums.schedulesdirect.org\/viewforum.php?f=6","datetime":"2021-08-26T00:03:40Z"}
[8/25/2021 5:03:40 PM] Refreshed token upon receiving an UNKNOWN_USER (5004) error code.
[8/25/2021 5:03:40 PM] New token detected in registry. token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

The bad? news is since we are caching all images that have been requested at any time, it does take up some drive space. Rough estimate is 30,000 images will be 1GB. Keep in mind that in order to cache 30,000 images you would have to average 1,000 new images a day, e.g. not previously cached, for 30 days. The images will remain until 30 days (may change) after the last time it was requested from any client. I'm letting mine run to see how it grows over time before this cleanup action starts running.

There is one thing I would like to get resolved with the TCP code in order to release a socket as quickly as possible rather than closing and it and letting it die out naturally after 2 or 4 minutes. It's fine the way it is, but it just bothers me a little and I want to understand how to do it. Once I figure it out, or table it, I'll get this next version released.

[Space]

Will WMC always be communicating with your HTTP server, or will it also communicate directly to the SD server under certain circumstances?