Question concerning v1.7.0.x

[Space]

...
For versions 1.7.0.0/1, the images downloaded and cached by WMC never expire (this is the way it has always been even pre-epg123). The images in the epg123 cache will expire 30 days after the last request from a client. I'm working on changing that now. The images provided by SD have a 30-day lifespan from when it was created, so there is an expiry date for each image. I'm going to cascade that into the epg123 cache as well as the image WMC downloads in the internet cache. As long as WMC respects the Expires field, when the image expires on the SD side, it will expire in WMC and the epg123 cache forcing a refresh when it is requested again.

Does WMC support conditional requests? Such as "If-None-Match" or "If-Modified-Since"? Does the SD server support them for images?

These can be used to further limit resource usage so that even if the image cache has expired, it will not have to re-download an image if it has not changed since the last time you downloaded it.

Does WMC not even use a conditional request to potentially update a previously cached image? It would be strange if it never updates a cached image.

[SwissCheese]

Hi!

I've been following EPG123s recent developments, but
haven't tried the new version yet, and have some questions...


1) How will image caching work for recordings we want to keep forever?

Most of my year-old recordings, and a few recent ones, have
already lost their cover art, presumably because their image
files are no longer available on some server.


2) Will there be a way we can save and backup
image files for recorded shows locally forever?


3) What if EPG123s cached images expire but can't be replaced because
they are no longer at SD or elsewhere, will they still get deleted
automatically anyway?


4) Also, do the image links in the mxf really have to be URLs, can't
they just refer to local file names?


5) Will running the EPG123 service prevent my PC from sleeping?
(I have only 1 PC and only use it in Full mode)


Thanks!

[garyan2]

Does WMC support conditional requests? Such as "If-None-Match" or "If-Modified-Since"? Does the SD server support them for images?

These can be used to further limit resource usage so that even if the image cache has expired, it will not have to re-download an image if it has not changed since the last time you downloaded it.

Does WMC not even use a conditional request to potentially update a previously cached image? It would be strange if it never updates a cached image.

"If-None-Match" and "If-Modified-Since" won't do any good even if supported. The image is automatically deleted on the amazon server at 30-days. When you request an image from SD, it will check to see if the image exists on the amazon server. If it does not, it will grab it from Gracenote, save it on the amazon server with the 30-day lifespan, and then provide the redirect to where to get the image. (Note this is why sometimes it can take 1-2 seconds to get the redirected image). The new image's Last-Modified date will be when the image was added to the amazon server even if it is the same (or equivalent) image that used to reside there before. I assume the ETag will be new as well.

Not sure what WMC does for images in the cache. I'm not sure it did anything other than accept whatever was there.

Note the word "equivalent" above. Gracenote has 2 different APIs for images... one is more stable and provides different sized images but can be stale. The other is pretty dynamic, but will only provide the very large sized images that would not be appropriate for WMC. If the large images is what SD uses, then SD will create new images for the different sizes by scaling the originals. So the resulting image may be identical to the last image that was on the server, but it went through the entire process of scaling and creating again and is by definition "new".

[garyan2]

1) How will image caching work for recordings we want to keep forever?

It won't. I am assuming you are talking about movie posters... the thumbs used for programs/recordings are embedded in the WTV file. So the poster art will disappear if any of the following happens:
1. You clear the internet cache (Internet Explorer) and SD no longer has it on the amazon server.
2. You have a WMC database recovery.
3. You perform a WMC Database Rebuild from the client.
4. You perform a Client Setup again.

SD will only cache images on the amazon server if the program it is for is still alive in current guide listings somewhere.

2) Will there be a way we can save and backup
image files for recorded shows locally forever?

Not easily for specific images. You would have to find out what the filename is and then copy it out of the cache (c:\programdata\garyan2\epg123\images\). The way to find the image name is reading the mxf/xmltv file, as long as it is still part of your lineups' guide listings. If not part of your guide listings, then you will have to eyeball the images in the cache to find it out of possibly thousands... assuming it hasn't expired yet.

3) What if EPG123s cached images expire but can't be replaced because
they are no longer at SD or elsewhere, will they still get deleted
automatically anyway?

Yes. I don't think anyone here wants the image cache folder to continually grow to be multiple GBs.

4) Also, do the image links in the mxf really have to be URLs, can't
they just refer to local file names?

Using URLs is pretty much the only way to be able to work in all configurations. If a client downloads the mxf file, there are no local files to point to and it won't get any images at all.

5) Will running the EPG123 service prevent my PC from sleeping?

Nope. Been running one of my machines like this for weeks and it has never failed to go to sleep multiple times throughout the day.

[SwissCheese]

OK, thanks for the information.

It's too bad movie posters keep disappearing.

[garyan2]

There is a way that I've used in the past that I am trying to remember better. It involved using Media Center Master (MCM) to create a dvdinfo file for each movie that would be in its own folder. In that folder would be the poster image to reside forever. I can't remember any of the specifics so maybe someone else can help out with that.

[SwissCheese]

Hi!

I've read about the image per folder way of adding poster art for DVDs, but it's
rather painful when there are hundreds of recorded TV shows and movies.
Also, recorded movies don't have readily accessible image files to put in those folders.

If images are going to keep disappearing once their shows are no longer in the
current schedule it doesn't seem worth the effort to have images at all since
they won't persist more than a few days.

Will there be an option to skip installing the service for Full mode systems?
The service seems to be mostly useful when there are separate clients.

Thanks

[SwissCheese]

Hi again

Maybe I'm wrong, but it seems like the big changes in EPG123 are pretty close to
solving the longstanding problem of disappearing movie poster art.

If instead of deleting cached images after they've expired due to some time limit,
what if EPG123 never deleted images that still belong to any shows in the Recorded TV folder?

Wouldn't that allow movie poster art and series thumbs to persist until the movie
or TV show got deleted by the user?

Thanks

[garyan2]

The service will be necessary in the future to retrieve any images from SD. It needs to add the token in the request in order to be allowed access to the images.

I think you are confusing a couple things. In Full Mode, with no other clients, or even with clients, you can run the service with image cache turned off which is the default. This is only for the service. It does not prevent WMC from caching the images in the internet cache like it always has. The images in the internet cache are what WMC uses and are persistent, and will stay there until you clear the internet cache or it gets squeezed out during a cache cleanup by IE (or maybe even the OS). If for any reason your WMC database gets deleted, then the connection between the movie and the cached image for WMC is lost, so you will "lose" the poster in WMC then as well.

Again, the images we are discussing are the movie posters that are only visible in either the guide, the movie guide, or the movie library for recorded movies. Specifically we are talking about the movie library. Eventually, over time, the poster art for the recorded movies will disappear from any number of the reasons I've stated above. This has nothing to do with the service and/or having the service cache images or not. Before MS shutdown their service, they could get away with this because there was still the ability to reach out to AMG (Rovi) to get extended information and artwork for movies.

Program/Show/Series images are only visible in the guide and nowhere else. You're not losing any of these one way or the other.

[garyan2]

Hi again

Maybe I'm wrong, but it seems like the big changes in EPG123 are pretty close to
solving the longstanding problem of disappearing movie poster art.

If instead of deleting cached images after they've expired due to some time limit,
what if EPG123 never deleted images that still belong to any shows in the Recorded TV folder?

Wouldn't that allow movie poster art and series thumbs to persist until the movie
or TV show got deleted by the user?

Thanks

The service is not going to interact with any other program or file, so it will not know what is in the recorded folder, let alone if it is a movie or program or even what image it uses. The only images the service will delete are the ones in the epg123 image cache. It does not delete the images that WMC has already cached.

[IT Troll]

Wouldn't that allow movie poster art and series thumbs to persist until the movie
or TV show got deleted by the user?

One alternative is to run the RecordedTVHD add-in which does maintain a separate local cache of images for recorded movies and TV shows (the guide will still use SD/EPG123 images). However, this is another thing to install, introduces a new UI and does have some API challenges on the horizon.

[StinkyImp]

[Explains a LOT of stuff...]

Hehehe... You know what they say about true expertise? It's when a person can explain complex subject matter in terms a common person can understand. That's you!

Based on all the things you've explained, it's a no-brainer to disregard any reservations I had and upgrade to the v1.7 (and above) versions.

Thank you!

One alternative is to run the RecordedTVHD add-in... ...does have some API challenges on the horizon.

That was my first thought for SwissCheese's dilemma but we have those problematic TheTVDB changes coming up soon. What's the proposed path forward for RTVHD?

[garyan2]

Big improvement on next release. The service will handle/respect If-Modified-Since requests from WMC and any other requestor. Thanks @Space for bringing it up for me to look at further. This also allowed me to resolve the issue as best I can for recorded movie poster art and expiry dates. If WMC has the image cached and checks to see if it is still "fresh" but it is no longer on the SD servers, the service will respond with a 304 so that WMC will continue to use the cached image. I think that's the best I can do.

Also, the new version will kind of invalidate anything you have cached since I am changing from using a "last accessed" date/time to using the "expires" date/time. The improvements are definitely worth it though from my tests so far.

[garyan2]

Oh, fun fact... with WMC, since it never had expiry dates on any of the cached images, it would ping the host with If-Modified-Since for every session and for every image before displaying.

[IT Troll]

What's the proposed path forward for RTVHD?

There is a plan, but it is really early days. When there is some news it will appear in the RTVHD thread.

[Space]

Me: What about X?
Gary: No, I don't think I can do X.
...
Gary: New release with X is now available.
Me: ...

I'm wondering if SD can also support If-Modified-Since. I know you said that the actual image download is only available for 2 minutes, but they can still keep a last modified date for the main URL (that is redirected to the temporary image) and return 304 if it hasn't changed since the given date. Of course SD would have to have the ability to recognize when the image from Gracenote changed, but even if not changed, maybe they can at least keep track of whenever they refresh the image from Gracenote (assuming they keep a cached copy that isn't updated with every guide update from Gracenote).

[garyan2]

If I wasn't able to laugh at myself, I'd be offended.

SD does support If-Modified-Since (before any redirect)... just verified it a couple hours ago. There is going to be another change coming from SD such that if the image version changes from Gracenote, SD will update their image on the server but the outside link to the image will remain the same. The image will be updated with a new modified date of course. It avoids getting the image not found error when Gracenote/SD updates the image and the cached link in epg123 would be wrong. I didn't want to have to refresh all image links on every update and still get some image not found errors.

When the change occurs, everyone should clear their cache and build new again like we did earlier this year. Kinda sucks, but smooth sailing afterwards.

[Space]

Didn't mean to offend, quite the contrary. You take peoples suggestions and really look in to them, even if your initial impression is negative. That's a great attribute!

Thanks for being so receptive to suggestions! I'm sure there are many others that also appreciate it!

[SwissCheese]

Hi again

More about disappearing movie and TV series images....

I'm referring to movie and TV series images seen in:
WMC -> TV -> Recorded TV -> select a recording -> Synopsis

And movie cover art images seen in:
WMC -> Movies -> Movie Library


The problem is that most of the images for the 200+ movies and
series episodes I recorded with WMC over the past 1.5 years
disappeared in the last few weeks.

Every movie and TV episode used to have a cover-art image, but
now most images are gone. The few that remain are mostly for recent
recordings.


My understanding is:

1) After WMC is launched, it re-downloads images when
they are accessed in Synopsis or Movie Library by the user.

WMC seems to only cache image links but not whole images because:

(a) there is a delay loading an image when WMC is first opened, and
(b) if internet is disabled there are no images, just blank placeholders, and
(c) if images were local or in the wtv files they wouldn't be disappearing.

No internet = no images.

2) Synopsis and Movie Library images are related. When an image
disappears from Synopsis it also disappears from Movie Library.

3) WMCs only source of image information is from the SD and TMDB links
that epg123 inserted into the mxf, such as

imageUrl="https://json.schedulesdirect.org/...jpg
imageUrl="http://image.tmdb.org/...jpg

4) Something changed at SD recently because image links that
worked for over 1.5 years no longer work today.


Now that epg123 will be caching images, it seems plausible that
it solves the disappearing movie and series image problem.

Hopefully, as long as the image stays cached by epg123, it won't
matter that SD will no longer serve older images.

But, if epg123 automatically deletes cached images after a time limit
that un-solves the problem.


So, here are some suggestions:


1) If would be great if epg123 did not delete cached images that
are still in use by movies and series episodes in the Recorded TV
folder, then maybe images will no longer disappear.

Alternatively:

2) Add the option to disable automatic image deletion by epg123,
and allow users to delete files manually.

The image cache could get quite large, but I for one would be OK with it
taking up an 8TB SSD.

3) Add the option to prefer TMDB images over SD images.
Maybe TMDB images will last longer than SD.

SD images that last only a few days are not very useful for
a movie library.

Thank you.

[SwissCheese]

Another suggestion to add to my previous message

4) Allow users to protect specific image files in the cache so epg123 can't delete them.

Thanks again.