Full metadata for mkvs? How does epg123 fit into this?

An evolving, supported alternative to Rovi
Forum rules
★ Download the latest EPG123 here: https://garyan2.github.io/ <> Setup guide here: https://garyan2.github.io/install.html
jachin99

Posts: 1293
Joined: Wed Feb 24, 2016 3:36 pm
Location:

HTPC Specs: Show details

Full metadata for mkvs? How does epg123 fit into this?

#1

Post by jachin99 » Wed Jan 16, 2019 7:28 pm

Im experimenting with getting consistent metadata across multiple plugins, for both wtv and move files using epg123 and mmui+, and I have a successful test library where I achieve this. My question is, Does epg123 either on it's own or via mmui+ support purposely give me this ability. I'm not talking about just rotten tomato scores. I get full cast and crew, similar movies etc for my test mkv file.

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#2

Post by garyan2 » Fri Jan 18, 2019 4:41 am

EPG123 will provide the title, year, description, cast & crew with character names as provided by Gracenote.

Since EPG123 restored the year to movies in WMC, and if you have the "Retrieve CD album art, media information for DVDs and movies, and Internet Services from the Internet" enabled, you will get the Full Plot, Similar Movies, and Reviews from AMG (Rovi). The download from AMG may also override the cover art and cast & crew when viewing.

MMUI+ uses the title/year to grab the Rotten Tomatoes and Flixter scores.

DVDs are the same as the movies using the DVDID to look up the movie, coverart, ....

Basically, for your MKV file, if it has the year released, it will look up the data from AMG to provide all the data.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

jachin99

Posts: 1293
Joined: Wed Feb 24, 2016 3:36 pm
Location:

HTPC Specs: Show details

#3

Post by jachin99 » Fri Jan 18, 2019 5:39 pm

Gary, thanks for the explanation, and I had completely forgotten I had that box checked on my machine. I have made quite a few observations about how WMC treats a Wtv file Vs MKVs, and how the retrieve additional data setting effects that. For my tests I separated my MKVs, from my Wtvs via folder structure so that I could easily exclude each file type from my library. I then installed Ceton MMC on my test machine so that it would generate dvdid.xml, and folder.jpg files on both my remote share, and local file system. The first few observations cover Wtv files, and were made without the retrieve additional information turned on, and without the creation of any dvdid.xml files. MMUI+ has been turned off, and all guide data for my Wtv movies has been generated by EPG123. None of my Wtv file or folder names include the year, and the title of the Wtv file is simply the title of the movie as generated by WMC itself configured through the EPG123 client interface. My MKV naming convention includes the title of the movie, and sometimes the year. I was changing my library and observing its behavior as I typed all of this.

1. For starters, WMC reads metadata differently for WTV files depending on which library you include them in. If Wtv movies are included in the Movie Library then it would seem all embedded metadata is ignored but I can get cover art for Wtv files included in the movie library by including folder.jpg in the movie's folder.
2. If Wtv files are included in the Recorded TV library, or in the Recorded TV folder then I get full metadata. If the embedded cover art has somehow been lost through either file conversion or file archiving then folder.jpg files are ignored with this configuration. I have one or two movies like this where the file was only moved but it somehow lost the embedded cover art, and placing a folder.jpg image in the movie's folder doesn't fix the problem of no cover art for that movie.

Those observations have been made with the retrieve additional info setting turned off so I'll go ahead and turn additional info back on to see if I get cover art and I do but only for a few files.

With Wtv movies in the recorded tv library, and the additional info setting turned on, I get full metadata for my Wtv files but I only get coverart for a few files, and this is where the problem comes in. I'm getting missing cover art via AMG but that data is unreliable, and will likely worsen as Win 7 starts to die. To get back on subject Its time to move those Wtv files to the movie library and see what happens. With my Wtv files in the movie library, and retrieve additional info turned on WMC can't read the embedded data so I lose that, AND WMC doesn't reliably connect to the AMG database so I lose mostly everything minus the inclusion of folder.jpg to show coverart.

To sum things up for Wtv...
-No metadata but coverart with the inclusion of folder.jpg when I include Wtv files in the movie library
-Metadata but no coverart with the inclusion of folder.jpg when I include Wtv files in the recorded tv library
-Turning on the option to retreive additional data does not adequately make up for the deficiencies in either configuration.

On to MKVs, which are simpler.

1. Obviously there is not embedded metadata for WMC to read in these files as there is with a Wtv file. This means metadata lookup has be achieved either via YAMMM or the retrieve additional info setting assuming the year is included in the Movie's Folder's name.
2. Gary you said the year is needed in order for WMC to recognize the movie, and retrieve additional info, and for MMUI+. I'll leave that mostly alone with the caveat that YAMMM DVDIDs might somehow provide the year but then the suggested configuration for YAMMM is to include the year so back to leaving that alone I suppose.
3. With MMC installed on my local machine the movie library seems to slow down, and I get random error messages about the Ceton Otay application at times.
4. The only known method to get metadata for MKV files is via the generation of DVDID xml files BUT generating "Original" or "Correct" DVDID xml files will only give you metadata for movies up to sometime around 2016 or 2017. YAMMM generated "Fake" DVDIDs that utilize YAMMM's TMDB lookup give you metadata for current movies. I tested this on Ready Player One and it worked for me.

A few more general observations...
-Library navgation with MMC installs slows down greatly. After I remove that program but leave the existing file caches alone, navigation seems to speed back up, and I keep the metadta for MKVs on that local machine.
-If I move my Wtv movie recordings that don't show a cover when they are stored on a remote share to the local file system and include them in my recorded TV library, I MIGHT get cover art to show for those files that are missing it. I'm basing this statement off of my memory but the rest of this post I tested as I typed.
-Even with the year included in a Movie's folder name, I could not get any additional AMG data to show for a few test MKVs on a PC which didn't have some form of YAMMM installed. This seems to point towards there needing to be a DVDID file on the local file system in order for AMG to perform its lookup.

Factoring in completeness of metadata, aesthetics, and speed of navigation it would seem to me that the best combination is...
-Install YAMM to let it generate remote and local DVD IDs to get coverart and at least a synopsis for MKVs, then uninstall YAMMM to speed up movie library navigation for Mkvs.
-For Wtv files, it seems I should store these either in a remote or local location, and include them in my recorded TV library.

But then when I get a new MKV, how will I get good metadata for that file.... Either way, I had pull all of this info from threads posted over the last couple of years, and responses I have received from Gary, and W10MC. I'm nowhere near done with all of this but at least now I have consolidated list of behavior observations about the movie library. Now I'm going to dig in to how YAMMM works, and how that differs from programs that generate "Real" DVDID xmls.

EDIT. Renaming my movies to include the year in both the file and folder name greatly increased metadata retrieval and UI speed - even with MMC installed. I'll test some more as I get time and confirm all of my above observations but if this last paragraph holds true over time then I'll leave yammm or MMC installed

jachin99

Posts: 1293
Joined: Wed Feb 24, 2016 3:36 pm
Location:

HTPC Specs: Show details

#4

Post by jachin99 » Thu Jan 24, 2019 7:35 pm

After comparing YAMMM generated DVDIDs with the Retrieve additional info setting turned on with actual DVDIDs downloaded from this site: http://www.dvdid.info I decided to use YAMMM for the time being but I'll have to donwload some DVDIDs for movies that YAMMM doesnt pick up. I tried manually creating a few YAMMM DVDIDs but creating YAMM"s JSON file is too much work for me to recommend to others. Although there have been a few sites that let users download DVDIDs, dvdid.info is the last site that I can find that is still up. I did a whois query for dvdid.info to see if it will be expiring soon, and the site owner just renewed his registration for a year in early January. Regardless of all of that, I am still able to find some current movies but there were quite a few missing. I don't know when the owner of DVDID.Info will take the site down, and the real DVDIDs still depend on the additional info setting.

None of this tells me when the extended info lookup will break so I did a whois lookup for api.rovicorp.com. It looks like the rovi domain is set to expire on february 23rd of this year so if WMC uses something like api.rovi.com to look up additional info then this will be breaking very soon but we will still get synopsis and actions, which don't depend on the additional lookup setting for both YAMMM and Real DVDIDs.

I forgot to mention that I looked at the real DVDIDs stored in the local cache, and they have chapter markers but I haven't tested these AND because I have renamed my movies to include the year YAMMM names its dvdIds with the release year at the end, while the DVDIDs I'm getting from dvdid.info just names them after movie so maybe YAMMM won't overwrite the files I'm downloading. The cover art for real DVDIDs retrieved by AMG is still terrible.

jachin99

Posts: 1293
Joined: Wed Feb 24, 2016 3:36 pm
Location:

HTPC Specs: Show details

#5

Post by jachin99 » Sun Jan 27, 2019 1:49 am

https://mspoweruser.com/microsoft-turni ... ia-player/

This is worth a read for all of us. Gary you might have a bunch of new business soon.

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#6

Post by garyan2 » Sun Jan 27, 2019 2:46 am

jachin99 wrote: Sun Jan 27, 2019 1:49 am https://mspoweruser.com/microsoft-turni ... ia-player/

This is worth a read for all of us. Gary you might have a bunch of new business soon.
Wow, you may be right. From the sounds of the notification, the guide for WMC from MS/Rovi could be shut off at any moment.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

Space

Posts: 2838
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#7

Post by Space » Sun Jan 27, 2019 5:43 am

It sounds like they are just shutting down the metadata server, not the listings (although you never know with these low-information news stories as they do mention "TV guide for movies" whatever that means). If that is the case, I don't believe it will affect WMC users still using Rovi/TiVo data very much, as it already doesn't look up movie data since none of the recorded movies have the year of the movie's release.

I guess it will affect things such as ripped DVDs and metadata that is pulled due to .dvdid files.

Although the mention of it still working on Win8 is a little disturbing, like they won't be shutting down services, just blocking Win7, which is something they could also do with the guide data eventually (prior to Win8's end of life).

jachin99

Posts: 1293
Joined: Wed Feb 24, 2016 3:36 pm
Location:

HTPC Specs: Show details

#8

Post by jachin99 » Sun Jan 27, 2019 6:40 am

If you look at the link to the MS page all wmc users are effected even on win 8 in terms of guide data. MS received recently pulled the live TV features from the android Xbox app. If the rovi domain and API is expiring I doubt MS has any control over that. Wmp will still likely read tag data so 3rd party programs can hopefully provide workarounds as long as they write wmp compatible tags. It just sucks for us that extended Info lookup might break. We'll see what happens I guess.

Space

Posts: 2838
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#9

Post by Space » Sun Jan 27, 2019 7:08 am

I looked on that MS page and it does say that all versions of WMC will be affected. But I don't believe this has anything to do with guide data, just the extended metadata, which I believe is primarily used for movies (but also some TV show stuff, like if you select an actor on the "Cast & Crew" page of the TV show it will show you movies that actor is in, but will only show TV shows they are in if they are currently in the TV listings, so I don't think that function requires the extended metadata).

It is the metadata for WMP that will still work on Win8 and Win10, but not work on Win7. I am not familiar with how metadata queries are done for WMP on Win7 vs the other versions, but if they use the same server/protocol, then Microsoft is just blocking Win7 rather than shutting down Win7 specific servers. If they are just blocking Win7, than that is setting a precedent for them to do the same thing for WMC guide data in the future (since WMC on Win7 and Win8 is basically the same and there are not separate servers for each).

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#10

Post by garyan2 » Sun Jan 27, 2019 3:24 pm

I believe you are right. The phrase "and TV guide for movies in Windows Media Center" threw me off a bit... poorly written statement regardless of which meaning is intended. For the guide, it looks like what is to be lost is the extended data for "Full Plot", "Similar Movies", "Movies", and "Cast & Crew". The area in WMC that will lose real functionality is the Movie Library with user files and DVDs.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#11

Post by garyan2 » Sun Jan 27, 2019 3:46 pm

Basically, they effectively disable this option.
autodownload.png
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

Space

Posts: 2838
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#12

Post by Space » Sun Jan 27, 2019 4:04 pm

garyan2 wrote: Sun Jan 27, 2019 3:24 pm I believe you are right. The phrase "and TV guide for movies in Windows Media Center" threw me off a bit... poorly written statement regardless of which meaning is intended. For the guide, it looks like what is to be lost is the extended data for "Full Plot", "Similar Movies", "Movies", and "Cast & Crew". The area in WMC that will lose real functionality is the Movie Library with user files and DVDs.
Ironically, because the Rovi/TiVo feed omits the year of release for movies, WMC instances using this data feed cannot properly do a lookup for "Full Plot", etc.

So this metadata shutdown actually affect EPG123 users more-so than Rovi/TiVo users, as EPG123 properly populates the movie year of release, so the metadata query works properly. Rovi/TiVo data users are already not getting the full benefit of the extended metadata, so EPG123 users will now also be missing this functionality.

I wonder if it is possible to emulate the metadata server so that the functionality can continue. This can either be done by setting up a server on the Internet that duplicates the MS server that is going away, or another way to do it is to have a server that runs on the WMC machine itself (or other machine on the local LAN), and translates the query from WMC in to queries to external movie databases (such as TMDb, etc.) and returns the data in the proper format.

As long as the query is not encrypted or authenticated in some way, it should be possible to intercept this query by changing the IP address of the domain name for the servers that are going away (such as by editing the hosts file or perhaps a WMC db/registry entry).

What needs to be done before this shutdown happens is to monitor/capture these queries on the network so that the proper formats can be reverse engineered and duplicated.
Last edited by Space on Sun Jan 27, 2019 5:43 pm, edited 1 time in total.

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#13

Post by garyan2 » Sun Jan 27, 2019 4:26 pm

It may be simpler than that. We do know that this extended information, after download, is stored in the internet cache somehow. We might just need to duplicate that method of saving the file so that when WMC checks for extended information, it will already be there.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

jachin99

Posts: 1293
Joined: Wed Feb 24, 2016 3:36 pm
Location:

HTPC Specs: Show details

#14

Post by jachin99 » Sun Jan 27, 2019 5:48 pm

I'm posting this here just in case it is useful. Below is a sample dvdid that was stored in my DVD info cache when I downloaded a real dvd.id file from dvd.info. I grabbed a dvdid.xml file, and stored it in this movie's folder, then Windows downloaded this to my cache. The big difference I see between YAMMM and DVDID is the chapter markers but I'm sure there are others.

Code: Select all

<?xml version="1.0"?>
<METADATA xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <MDR-DVD>
    <MetadataExpires>2019-04-24T15:14:27.3566137-04:00</MetadataExpires>
    <version>5.0</version>
    <dvdTitle>Harry Potter and the Goblet of Fire</dvdTitle>
    <studio>Warner Home Video</studio>
    <leadPerformer>Daniel Radcliffe; Rupert Grint; Emma Watson; Robbie Coltrane; Ralph Fiennes</leadPerformer>
    <director>Mike Newell</director>
    <MPAARating>PG13</MPAARating>
    <language />
    <releaseDate>2005 01 01</releaseDate>
    <genre>Fantasy</genre>
    <largeCoverParams>cov150/dru700/u720/u72008ro8an.jpg</largeCoverParams>
    <smallCoverParams>cov075/dru700/u720/u72008ro8an.jpg</smallCoverParams>
    <dataProvider>AMG</dataProvider>
    <duration>157</duration>
    <title>
      <titleNum>1</titleNum>
      <titleTitle>Harry Potter and the Goblet of Fire</titleTitle>
      <studio>Warner Home Video</studio>
      <director>Mike Newell</director>
      <leadPerformer>Daniel Radcliffe; Rupert Grint; Emma Watson; Robbie Coltrane; Ralph Fiennes</leadPerformer>
      <MPAARating>PG13</MPAARating>
      <genre>Fantasy</genre>
      <synopsis />
      <chapter>
        <chapterNum>1</chapterNum>
        <chapterTitle>The Riddle House [3:45]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>2</chapterNum>
        <chapterTitle>The Portkey [2:22]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>3</chapterNum>
        <chapterTitle>The Quidditch World Cup [3:41]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>4</chapterNum>
        <chapterTitle>The Dark Mark [3:55]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>5</chapterNum>
        <chapterTitle>The Triwizard Tournament [5:29]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>6</chapterNum>
        <chapterTitle>The Goblet of Fire [3:54]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>7</chapterNum>
        <chapterTitle>The Unforgivable Curses [5:50]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>8</chapterNum>
        <chapterTitle>The Four Champions [7:24]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>9</chapterNum>
        <chapterTitle>Let Events Unfold [2:40]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>10</chapterNum>
        <chapterTitle>Rita Skeeter [2:27]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>11</chapterNum>
        <chapterTitle>Sirius Conversation [4:03]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>12</chapterNum>
        <chapterTitle>The Hungarian Horntail [3:31]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>13</chapterNum>
        <chapterTitle>Transforming Malfoy [4:58]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>14</chapterNum>
        <chapterTitle>The First Task [8:40]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>15</chapterNum>
        <chapterTitle>Best Foot Foward [6:25]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>16</chapterNum>
        <chapterTitle>An Unexpected Challenge [5:30]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>17</chapterNum>
        <chapterTitle>The Yule Ball [3:24]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>18</chapterNum>
        <chapterTitle>The Egg's Glue [4:19]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>19</chapterNum>
        <chapterTitle>The Second Task [8:06]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>20</chapterNum>
        <chapterTitle>Never Whole Again [8:53]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>21</chapterNum>
        <chapterTitle>The Pensieve [3:13]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>22</chapterNum>
        <chapterTitle>Dreams and Suspicions [4:41]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>23</chapterNum>
        <chapterTitle>The Thrid Task [2:55]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>24</chapterNum>
        <chapterTitle>Flesh, Blood, and Bone [8:34]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>25</chapterNum>
        <chapterTitle>The Death Eaters [3:07]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>26</chapterNum>
        <chapterTitle>Priori Incantatem [4:34]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>27</chapterNum>
        <chapterTitle>Veritaserum [3:24]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>28</chapterNum>
        <chapterTitle>Parting Ways [7:35]</chapterTitle>
      </chapter>
      <chapter>
        <chapterNum>29</chapterNum>
        <chapterTitle>End Credits [6:10]</chapterTitle>
      </chapter>
    </title>
  </MDR-DVD>
  <NeedsAttribution>true</NeedsAttribution>
  <DvdId>2f21a6b3|31089638</DvdId>
</METADATA>

Space

Posts: 2838
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#15

Post by Space » Mon Jan 28, 2019 7:13 am

garyan2 wrote: Sun Jan 27, 2019 4:26 pm It may be simpler than that. We do know that this extended information, after download, is stored in the internet cache somehow. We might just need to duplicate that method of saving the file so that when WMC checks for extended information, it will already be there.
Simpler? Maybe, but that would mean you would have to download the metadata for ALL movies on the system, even if you only look at the metadata for a few of them. The existing system (and the proxy implementation I described) seems to only get the extended metadata when you request it for any particular movie (on-demand).
Last edited by Space on Mon Jan 28, 2019 9:34 pm, edited 1 time in total.

jachin99

Posts: 1293
Joined: Wed Feb 24, 2016 3:36 pm
Location:

HTPC Specs: Show details

#16

Post by jachin99 » Mon Jan 28, 2019 6:38 pm

I noticed a delay in the old system. On top of that it would take 3 or 4 tries to find the data which is part of what started this entire post. I'm not sure which would be faster but I also noticed a slight delay in MMUIs lookup as well which makes me think a local file could be a better solution. Yammm with the additional info setting turned off seems to bring up the description instantly so I'm basing some of my view on that as well as other factors.

Kick4U

Posts: 43
Joined: Mon Mar 05, 2012 7:36 am
Location:

HTPC Specs: Show details

#17

Post by Kick4U » Mon Jan 28, 2019 8:08 pm

I just noticed that the 'similar movies' tab in WMC is not populating. I'm wondering if this is due to the metadata issue discussed or something on my computer.

Is it still working for all of you?

jachin99

Posts: 1293
Joined: Wed Feb 24, 2016 3:36 pm
Location:

HTPC Specs: Show details

#18

Post by jachin99 » Mon Jan 28, 2019 8:46 pm

It looks dead to me on my test PC. I'll check the others at some point but it feels pointless until a new solution is offered if that happens.

Kick4U

Posts: 43
Joined: Mon Mar 05, 2012 7:36 am
Location:

HTPC Specs: Show details

#19

Post by Kick4U » Thu Jan 31, 2019 9:19 pm

Seems it still is working. My system somehow had the automatic download option unchecked!
Image

I really hope a solution to this is found, I love searching finding movies by digging around what other movies certain actors were in or using the 'similar movies' tab.

jachin99

Posts: 1293
Joined: Wed Feb 24, 2016 3:36 pm
Location:

HTPC Specs: Show details

#20

Post by jachin99 » Thu Mar 21, 2019 1:25 pm

Mine is still working also. I checked the WHOIS info for rovicorp.com today, and according to that the domain expired in February but if I go to their api console, or website its still up. Rovicorp.com looks to be down. http://developer.rovicorp.com/io-docs rovicorp.com

Post Reply