OAD for SH type shows

[Space]

Any progress on the research about adding OAD for SH type shows (as opposed to EP episodes)?

From my analysis of a sample MXF file, it looks like you can add OAD (if it is available from SD) to any SH type show where isSeries is not true, perhaps leaving it off of those marked with isPaidProgramming. The only ones in the sample MXF that I have found that are not isSeries are: isPaidProgramming, isSpecial, and isSports.

So by adding OAD to any SH show that does not have isSeries=true (and excluding those that have isPaidProgramming=true) it should cover the ones that have isSpecial and/or isSports set to true (and anything else that my sample MXF file did not have an example of).

[Sammy2]

Any progress on the research about adding OAD for SH type shows (as opposed to EP episodes)?

From my analysis of a sample MXF file, it looks like you can add OAD (if it is available from SD) to any SH type show where isSeries is not true, perhaps leaving it off of those marked with isPaidProgramming. The only ones in the sample MXF that I have found that are not isSeries are: isPaidProgramming, isSpecial, and isSports.

So by adding OAD to any SH show that does not have isSeries=true (and excluding those that have isPaidProgramming=true) it should cover the ones that have isSpecial and/or isSports set to true (and anything else that my sample MXF file did not have an example of).

Excuse the ignorance but what is an SH type show?

[Space]

These are terms used by Gracenote and appear in the tmsId of a program:

http://developer.tmsapi.com/docs/read/data_v1_1

[Sammy2]

These are terms used by Gracenote:

http://developer.tmsapi.com/docs/read/data_v1_1

Thanks!

So Gracenote provides data to Zap2it?

[Space]

...

Thanks!

So Gracenote provides data to Zap2it?

Yes, Gracenote and zap2it are owned by the same parent company (Tribune Media).

[KenL]

...From my analysis of a sample MXF file, it looks like you can add OAD (if it is available from SD) to any SH type show where isSeries is not true, perhaps leaving it off of those marked with isPaidProgramming. The only ones in the sample MXF that I have found that are not isSeries are: isPaidProgramming, isSpecial, and isSports.

Can you think of a downside to populating the OAD for any SH for which it's available?

Looks like many (if not most) SH besides PaidProgramming have OAD available. Even some generic listings for isSeries seem have an OAD in the feed (earlier than the eventual ESI OAD) which I presume represents the start of the series. Since these "generic" listings should never survive to record time (thanks to EPG-123!) populating (series) OAD for the generic listing shouldn't cause trouble? Nor should any (available) OAD cause recording algorithm trouble for the case of SH not isSeries.

[Space]

With Rovi data, you can tell if an airing has ESI if the episode title is populated (ESI=Episode Specific Info, and in our case I guess we can say that EP = has ESI, SH = does not have ESI (at least for isSeries)). If a show has ESI, but that particular episode does not have an episode title, then Rovi makes the episode title = show name. This is how you can tell if an episode has ESI or not. Has an episode name? Then it has ESI. This is the one change that the switch-over to Rovi made that I actually like.

With epg123, how can you tell the difference between an episode without ESI and an episode with ESI but no episode title (and just uses a generic series description)?

If epg123 follows the lead of Rovi by adding a title to all airings that have ESI (or implements some other way of knowing it is an EP and not an SH), then I would think adding the OAD to all SH shows would be fine.

Here is an example. The first one is a EP with ESI, the next two are SH without ESI, one way you can tell that the first one has ESI is because it has an OAD. The other two do not. If you add OAD to those, then it is harder to tell if they have ESI or not. The description for the EP is very generic.
If an episode has an episode number or season/episode, I guess that can tell you it has ESI, but I don't think they all do.

It is important to know if an episode has ESI because you know it is set to record because it is an episode that you haven't recorded before, otherwise it could be old episodes that are only recording because they don't have ESI.

Code: Select all

      <Program id="18795" uid="!Program!EP01332055_0513" title="Family Feud" des
cription="Teams compete to match answers of nationwide surveys." episodeNumber="
513" originalAirdate="2013-02-12" keywords="k6,k6004" series="si688" isSeries="t
rue">

      <Program id="18808" uid="!Program!SH01332055_0000" title="Family Feud" des
cription="Steve Harvey assumes the hosting duties for this daytime game show in
which five members of one family are pitted against five members of another fami
ly. Each team's goal is to guess the results of audience survey questions. A pla
yer from each team initially faces off, answering the question to earn the chanc
e to either pass or play. The winning family gets a chance to earn extra cash in
 a bonus round." shortDescription="Teams compete to match answers of nationwide
surveys." keywords="k6,k6004" series="si688" isSeries="true" isGeneric="true">

      <Program id="18809" uid="!Program!SH00522855_0000" title="Family Feud" des
cription="Five members of a family are pitted against another family. The goal i
s to guess the results of audience survey questions. The initial face off includ
es a player from each family, answering the question to earn the chance to eithe
r pass or play. The winning family gets to play the bonus round for a cash prize
." shortDescription="Teams compete to match answers of nationwide surveys." keyw
ords="k6,k6004" series="si3202" isSeries="true" isGeneric="true">

[KenL]

...With epg123, how can you tell the difference between an episode without ESI and an episode with ESI but no episode title (and just uses a generic series description)?

When using SD (json) the ESI listing is EP and the generic is SH. Not sure about ESI w/o title but I assume it would be EP.

For example: The Late Late Show With James Corden "season":1,"episode":170

Code: Select all

{"programID":"EP019952930172","titles":[{"title120":"The Late Late Show With James Corden"}],"eventDetails":null,"descriptions":{"description100":null,"description1000":[{"descriptionLanguage":"en","description":"Actor Colin Farrell; comic Wanda Sykes; animal handler Jack Hanna; Demi Lovato performs."}]},"originalAirDate":"2016-03-16","genres":["Talk","Comedy"],"officialURL":null,"keyWords":null,"episodeTitle150":"Colin Farrell; Wanda Sykes; Jack Hanna; Demi Lovato","metadata":[{"Gracenote":{"season":1,"episode":170}}],"contentRating":[{"body":"USA Parental Rating","code":"TV14"}],"contentAdvisory":null,"movie":null,"cast":[{"billingOrder":"01","role":"Host","nameId":"241456","personId":"237929","name":"James Corden","characterName":null},{"billingOrder":"02","role":"Guest","nameId":"203085","personId":"199695","name":"Colin Farrell","characterName":null},{"billingOrder":"03","role":"Guest","nameId":"228779","personId":"225924","name":"Wanda Sykes","characterName":null},{"billingOrder":"04","role":"Guest","nameId":"67385","personId":"67385","name":"Jack Hanna","characterName":null},{"billingOrder":"05","role":"Musical Guest","nameId":"535932","personId":"518781","name":"Demi Lovato","characterName":null}],"crew":[{"billingOrder":"01","role":"Executive Producer","nameId":"744028","personId":"713680","name":"Ben Winston","characterName":null},{"billingOrder":"02","role":"Executive Producer","nameId":"882935","personId":"840877","name":"Rob Crabbe","characterName":null},{"billingOrder":"03","role":"Executive Producer","nameId":"569758","personId":"546894","name":"Mike Gibbons","characterName":null}],"entityType":"Episode","showType":"Series","recommendations":null,"hasImageArtwork":true,"md5":"26BxVyyXlZYDTUgqo7UeQQ"}

The Late Late Show With James Corden (generic)

Code: Select all

{"programID":"SH019952930000","titles":[{"title120":"The Late Late Show With James Corden"}],"eventDetails":null,"descriptions":{"description100":[{"descriptionLanguage":"en","description":"The English actor interviews guests and hosts musical performances."}],"description1000":[{"descriptionLanguage":"en","description":"Broadway, television and film star James Corden takes over the reins of the late-late-night franchise at CBS from fellow U.K. import Craig Ferguson. The British performer -- whose previous hosting gigs include five years of The Brit Awards -- puts his charm, warmth and creative instincts to use as he interviews celebrities and newsmakers, and hosts musical performances in his post-\"Late Show\" time slot."}]},"originalAirDate":"2015-03-23","genres":["Talk","Comedy"],"officialURL":null,"keyWords":{"Mood":["Amusing","Charming","Cheery"],"Time Period":["2010s"],"Theme":["Adventure","Discovery","Pursuit"],"Character":["Host","Celebrity","Actor","Musical guest","Audience","Narrator"],"Setting":["Studio","Stage","Street","Office","Backstage","Dressing room"],"Subject":["Celebrities","Current events","Music","Movies","Entertainment","News"],"General":null},"episodeTitle150":null,"metadata":null,"contentRating":[{"body":"USA Parental Rating","code":"TV14"}],"contentAdvisory":null,"movie":null,"cast":[{"billingOrder":"01","role":"Host","nameId":"241456","personId":"237929","name":"James Corden","characterName":null}],"crew":[{"billingOrder":"01","role":"Executive Producer","nameId":"744028","personId":"713680","name":"Ben Winston","characterName":null},{"billingOrder":"02","role":"Executive Producer","nameId":"882935","personId":"840877","name":"Rob Crabbe","characterName":null},{"billingOrder":"03","role":"Executive Producer","nameId":"569758","personId":"546894","name":"Mike Gibbons","characterName":null}],"entityType":"Show","showType":"Series","recommendations":[{"programID":"SH000768380000","title120":"Late Show With David Letterman"},{"programID":"SH013323920000","title120":"Conan"},{"programID":"SH017211070000","title120":"The Tonight Show Starring Jimmy Fallon"}],"hasImageArtwork":true,"md5":"eBy6/OcONuJOM2y2plAXEw"}

Notice the different OAD.

Edit: confusing comparing rovi mxf to EPG-123 json cache entries. I guess what I'm trying to say is if the json OAD is present for the generic SH, I can't think of a problem in using it. The generic listing gets replaced before recording date (with EPG-123) and the uid are different so I don't think it should blow up the db?

Added: That isn't rovi mxf in your example since it's using EP and SH? At any rate OAD isn't vital for generic listings really just part of the rich data available. But it is important for the non episodic SH.

[KenL]

Since we were spun off from "bug report" to "feature request" here is the issue addressed in the original thread:

Missing OAD on a number of Smithsonian Channel programs loaded with EPG123 (last few versions?) One example: "The True Story of the Mary Celeste"

Code: Select all

<Program id="35592" uid="!Program!SH00033862_0000" title="The True Story of the Mary Celeste" description="A newly discovered transcript of the vessel's final five days informs about the mystery of the 135-year-old "ghost ship."" shortDescription="A transcript informs about the 135-year-old "ghost ship."" keywords="k7,k7009,k7" series="si5193" isSpecial="true" isGeneric="true" />

Looks to me like it's missing from the mxf?

Same channel "The Real Story Pirates of the Caribbean" with normal OAD as expected.

Code: Select all

< Program id="35591" uid="!Program!EP01149155_0011" title="The Real Story" episodeTitle="s03e01 Pirates of the Caribbean" description="Investigating the original Caribbean pirates that sailed the corridors of the Atlantic Ocean." seasonNumber="3" episodeNumber="1" originalAirdate="2011-05-15" keywords="k6,k6037,k6035" series="si5178" isSeries="true" />

Checked Zap2it.com seems all Smithsonian programs (including "The True Story of the Mary Celeste") have OAD. Did a quick check of my native DISH guide (also Gracenote) looks like all Smithsonian Channel programs for the next 7 days have the expected OAD. If it's present in JSON, then it must be EPG123 missing it under certain circumstances?

If the uid has a "SH" before the number (which is taken directly from the Gracenote data), it is treated differently than those that start with "EP", so perhaps EPG123 is not including OAD for "SH" type shows?
Generally I think "SH" is used when there is no episode specific info for a show (in which case the OAD may mean the day the entire series started), but it may also be used for shows that don't really have any episodes like specials (which this seems to be) and such, in which case I guess the OAD should be included in the MXF (perhaps if it is not marked as a series, you should include the OAD, if specified?).

That is correct. If it is not specific episodic information (starting with SH) then I ignore the OAD. I'll take a look on handling specials in this case ... get back to you.

And since I dug it out originally to check for the OAD here is the cache entry for the first example:

Code: Select all

{"programID":"SH000338620000","titles":[{"title120":"The True Story of the Mary Celeste"}],"eventDetails":null,"descriptions":{"description100":[{"descriptionLanguage":"en","description":"A transcript informs about the 135-year-old \"ghost ship.\""}],"description1000":[{"descriptionLanguage":"en","description":"A newly discovered transcript of the vessel's final five days informs about the mystery of the 135-year-old \"ghost ship.\""}]},"originalAirDate":"2007-11-04","genres":["Documentary","Special"],"officialURL":null,"keyWords":null,"episodeTitle150":null,"metadata":null,"contentRating":[{"body":"USA Parental Rating","code":"TVPG"}],"contentAdvisory":null,"movie":null,"cast":null,"crew":null,"entityType":"Show","showType":"Special","recommendations":null,"hasImageArtwork":true,"md5":"whUbdnOLwyQq8TA3wm3vyQ"}

I've noticed the discarded OAD is an issue on various other channels also, basically any case of a non-episodic documentary. At least for such cases hopefully there shouldn't be an issue to populate those.

[Space]

Yes, my examples are from an epg123.mxf file. Yes, ESI without episode title are EP, in fact that is the example I gave, that episode of Family Feud is an EP but has no episode title. The OAD on SH shows that are isSeries is usually the date that the series first premiered (s01e01).

I am just saying that with Rovi data in WMC, if you go to "view scheduled", you can easily tell which episodes have ESI and which don't (the ones with ESI show the episode title in quotation marks next to the show name, or if you highlight the show, at the bottom).

For instance right now I am skimming through the list on my WMC box and quickly see that "The Blacklist" airing on 3/24 @ 9pm ET has no ESI. I am pretty sure that this is going to be a repeat though and will probably be updated before airing, but if not, I can check out other sources and see if is new or a repeat as it gets closer to airtime. zap2it also shows that it has no ESI, so you can probably see it in epg123 systems as well.

If the show was a new episode, with ESI, but no real details yet (no episode title) it would show up as The Blacklist - "The Blacklist". This tells me two things, it is set to record because it is an episode I haven't recorded yet and the details of the episode (the title) have not yet been supplied, however this is the equivalent of an EP episode and therefore is a uniquely identified episode.

And while Gracenote data is better than Rovi, I still had generic episodes in the guide at air time occasionally (before the Rovi switch-over). To be honest, that hasn't really been much of a problem for me with Rovi data (at least for the shows I record) I very rarely have an episode recorded without ESI (the problem with Rovi is that it often gets the OAD wrong).

[KenL]

...Space, you've been rummaging around quite a bit in my mxf!

I use everything I can that I understand what they do. There are many flags still available but if there is no difference to the user experience I won't add it. Nosing through the mcepg and mcstore libraries, there is evidence of what I suspect to be heavy influence by zap2it. There are flags for series premiere, season premiere, categories like thriller, drama, action, .... The problem is I can populate all those fields based on the information I get from SD and they could do absolutely nothing for you.

Discovering the "isGeneric" flag was the final key, in my mind, to making epg123 viable. A little info for those paying attention:
Not all episodes will have information available from Gracenote initially, so all that will be provided is a generic description of the series and no episode specific information. When this is the case, the program id starts with "SH" as in SH012345670000. When the episode information becomes available, then the program becomes an "EP" as in EP012345670012 (the last four numbers are the production number of the series so you could read this as possibly episode 12). The problem is when the updated episode specific information is imported where the generic information already exists ... the generic information stays. We are not overwriting the generic information, but are actually trying to break the link between the schedule entry and the generic program to create a new link to the new program with a real episode ID. Using the "isGeneric" flag makes the link to the schedule entry a soft link which can be broken and replaced with the new information. Yay!

Right now, for all "SH" programs, epg123 treats them as generic and applies the flag. It won't stay this way, but I wanted to be sure the flag worked as I expected. There are quite a few people out there that have been using epg123 for more than 2 weeks, and even some that collect 3+ weeks worth of data at a time. I have not heard of anybody having generic information in the guide when there should have been episode specific information. I would call this a success. A downside? of using the flag is since it is considered generic, WMC ignores the OAD and won't even show it in the guide. This applies to some specials, sports, newscasts, .... I will be developing logic to a more targeted application of the flag rather than using a blanket rule. The low hanging fruit are specials and news.

Posting this here for continuity.

Nearly everything I'm seeing which would most benefit from keeping the OAD are "low hanging fruit" tagged "showType":"Special".

[KenL]

Code: Select all

<Program id="33334" uid="!Program!SH00033862_0000" title="The True Story of the Mary Celeste" description="A newly discovered transcript of the vessel's final five days informs about the mystery of the 135-year-old "ghost ship."" shortDescription="A transcript informs about the 135-year-old "ghost ship."" originalAirdate="2007-11-04" keywords="k7,k7005,k7" series="si4143" isSpecial="true" />

OAD for SH specials has landed in EPG123 v0.9.1! THANKS!

[STC]

Yes I noticed that. Great to have more info.

[garyan2]

Sneaked that one in there. In fact, I did it so long ago when working on v0.9.0 I completely forgot. By default, I now put any OAD in the file regardless of the show type. Also, for the isGeneric flag, I apply that to all SH type programs that are not identified as a series. Could backfire on some sports showings, but don't know at the moment.