Lineup Selector (Auto import MXF lineup or switch lineups)

glugglug

Posts: 391
Joined: Thu Jun 09, 2011 1:34 am
Location:

HTPC Specs: Show details

Lineup Selector (Auto import MXF lineup or switch lineups)

#1

Post by glugglug » Fri Oct 02, 2015 2:29 am

I have created a tool to change the listings for all your channels to match a different lineup (optionally adding whatever channels are missing and/or removing channels that aren't in the new lineup). The intended purpose of this tool is to auto-add all the channels in the SchdulesDirect->Big Screen EPG lineup to use Zap2It instead of Rovi without manually setting hundreds of CableCARD channels. You could also use it to switch between lineups without redoing TV setup.

Only channels which have channel numbers in the lineup will be included. This means if you are using BSEPG, "Include Channel Numbers in Output", and for OTA ATSC "Include Subchannel Numbers in Output" MUST be selected. For ClearQAM or other tuner types, adding new channels will not work properly, but it should be able to update the listings of existing channels #s to use the ones from your new lineup. The app has only been tested with an InfiniTV CableCARD tuner. I haven't actually tried it with OTA/ClearQAM.

The current binary is Windows 7 x64 only. For other OSes, the source must be compiled on those OSes.

Binary: https://www.dropbox.com/s/8z8z0fpksqpwu ... y.zip?dl=0
Source: https://www.dropbox.com/s/vko7y127h80cj ... e.zip?dl=0

Basic usage (many steps, read this first):
0. Back up. Who knows in what ways we can screw up the WMC setup!
1. Optional: Turn off automatic downloads of guide data in WMC (But you will need to have TV setup completed, just not doing auto downloads). I'm not sure if some channels may revert to using Rovi if you leave the automatic downloads enabled. I don't think so, but just in case.
2. Set up SchedulesDirect account
3. download mc2xml.exe
4. Use mc2xml.exe to download xmltv.dtd, as described in Bob O Dell's post
5. download xmltv.dtd
6. Set up a scheduled task to repeat this download daily.
7. Install and configure BigScreen EPG to read your downloaded xmltv.dtd, and in particular, make sure the "Include Channel Numbers in Output" checkbox is selected in the advanced options->channel numbers section. I have only tried this with CableCARD, but made a recent modification to let you select a scanned lineup to pick tuners from which should allow use with ATSC OTA, maybe. For ATSC OTA to have a chance of working, you'll want to make sure the "Include Subchannel Numbers in Output" checkbox is checked as well. The LineupSelector app matches/creates channels by number, so the numbers need to be there in your .MXF file for it to work.
Image
8. Set up Big Screen EPG scheduled task, ideally set to run shortly after the mc2xml daily fetch, but with a long enough gap to be sure the download will complete first.
9. Hit the Process & Upload button in Big Screen EPG. Hit continue when needed until it completes (this will take awhile).
10. Launch the new LineupSelector app. UI Looks like this:

Image
11. Make sure all the options are set as you want. As far as I know, the merged lineup combo box at the top will only have one choice. The WMI Lineup box should automatically select the Big Screen EPG lineup by default, you can choose another if you want to try one of the other choices WMC gives you in your zipcode without redoing TV setup for example. The Scanned Lineup box should have one option for each tuner type found in your TV setup, and controls which tuners will be used for any new channels that are created for channels in the BSEPG lineup that weren't already in your guide.
12. Hit Sync Now to update your guide to use all the listings from your selected WMI lineup. This should only take a couple seconds.

As far as why I used BSEPG and my own guide updating tool rather than MCImportXMLTV, there are 2 things lacking in MCImportXMLTV, that I think would be dealbreakers for CableCARD users. If these two were fixed, I guess we would have the makings of a free app for use with SchedulesDirect:

1. The automatic channel mapping options pretty much completely fail and everything has to be assigned manually. This would be easy to fix! The name they show in the "EPG streams in XMLTV file" list start with the channel numbers that are perfect for matching with the guide! This could be solved by either adding an option to parse the number each channel in the XMLTV file starts with to get the channel number for matching or allow a regex to specify it.
2. It doesn't flag any channels as HD by default. BSEPG at least has an option to automatically flag channels with names ending in HD or HDTV. For local channels in the US with the ZapToIt data, this should ideally be able to match any channel ending in "DT" as well (or allow the user to supply matching regexes), but the DT suffix accounts for only about 15 HD channels out of about 200.

audinutt

Posts: 114
Joined: Thu May 14, 2015 2:43 am
Location:

HTPC Specs: Show details

#2

Post by audinutt » Fri Oct 02, 2015 8:56 pm

Awesome, thanks so much, where can I donate?

I definitely support this!
I will be testing this later in the next few days, sure would have saved me a few nights of work that is for sure!

-Richard

audinutt

Posts: 114
Joined: Thu May 14, 2015 2:43 am
Location:

HTPC Specs: Show details

#3

Post by audinutt » Fri Oct 02, 2015 9:25 pm

audinutt wrote:Awesome, thanks so much, where can I donate?

I definitely support this!
I will be testing this later in the next few days, sure would have saved me a few nights of work that is for sure!

-Richard
Okay, I got giddy and tested this.
One thing that would be nice... The channel # is added to the channel name with this method. Can it be removed somehow?
It causes issues with Mc Channel Logos. and also it looks kind of repetitive with the # listed twice.
Image

I can't thank you enough though this is a HUGE development to enable us to keep Zap2it and ditch the crappy Rovi guide.
:thumbup: :thumbup: :thumbup: :thumbup: :thumbup:

-Richard

glugglug

Posts: 391
Joined: Thu Jun 09, 2011 1:34 am
Location:

HTPC Specs: Show details

#4

Post by glugglug » Fri Oct 02, 2015 10:05 pm

I don't have the channel #s as part of the name like that in WMC, in fact I am also using My Channel Logos, but that is pretty much what the channel names looked like in MCImportXML for me unless I used a flag in mc2xml to swap the name and number ordering (in which case it looked more normal in MCImportXML, but I haven't tried to see how BSEPG handles that reversed ordering. In the XML file from mc2xml, the number looks like an alternate name for the channel, rather than having a distinct XML element name.)

I should be able to change the callsigns to get rid of any numeric prefixes followed by space during the sync.... But you have got me curious: Are you using MCImportXML instead of BSEPG to create and load the MXF? If so I think it might be easier to use the mc2xml flag that swaps the name and number order....
But it has also got me thinking that maybe having the numbers in the MXF callsigns like that from MC2XML can be useful, if it is set to build the file but not import it, it should be simple to create an extra process that transforms the MXF to use those numbers as channel numbers instead of part of the name, and auto sets HD flags before calling loadxmf.

audinutt

Posts: 114
Joined: Thu May 14, 2015 2:43 am
Location:

HTPC Specs: Show details

#5

Post by audinutt » Fri Oct 02, 2015 10:16 pm

audinutt wrote:Awesome, thanks so much, where can I donate?

I definitely support this!
I will be testing this later in the next few days, sure would have saved me a few nights of work that is for sure!

-Richard
Okay, I got giddy and tested this.
Works beautifully. My channel logos doesn't autopopulate logs, but this isn't very difficult to update, time consuming, yes but I don't mind this much.

Wonderful piece of software. Thanks for taking the time to develop this!

-Richard

audinutt

Posts: 114
Joined: Thu May 14, 2015 2:43 am
Location:

HTPC Specs: Show details

#6

Post by audinutt » Fri Oct 02, 2015 10:19 pm

glugglug wrote:I don't have the channel #s as part of the name like that in WMC, in fact I am also using My Channel Logos, but that is pretty much what the channel names looked like in MCImportXML for me unless I used a flag in mc2xml to swap the name and number ordering (in which case it looked more normal in MCImportXML, but I haven't tried to see how BSEPG handles that reversed ordering. In the XML file from mc2xml, the number looks like an alternate name for the channel, rather than having a distinct XML element name.)

I should be able to change the callsigns to get rid of any numeric prefixes followed by space during the sync.... But you have got me curious: Are you using MCImportXML instead of BSEPG to create and load the MXF? If so I think it might be easier to use the mc2xml flag that swaps the name and number order....
But it has also got me thinking that maybe having the numbers in the MXF callsigns like that from MC2XML can be useful, if it is set to build the file but not import it, it should be simple to create an extra process that transforms the MXF to use those numbers as channel numbers instead of part of the name, and auto sets HD flags before calling loadxmf.
I am NOT using MCIMPORTXML, I am using BSEPG and started with a fresh configuration.
Curious what your entire configuration looks like?
I am using xmltv to grab the xml info from schedules direct, are you using a different program?

many thanks!
-Richard

glugglug

Posts: 391
Joined: Thu Jun 09, 2011 1:34 am
Location:

HTPC Specs: Show details

#7

Post by glugglug » Sat Oct 03, 2015 2:01 am

Using mc2xml to download from Schedules Direct (http://mc2xml.awardspace.info/). Running it with a -d 360 flag to attempt to download 15 days of guide data (it only gives me 13-14). If you run it with a -F flag it reverses the order of the name and number in the XML, which seems to be what MCImportXML wants, but not sure what BSEPG would do with that.

Another setting that may be of interest (I've saw you mentioned changing it in the other thread?) is that I have the "Don't use channel MatchNames in output" button selected, as I am matching by number, not by name/callsign.

glugglug

Posts: 391
Joined: Thu Jun 09, 2011 1:34 am
Location:

HTPC Specs: Show details

#8

Post by glugglug » Sat Oct 03, 2015 3:11 am

Win8 x64 binary built: https://www.dropbox.com/s/2z2vn8qxqyl1s ... 4.zip?dl=0

Win8.1 version coming soon (after the 156 Win8 updates finish and I create another VM)

audinutt

Posts: 114
Joined: Thu May 14, 2015 2:43 am
Location:

HTPC Specs: Show details

#9

Post by audinutt » Sat Oct 03, 2015 8:37 am

glugglug wrote:Using mc2xml to download from Schedules Direct (http://mc2xml.awardspace.info/). Running it with a -d 360 flag to attempt to download 15 days of guide data (it only gives me 13-14). If you run it with a -F flag it reverses the order of the name and number in the XML, which seems to be what MCImportXML wants, but not sure what BSEPG would do with that.

Another setting that may be of interest (I've saw you mentioned changing it in the other thread?) is that I have the "Don't use channel MatchNames in output" button selected, as I am matching by number, not by name/callsign.
mc2xml does work better with your application.
The channel #'s are not duplicated now.

Thanks for the tip.
I was able to rip everything out of media center, start from scratch and have a full guide with channel logos in under 30 minutes :)

I will be doing some further testing but this is about to go in production in a couple more weeks (this means on the wife's media center; left her on rovi while i developed a solution that I won't have to mess with)
Thanks again for your hard work your application is exactly what was needed for this to be streamlined.

-Richard

webminster

Posts: 657
Joined: Tue Dec 20, 2011 11:05 pm
Location:

HTPC Specs: Show details

#10

Post by webminster » Sun Oct 04, 2015 4:17 am

For the LineupSelector app.. don't see mention of setting up a scheduled task for this. Is this a one-time run after you've done the initial BSEPG setup and sync, or is this app run in a non-GUI mode after the BSEPG task? Or how often afterward need it run?
-Alan

glugglug

Posts: 391
Joined: Thu Jun 09, 2011 1:34 am
Location:

HTPC Specs: Show details

#11

Post by glugglug » Sun Oct 04, 2015 6:28 am

webminster wrote:For the LineupSelector app.. don't see mention of setting up a scheduled task for this. Is this a one-time run after you've done the initial BSEPG setup and sync, or is this app run in a non-GUI mode after the BSEPG task? Or how often afterward need it run?
It doesn't need to be run daily like the mc2xml and BSEPG tasks. But when channels are added/moved around/removed by your cable company, that won't be reflected without re-running it, so you may want to rerun it every few weeks or so.

I think I found a better solution and should have another app out soon that lets you permanently set up the lineups associated with each tuner, so any new channels in the daily imported MXFs will get picked up without rerunning it. (see next post :-) )
Last edited by glugglug on Sun Oct 04, 2015 6:29 am, edited 1 time in total.

glugglug

Posts: 391
Joined: Thu Jun 09, 2011 1:34 am
Location:

HTPC Specs: Show details

#12

Post by glugglug » Sun Oct 04, 2015 6:29 am

Win8.1 binary built: https://www.dropbox.com/s/v2vivfi9j3ks8 ... 4.zip?dl=0

I think I've found a better way to go about this......
Before writing the bulk channel modification app, I had tried to change the lineups associated with the merged lineup (aka the channel list shown in the guide), so that it would automatically pick up changes to those lineups without needing to resync to pick up new channels, but failed.

Then I remembered that during TV setup it asks you which lineup to associate *with each type of tuner*. Looking through the tuner device properties, I found where the association between the tuner and a lineup is made, and successfully updated it, so that changes to the channel list (not just the scheduled shows) in the MXF are automatically picked up. As an added benefit, if done this way, leaving the normal guide downloads enabled won't cause any channels to revert, and the lineup is actually assigned per tuner (not per tuner type), so you could do something like associating some tuners to the Rovi guide with others on the Zap2It guide.

Should have an app to do this soon. Will still need to use BSEPG rather than MCImportXML so that the channel numbers end up there, for now.

glugglug

Posts: 391
Joined: Thu Jun 09, 2011 1:34 am
Location:

HTPC Specs: Show details

#13

Post by glugglug » Mon Oct 05, 2015 2:51 am

The alternative app I'm working on that I thought would be "better" isn't really working out. It changes the lineup listed in the "About Guide Listings" screen to BSEPG, which keeps the mcupdate downloads from changing the channel list, but it doesn't automatically rebuild the channel list to match this new assignment, and my attempts to trigger WMC to do an automatic rebuild of the channel list instead of bulk channel editing have mostly resulted in it just deleting all the Rovi channels and ignoring the BSEPG ones, then complaining that the guide is empty and triggering TV setup again. I managed to get it to trigger the auto-calculated channel list to match the BSEPG lineup once, but haven't been able to reproduce that since.

So for now, the current LineupSelector is it...

grittree

Posts: 65
Joined: Tue Mar 18, 2014 3:09 pm
Location:

HTPC Specs: Show details

#14

Post by grittree » Wed Oct 07, 2015 8:57 pm

glugglug wrote:I have created a tool to change the listings for all your channels to match a different lineup

4. Use mc2xml.exe to download xmltv.dtd, as described in Bob O Dell's post
Could you give a link to the O Dell post?

jbf154

Posts: 55
Joined: Mon Jan 14, 2013 7:41 pm
Location:

HTPC Specs: Show details

#15

Post by jbf154 » Thu Oct 08, 2015 5:55 pm

The sample responses from SchedulesDirect (http://forums.schedulesdirect.org/viewt ... ?f=8&t=568) don't seem to have any of the values for the "shortDescription" field in the MXF format. This gets shown in the guide, but when you click more info (to bring up the synopsis) you'll get a slightly longer description.

Also, are you getting review information for movies (the full text review)? What about artwork for shows & movies? Is the artwork accurate or hit/miss like the default Rovi data?

I could work on writing a utility to take the XMLTV format and converting it to the MXF if it loads easily into MC and there isn't any missing data from SD.

glugglug

Posts: 391
Joined: Thu Jun 09, 2011 1:34 am
Location:

HTPC Specs: Show details

#16

Post by glugglug » Mon Oct 12, 2015 10:08 pm

I am in the early stages of making a JSON grabber. I had started with XML parsing, but switched when I realized that the XMLTV spec leaves way too much open, and you get very different XML depending not only on what grabber you use but even depending on settings you might not expect with the same grabber, and that they were all missing many things described in the json docs.

So far I am just pulling the lineup and channel list with the json, haven't gotten around to pulling the schedule yet. Tried to download lineups i dont have tied to my account, in order to see what the channel lists look like for qam and dvb, and fogured out it doesn't let me do that without adding them to my account. Which has to also be done with the API - other country and QAM lineups don't show up on the website.

Hoping the download limits aren't near as tight as the docs say - it says it limits you to 5000 programs per request and 200 requests per 2 hours, or you get blocked for 24 hours. With the # of channels in CableCARD lineups I'd estimate that means I can pull my guide data 3 times in 2 hours before hitting the limit, so caching the program MD5s and only downloading changes may be critical.

If you tell mc2xml to use the JSON API instead of the legacy API, it generates an XML file about double the size because of more descriptions - but it doesn't include any station logo URLs.

According to the API docs, the program IDs use the format EP<seriesID><episodenum> when the particular episode is known, and SH<seriesID>0000 when it is not. Requesting the program info with the SH... programID is supposed to return a more generic description.

glugglug

Posts: 391
Joined: Thu Jun 09, 2011 1:34 am
Location:

HTPC Specs: Show details

#17

Post by glugglug » Tue Oct 13, 2015 3:19 am

jbf154 wrote:The sample responses from SchedulesDirect (http://forums.schedulesdirect.org/viewt ... ?f=8&t=568) don't seem to have any of the values for the "shortDescription" field in the MXF format. This gets shown in the guide, but when you click more info (to bring up the synopsis) you'll get a slightly longer description.

Also, are you getting review information for movies (the full text review)? What about artwork for shows & movies? Is the artwork accurate or hit/miss like the default Rovi data?

I could work on writing a utility to take the XMLTV format and converting it to the MXF if it loads easily into MC and there isn't any missing data from SD.
More answers to this:
I decided that I've been spending too much time on the UI for each piece, and am going through making sure I can do all the API calls first.
I downloaded the schedules and programs for SyFy HD and Comedy Central HD as a test tonite, JSON response is here, you can see the descriptions there:https://www.dropbox.com/s/maqjp7sbmgynp ... s.zip?dl=0 There are generally 2 descriptions per show, one with a 100 character limit, and one with a 1000 character limit (there can actually be an array of descriptions in each length category).

There is a separate section in the docs on how to get the artwork for each programID in schedulesdirect. The URLs are not returned in the same call as the program info - you have to make separate requests for them. https://github.com/SchedulesDirect/JSON ... ing-images I haven't tried to retrieve the artwork yet. Apparently it is supposed to return a whole bunch of artwork for each program, and we will have to heuristically select an image of an appropriate size. Since only the first 10 characters of the programID are used, this is per-series, not per episode.

audinutt

Posts: 114
Joined: Thu May 14, 2015 2:43 am
Location:

HTPC Specs: Show details

#18

Post by audinutt » Tue Oct 13, 2015 6:07 pm

glugglug wrote:
jbf154 wrote:The sample responses from SchedulesDirect (http://forums.schedulesdirect.org/viewt ... ?f=8&t=568) don't seem to have any of the values for the "shortDescription" field in the MXF format. This gets shown in the guide, but when you click more info (to bring up the synopsis) you'll get a slightly longer description.

Also, are you getting review information for movies (the full text review)? What about artwork for shows & movies? Is the artwork accurate or hit/miss like the default Rovi data?

I could work on writing a utility to take the XMLTV format and converting it to the MXF if it loads easily into MC and there isn't any missing data from SD.
More answers to this:
I decided that I've been spending too much time on the UI for each piece, and am going through making sure I can do all the API calls first.
I downloaded the schedules and programs for SyFy HD and Comedy Central HD as a test tonite, JSON response is here, you can see the descriptions there:https://www.dropbox.com/s/maqjp7sbmgynp ... s.zip?dl=0 There are generally 2 descriptions per show, one with a 100 character limit, and one with a 1000 character limit (there can actually be an array of descriptions in each length category).

There is a separate section in the docs on how to get the artwork for each programID in schedulesdirect. The URLs are not returned in the same call as the program info - you have to make separate requests for them. https://github.com/SchedulesDirect/JSON ... ing-images I haven't tried to retrieve the artwork yet. Apparently it is supposed to return a whole bunch of artwork for each program, and we will have to heuristically select an image of an appropriate size. Since only the first 10 characters of the programID are used, this is per-series, not per episode.
Amazing progress thanks for all of your efforts so far. Long live wmc Image

User avatar
STC

Posts: 6808
Joined: Mon Jun 06, 2011 4:58 pm
Location:

HTPC Specs: Show details

#19

Post by STC » Tue Oct 13, 2015 7:40 pm

Great work both of you. Look forward to the fruits.
By the Community, for the Community. 100% Commercial Free.

Want decent guide data back? Check out EPG123

jziggity

Posts: 146
Joined: Tue Dec 04, 2012 2:11 am
Location:

HTPC Specs: Show details

#20

Post by jziggity » Mon Oct 19, 2015 11:13 pm

So, I'm hoping glugglug's program can help me out with a guide "situation" of sorts. I am helping a family member with their Media Center setup, Windows 7, OTA tuners. They reside in Wisconsin smack in the middle of two larger metropolitan areas, being Green Bay and Milwaukee. Using the appropriate zipcode for the area they live in, Media Center automatically pulls in a mostly correct channel list. Some channels need to be manually added for the tuners to tune correctly. A rooftop dual antenna system ensures pretty solid reception of both markets, so no problems there.

All Milwaukee area channels are correct in the guide. All Green Bay area channels show incorrect data. It appears Media Center is ONLY using the Milwaukee area OTA lineup data. It shows the Green Bay area channels, however the guide shows incorrect channel callsigns and thusly incorrect program listings. Apparently it is not able to map two large metro area OTA lineups on its own. If I try to use the "Edit Listings" for the channels with incorrect data, the appropriate Green Bay OTA sources are not a choice.

So how do I get Media Center to accept a Green Bay area OTA lineup as a possible choice for those channels? Right now Media Center is trying its best to match up the Green Bay channels to something close in the Milwaukee lineup, and it's (obviously) not correct. It seems like this program should be able to help with this issue, but I don't want to just start experimenting lest it gets screwed up more than it is.

I'm not interested in going the whole BigScreen EPG route, as guide data quality is solid for the Milwaukee area channels. I would imagine there is a process to manually accomplish this as well, but I'm not sure what that would all involve. I did manually change to a Green Bay area zipcode through the TV Signal Setup process, and the results were as expected: all Green Bay area channels showed correct callsigns and guide data, but all Milwaukee stations were now incorrect. I did make a note of the guide ID # when it was switched to a Green Bay area zipcode, as I believe it will probably be needed at some point.

Any assistance with this project would be greatly appreciated. I feel like I'm on the right track here, I just need some more guidance. Thanks!

Post Reply