Page 5 of 23

Re: Microsoft ending EPG for WMC in January

Posted: Thu Nov 14, 2019 3:21 pm
by garyan2
@DavidinCT @tonywagner:

There are 2 different methods of getting guide data using EPG Collector.

1) as proposed by StinkyImp is to use the data transmitted with the OTA station to provide a short amount of time... typically less than 24 hours. This method will require EPGC to access a tuner to capture that data repeatedly.

2) as proposed by ferrellsl is to use EPGC strictly as a means to convert an existing XMLTV file to the MXF format and import it. You will need to provide the XMLTV file and there are risks associated as I mentioned above.

Re: Microsoft ending EPG for WMC in January

Posted: Thu Nov 14, 2019 6:06 pm
by tonywagner
tonywagner wrote: Tue Nov 12, 2019 6:30 pm Here's that "xmltv-to-mxf" Java program

...

But when I ran it on Windows with zap2xml input, I got a new error: "Invalid byte 2 of 3-byte UTF-8 sequence". Then the resulting MXF file can't successfully import into WMC either. But running it on my Mac works just fine (and presumably Linux would too), so presumably this is Windows character encoding weirdness.
Okay, I fixed that error too! You need to include "-Dfile.encoding=UTF-8" in the Java command on Windows to make sure the MXF output file has valid encoding. (Apparently you don't need that option on Mac/UNIX but it shouldn't hurt to include it there either.)

Re: Microsoft ending EPG for WMC in January

Posted: Thu Nov 14, 2019 6:54 pm
by tonywagner
garyan2 wrote: Thu Nov 14, 2019 3:21 pm 2) as proposed by ferrellsl is to use EPGC strictly as a means to convert an existing XMLTV file to the MXF format and import it. You will need to provide the XMLTV file and there are risks associated as I mentioned above.
Interesting. How exactly do you do that? I just tried to import XML in EPGC (enabling WMC in the "Output" tab, and adding/loading the XMLTV file in the "Import" tab) -- but when I run "Collect EPG Data", after logging the channels and program titles, it crashes here:

Code: Select all

12:42:29:696 <S> Summary: Total Stations = 39 Total Gaps = 1188 Total Overlaps = 0 Total Time = 00:00:32.3906250
12:42:29:696 Deleting any existing version of output file
12:42:29:962 File delete exception: Could not find file 'C:\Users\User\AppData\Local\Geekzone\EPG Collector\TVGuide.mxf'.
12:42:29:962 Import name set to 'EPG Collector'
12:42:29:962 Creating output file: C:\Users\User\AppData\Local\Geekzone\EPG Collector\TVGuide.mxf
12:42:29:977 <E> ** The program has failed with an exception of type NullReferenceException
12:42:29:977 <E> ** Exception: Object reference not set to an instance of an object.
12:42:29:977 <E> ** Location:    at DomainObjects.OutputFileMXF.removeSpecialCategories(String category)
   at DomainObjects.OutputFileMXF.processCategory(XmlWriter xmlWriter, Collection`1 groups, String category)
   at DomainObjects.OutputFileMXF.processKeywords(XmlWriter xmlWriter)
   at DomainObjects.OutputFileMXF.Process()
   at DomainObjects.OutputFile.Process()
   at EPGCollector.Program.processTunerCollection()
   at EPGCollector.Program.runNormalCollection()
   at EPGCollector.Program.Main(String[] args)
12:42:29:977 <E> Exiting with code = 5
I've got a working method using the Java program I mentioned above, and can share that if it proves superior to EPGC. But I'd like to get EPGC working for comparison purposes!

Re: Microsoft ending EPG for WMC in January

Posted: Thu Nov 14, 2019 7:03 pm
by garyan2
I don't recommend using EPGC for reasons given earlier in this thread. It was a pain to get running, but I think the only way I got it to work was to install using the MSI file in the downloads, and not the update packages.

Re: Microsoft ending EPG for WMC in January

Posted: Thu Nov 14, 2019 9:41 pm
by tonywagner
garyan2 wrote: Thu Nov 14, 2019 7:03 pm I don't recommend using EPGC for reasons given earlier in this thread. It was a pain to get running, but I think the only way I got it to work was to install using the MSI file in the downloads, and not the update packages.
So did you ever get EPGC to import data into WMC? I've got it installed, and I can successfully scan the tuners and write the PSIP program data into an XML file -- but I haven't actually been able to import any data to WMC using EPGC. I always get an error writing the MXF file.

Could you perhaps share your EPGC ".ini" file?

Re: Microsoft ending EPG for WMC in January

Posted: Thu Nov 14, 2019 9:52 pm
by garyan2
I did get it to work, but I had to remove EPGC from my system and only install from the msi file.

I'll have to se if I still have the ini file. The program lasted long enough to do the compares and analysis. Again, I did the XMLTV and not the PSIP from tuner scans.

Re: Microsoft ending EPG for WMC in January

Posted: Thu Nov 14, 2019 10:29 pm
by tonywagner
garyan2 wrote: Thu Nov 14, 2019 9:52 pm I did get it to work, but I had to remove EPGC from my system and only install from the msi file.

I'll have to se if I still have the ini file. The program lasted long enough to do the compares and analysis. Again, I did the XMLTV and not the PSIP from tuner scans.
Interesting. I can do the PSIP tuner scans, and even write that to XML and import to WMC. But I can't import a zap2xml XML file into WMC using EPGC. I've tried both the regular install and the fixpak...

Oh well, not a big deal. Doesn't look like it's particularly promising anyway! I'll just share my other method.

Re: Microsoft ending EPG for WMC in January

Posted: Thu Nov 14, 2019 10:37 pm
by garyan2
I dont remember the switch, but try to run zap2xml with the switch to write Series category for non-movie programs.

Re: Microsoft ending EPG for WMC in January

Posted: Thu Nov 14, 2019 10:45 pm
by tonywagner
I just tried that switch ("-j") and it doesn't seem to have changed the result in EPGC.

Re: Microsoft ending EPG for WMC in January

Posted: Thu Nov 14, 2019 10:47 pm
by tonywagner
So here's how I got a free, non-Rovi guide right now using zap2xml and xmltv-to-mxf. First steps:
  1. Download and install epg123
  2. Run EPG123 Client, Client Setup, steps 1 & 2 (which will reset WMC -- do a backup first, if you like)
    While setting up WMC, be sure to enter 00000 as your zip code to get a blank guide -- and don't forget to manually add any VHF channels in WMC if necessary
  3. Create the folder C:\epg
  4. Download and install zap2xml, put the resulting "zap2xml.exe" file in C:\epg
  5. Download and install a Java JRE
  6. Download my "xmltv-to-mxf-0.0.2-SNAPSHOT.jar" file (which is just the GitHub version plus my period subchannel fix) and put it in C:\epg as well
    https://file.io/ndLsGO
Once that's done, you can use a batch file for the update process. Save the following as "update.bat" in C:\epg (substituting your own Zap2It username and password, of course!):

Code: Select all

cd C:\epg
zap2xml -u username -p password
java -Dfile.encoding=UTF-8 -jar xmltv-to-mxf-0.0.2-SNAPSHOT.jar xmltv.xml
C:\Windows\ehome\loadmxf -i mxf.xml
%SystemRoot%\ehome\ehPrivJob.exe /DoReindexSearchRoot
Finally, the first time you set this up, you will also have to associate the WMC channels with their guide listings. I just had to click "Match by # Number" in the EPG123 Client and it was done. But future updates can just run the batch file again, unless your channel lineup changes.

Still missing categories in WMC, so I might look into improving the "xmltv-to-mxf" program further.

Also, My Channel Logos seems to behave strangely after this -- but I wasn't really using the logos before, so it's no big deal for me.

Re: Microsoft ending EPG for WMC in January

Posted: Fri Nov 15, 2019 6:07 pm
by Green Lantern
tonywagner wrote: Thu Nov 14, 2019 10:47 pm So here's how I got a free, non-Rovi guide right now using zap2xml and xmltv-to-mxf. First steps:
  1. Download and install epg123
  2. Run EPG123 Client, Client Setup, steps 1 & 2 (which will reset WMC -- do a backup first, if you like)
    While setting up WMC, be sure to enter 00000 as your zip code to get a blank guide -- and don't forget to manually add any VHF channels in WMC if necessary
If I wait a couple weeks past the January cutoff (so all the prior Rovi data is gone) can I avoid all the extra time and hassle of the reset/setup? If so, how? Thank you!

Re: Microsoft ending EPG for WMC in January

Posted: Fri Nov 15, 2019 6:47 pm
by DavidinCT
tonywagner wrote: Thu Nov 14, 2019 10:47 pm So here's how I got a free, non-Rovi guide right now using zap2xml and xmltv-to-mxf. First steps:
  1. Download and install epg123
  2. Run EPG123 Client, Client Setup, steps 1 & 2 (which will reset WMC -- do a backup first, if you like)
    While setting up WMC, be sure to enter 00000 as your zip code to get a blank guide -- and don't forget to manually add any VHF channels in WMC if necessary
  3. Create the folder C:\epg
  4. Download and install zap2xml, put the resulting "zap2xml.exe" file in C:\epg
  5. Download and install a Java JRE
  6. Download my "xmltv-to-mxf-0.0.2-SNAPSHOT.jar" file (which is just the GitHub version plus my period subchannel fix) and put it in C:\epg as well
    https://file.io/ndLsGO
Tony,

Thanks for this. As I am not sure, when you import via loadmxf, does it delete the XML as it imports ? Or does it give a status of "complete" ?

Just looking for possible logging or a smart batch that will re-run it if the import fails. Maybe even a pop up in WMC, Guide data update/download failed, type thing...

You know... "It does not work" type posts...

Re: Microsoft ending EPG for WMC in January

Posted: Fri Nov 15, 2019 6:54 pm
by garyan2
The xmltv file stays. You could actually use EPG123 client, with a scheduled task, to import the mxf file with logging.

I should be ready with ZAP2MXF or whatever it will be called this weekend. The batch script will be very similar to above. You should add the pvr index task at the end as well. Of course, using epg123 client will take care of that as well.

Tony, I get file not found errors with your link. Can't check it out.

Re: Microsoft ending EPG for WMC in January

Posted: Fri Nov 15, 2019 11:03 pm
by tonywagner
garyan2 wrote: Fri Nov 15, 2019 6:54 pmTony, I get file not found errors with your link. Can't check it out.
Sorry, apparently that site expires the link as soon as it is downloaded! Here's another take on that "xmltv-to-mxf" JAR file, if you want to try it out:

https://uploadfiles.io/vnnm8q4i

Re: Microsoft ending EPG for WMC in January

Posted: Sat Nov 16, 2019 12:26 am
by tonywagner
Green Lantern wrote: Fri Nov 15, 2019 6:07 pm If I wait a couple weeks past the January cutoff (so all the prior Rovi data is gone) can I avoid all the extra time and hassle of the reset/setup? If so, how? Thank you!
Actually, the reset stuff is probably unnecessary. I was just doing that for testing -- I wanted to be sure I could set up WMC from scratch beyond the January shutdown.

If you don't do a reset, you can just turn off automatic downloads within WMC, then load the new MXF file, and finally use EPG123 Client to select "MainLineup" and then map the existing channels to the new listings (again, Map by # Number pretty much took care of that with 1 click for me!).

The usual warnings about making a backup / restore point, I'm not liable for anything, yadda yadda yadda :) .

Re: Microsoft ending EPG for WMC in January

Posted: Sat Nov 16, 2019 1:02 am
by garyan2
Okay, couple pointers right off-hand... may have more later.

1. You are duplicating programs unnecessarily. Using xmltv-to-mxf, for my XMLTV file it created 11,372 programs but there were only 7,090 unique programs. It doesn't harm anything, just makes the file inefficient and bloated.
2. Programs that start with "SH" are being give a unique GUID number (which contributes to #1). I understand the reason, but the same can be accomplished by just setting the 'isGeneric' flag of the program.
3. Episode number is off by 1 which I thought you fixed? Season number is not present.
4. Channel UIDs in the Lineups are risky with just the channel number (!Channel!MainLineup!4_1). It makes an assumption that there will be no duplicate channel numbers in the XMLTV file.
5. Speaking of Lineup, you should add a name value to the lineup to be distinguishable and unique to xmltv-to-mxf.
6. The Service id doesn't have to be so complex; "s1", "s2", etc. is just fine. It is the UID that has to be unique and constant between loads.
7. The program original air dates just need to be yyyy-MM-dd... don't need the T00:00:00:000Z.
8. Programs that are movies should use the "MV" id given in the data; OAD should be empty; year should be populated; the isMovie flag should be set.

The real hard part is the categories. It took me quite a while to figure how to do it best, but can't help to think it can always be better. I don't think it will ever be perfect.

Re: Microsoft ending EPG for WMC in January

Posted: Sat Nov 16, 2019 1:12 am
by garyan2
Green Lantern wrote: Fri Nov 15, 2019 6:07 pmIf I wait a couple weeks past the January cutoff (so all the prior Rovi data is gone) can I avoid all the extra time and hassle of the reset/setup? If so, how? Thank you!
OTA or cable? If cable, I would say go through the hassle. If OTA, you can probably do without.

Re: Microsoft ending EPG for WMC in January

Posted: Sat Nov 16, 2019 2:36 am
by tonywagner
garyan2 wrote: Sat Nov 16, 2019 1:02 am Okay, couple pointers right off-hand... may have more later.
Thanks for the tips! I had identified the season/episode and original air date problems, but hadn't actually fixed them yet. My only fix so far was supporting period subchannels (without that, it wouldn't work at all on the zap2xml file).

I'm just OTA, and a fairly light TV user in general, so as long as series update and record properly, I'm satisfied, and this makeshift system does that, at least.

Re: Microsoft ending EPG for WMC in January

Posted: Sat Nov 16, 2019 2:38 am
by tonywagner
DavidinCT wrote: Fri Nov 15, 2019 6:47 pm Tony,

Thanks for this. As I am not sure, when you import via loadmxf, does it delete the XML as it imports ? Or does it give a status of "complete" ?

Just looking for possible logging or a smart batch that will re-run it if the import fails. Maybe even a pop up in WMC, Guide data update/download failed, type thing...

You know... "It does not work" type posts...
Nothing gets deleted in that workflow.

There is output from each command, I think -- you could try running each line manually on a command prompt and you can see it. I don't have any experience with logging and such on a Windows command prompt, but in theory you should be able to check the output of each command for success/error before proceeding? Just be sure you're not doing unnecessary extra Zap2It scraping.

An easy visual check could be simply to see how many days of guide data are loaded in WMC (although it doesn't update the "About Guide Listings" thing like the Rovi updates...).

Re: Microsoft ending EPG for WMC in January

Posted: Sat Nov 16, 2019 6:18 pm
by StinkyImp
tonywagner wrote: Sat Nov 16, 2019 2:36 amI'm just OTA, and a fairly light TV user in general, so as long as series update and record properly, I'm satisfied...
My 2 cents...

I'm OTA and just want to record TV shows. I ran EPG Collector on an air-gapped Win7 test machine for close to two months. After the initial setup it automatically extracted program information from the PSIP, imported it into WMC, and did it flawlessly. I had task scheduler run it daily at around 2:00AM.

The primary downside was (as garyan2 mentioned) the length of programming data available in a given 24 hour period. The other downsides were guide features and episode details. You get a bland time grid with station identifiers, program name, and a basic description.

If a person had no internet access, didn't care about guide aesthetics, ran it (through task scheduler) a few times a day, and just wanted to be able to schedule a single program or and entire series, EPG Collector works just fine.