Introducing EPG123

An evolving, supported alternative to Rovi
Forum rules
★ Download the latest EPG123 here: http://epg123.garyan2.net <> Setup guide here: http://epg123.garyan2.net/downloads/epg123_Guide.pdf
User avatar
garyan2

Posts: 5917
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

Re: Introducing EPG123

#1801

Post by garyan2 » Fri Mar 20, 2020 2:09 am

glorp wrote:
Thu Mar 19, 2020 11:55 pm
I'm glad to see that the cache will finally be a single file. That should substantially improve the disk thrashing. Are there any instructions or concerns about upgrading? What happens to the tens of thousands of old cache files on an upgraded system and what do you suggest as memory requirements in order to see any performance gain?
Nothing special to do or any concerns. It's just the first time you run the new version, it will take a bit longer to create the new cache file and delete all the old ones.

Memory requirements will certainly be dependent on the number of stations and days you are downloading. With 680 stations and 15 days, with no cached files and creating an XMLTV, memory usage got up to 1650MB for my tests. The same number of stations and days, with all data cached and no XMLTV, required 870MB.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

glorp

Posts: 357
Joined: Sun Sep 23, 2012 2:54 pm
Location:

HTPC Specs: Show details

#1802

Post by glorp » Fri Mar 20, 2020 4:45 pm

Yup got rid of ~60,000 cache files for one ~140K JSON with the first run this morning. That should speed things up going forward and reduce disk thrashing substantially. Nice job. Didn't really pay attention to memory use since it's being run on the headless server but that has 16GB available so shouldn't be an issue.

glorp

Posts: 357
Joined: Sun Sep 23, 2012 2:54 pm
Location:

HTPC Specs: Show details

#1803

Post by glorp » Sun Mar 22, 2020 9:28 pm

My task run times have fallen from 12-15 minutes to under 2.

User avatar
garyan2

Posts: 5917
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#1804

Post by garyan2 » Sun Mar 22, 2020 10:29 pm

Yah, I really wish I had done this a long time ago. I had always imagined I'd have to learn to use SQL/SQLite or something, but once I sat down and really looked at it, my solution was simple and the efforts paid out in spades. It's also nice to have a single file that I can search through to find info for troubleshooting if need be.

I'm now researching a little into how to compress the file, but I think it might take too much memory (serializing the data to a memory stream, to then send to a compression stream and to a file stream), but I could probably get the cache file to 15% of its' current size with normal compression.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

User avatar
garyan2

Posts: 5917
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#1805

Post by garyan2 » Mon Mar 23, 2020 2:16 am

All right, I've got compression going on. The table below says it all. This is with maximum compression. I'm going to try and benchmark on some slow systems to see if there are any side effects.
cacheImprovements.PNG
cacheImprovements.PNG (6.85 KiB) Viewed 994 times
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

Space

Posts: 2245
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#1806

Post by Space » Mon Mar 23, 2020 2:42 am

Personally I think it's fine the way it is. Not sure disk space is a huge concern for most people. The only reason I mentioned the disk space was due to the slack space and the efficiency aspect. The actual amount of space it took up was of little concern to me.

Plus I also like the idea that I can look through the cache file, although I probably will never do it again.

I'd rather have it run fast then save some disk space, but if it is a negligible increase in execution time then I guess it's fine.

Note that ages ago I did some tests with different compression algorithms for text (not binary) data and the ones that are better at text compression can be orders of magnitude better then generic compression algorithms that are designed for binary data.

Although, like I said, that was years ago, so compression may have improved since then.

EDIT:
Just looked back at my old test results and the best algorithm for compressing text at that time was PPMd with 7zip. Still looks like this is one of the best ones as far as speed and compression.

https://www.quora.com/What-is-the-best- ... -algorithm

Here's one of the notes I took:
gzip 16.697% of original size
7zip 2.727% of original size

I believe that gzip was using Lempel-Ziv and 7zip was using PPMd.

User avatar
IT Troll

Posts: 912
Joined: Sun Nov 27, 2011 9:42 am
Location: Edinburgh, UK

HTPC Specs: Show details

#1807

Post by IT Troll » Tue Mar 24, 2020 7:34 am

Oh interesting, I’ll have to get upgraded. I am guessing those running from SSD will not see much difference though?
Are you a Recorded TV HD user or want to give it a try? Check out the new community-made update; Recorded TV HD v2.1.1

User avatar
garyan2

Posts: 5917
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#1808

Post by garyan2 » Tue Mar 24, 2020 3:00 pm

Not quite as much, but I did see a difference. For an update run that was typically around 22s, I am now around 16s with the .20 version. This is on a circa '16 Mushkin SSD.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

npradeepkumar

Posts: 16
Joined: Sat Nov 26, 2011 10:00 am
Location:

HTPC Specs: Show details

#1809

Post by npradeepkumar » Mon Aug 17, 2020 9:54 pm

I have a question. Will this work with Windows 10, if I buy the media center pack? I had this up and running and was working fine, but recently my machine crashed and I had to re-install the OS and start over. Since Windows 7 is officially not supported anymore, it has been throwing hurdles every step of the way with lots of other software. So, I am debating if I should bite the bullet and switch to Windows 10.

I don't know if there are other issues or gotchas that I need to be aware of (if I am switching to Windows 10).

Thanks

stuartm

Posts: 540
Joined: Mon Nov 05, 2012 8:05 pm
Location: Longmont, CO

HTPC Specs: Show details

#1810

Post by stuartm » Mon Aug 17, 2020 11:27 pm

Be aware that Windows 10 does not support WMC, it will only work with a fair bit of hacking on your part. Also AIUI only OTA works, I don't think cable card will work with the win10 hacks (and definitely not any protected content). There is a whole subforum about WMC and win10 here viewforum.php?f=97. That said, if you go that route EPG123 does work with the hacked in version of WMC for win10.

npradeepkumar

Posts: 16
Joined: Sat Nov 26, 2011 10:00 am
Location:

HTPC Specs: Show details

#1811

Post by npradeepkumar » Mon Aug 17, 2020 11:33 pm

Thanks for the warnings. If protected content and premium channels like ESPN etc. do not work, then it is a no go for me. I guess, I have to stay on Windows 7 and make it work.

I have been looking at other options for whole home media solutions. I briefly looked at Plex, Silicondust (hd homerun). There seems to be no good option out there that can do all the things that WMC can do on Windows 7. What else am I missing? Is Direct Tv a better solution or is there any other solution that is better or at least equivalent with all the features that WMC has (Connectivity in multiple rooms, ability to play all channels including premium, DVR functionality etc.)

Apologize if this is not the right place to ask this question.

sptrainman

Posts: 51
Joined: Thu Jul 06, 2017 9:27 pm
Location:

HTPC Specs: Show details

#1812

Post by sptrainman » Mon Aug 24, 2020 4:39 pm

Hi,

My guide has not updated for the last two nights now. The problem seems to be due to missing channels in the data coming from Schedules Direct. Last night, the mxf imported, but the file is 72 hours old. So I'm not sure what that means.

I sent the trace log to you via email.

Thanks,
Jerry

User avatar
garyan2

Posts: 5917
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#1813

Post by garyan2 » Mon Aug 24, 2020 5:34 pm

sptrainman wrote:
Mon Aug 24, 2020 4:39 pm
Hi,

My guide has not updated for the last two nights now. The problem seems to be due to missing channels in the data coming from Schedules Direct. Last night, the mxf imported, but the file is 72 hours old. So I'm not sure what that means.

I sent the trace log to you via email.

Thanks,
Jerry
Jerry, I haven't received your trace.log file. Please send to support@garyan2.net.

EPG123 has a safety check whereas if the available number of stations being downloaded is less than 95% of the number of stations that were available when you last saved the configuration file, it will abort the update. This avoids a massive loss of channels in your guide that may or may not be real.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

User avatar
garyan2

Posts: 5917
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#1814

Post by garyan2 » Mon Aug 24, 2020 6:53 pm

garyan2 wrote:
Mon Aug 24, 2020 5:34 pm
sptrainman wrote:
Mon Aug 24, 2020 4:39 pm
Hi,

My guide has not updated for the last two nights now. The problem seems to be due to missing channels in the data coming from Schedules Direct. Last night, the mxf imported, but the file is 72 hours old. So I'm not sure what that means.

I sent the trace log to you via email.

Thanks,
Jerry
Jerry, I haven't received your trace.log file. Please send to support@garyan2.net.

EPG123 has a safety check whereas if the available number of stations being downloaded is less than 95% of the number of stations that were available when you last saved the configuration file, it will abort the update. This avoids a massive loss of channels in your guide that may or may not be real.
Got it. When you last saved your configuration file, you had 674 stations selected for download. On 8/21 you were downloading 687 stations meaning there were "new" stations added since then. The problem is on 8/22 that number of stations available for download was decreased to 633 which is 93.9% of the original 674. So it is 41 stations less than what were available when you last saved your configuration file, and 54 less than what was available on 8/21.

I'm not going to be able to tell what disappeared and/or added since you last saved your configuration file, but I suggest you open the configuration GUI and inspect to see if any stations you care about are missing. If you are okay with what is listed, just click the [Save & Execute] and everything will complete. Performing a save re-baselines the number of stations you are expecting to download to 633 instead of 674 and it will pass the safety check.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

NaivekGB

Posts: 2
Joined: Sat Mar 27, 2021 12:18 am
Location:

HTPC Specs: Show details

#1815

Post by NaivekGB » Sat Mar 27, 2021 12:30 am

Since last week, I am noticing the same error reported in this thread, did a upgrade to recent build epg123Setup_v1.6.3.0 and it keep failing to update the guide. Could you please look at the logs and let me know if there is a way to fix this?

Thanks

Code: Select all

[3/25/2021 12:00:01 AM] ===============================================================================
[3/25/2021 12:00:01 AM]  Beginning epg123 update execution. version 1.6.3.0
[3/25/2021 12:00:01 AM] ===============================================================================
[3/25/2021 12:00:01 AM] [ INFO] Beginning EPG123 update execution. 2021-03-25 07:00:01Z
[3/25/2021 12:00:01 AM] [ INFO] DaysToDownload: 14 , TheTVDBNumbers : True , PrefixEpisodeTitle: False , PrefixEpisodeDescription : False , AppendEpisodeDesc: False , OADOverride : True , TMDbCoverArt: True , IncludeSDLogos : True , AutoAddNew: True , CreateXmltv: False , ModernMediaUiPlusSupport: False
[3/25/2021 12:00:02 AM] [ INFO] Token request successful. serverID: 20141201.web.1
[3/25/2021 12:00:02 AM] [ INFO] Status request successful. account expires: 2021-07-22T00:27:01Z , lineups: 1/4 , lastDataUpdate: 2021-03-24T22:32:16Z
[3/25/2021 12:00:02 AM] [ INFO] system status: Online , message: No known issues.
[3/25/2021 12:00:04 AM] [ INFO] Successfully retrieved TMDb configurations.
[3/25/2021 12:00:04 AM] [ INFO] Successfully requested listing of client lineups from Schedules Direct.
[3/25/2021 12:00:04 AM] Entering buildLineupServices() for 1 lineups.
[3/25/2021 12:00:05 AM] [ INFO] Successfully retrieved the station mapping for lineup USA-WA62449-X.
[3/25/2021 12:00:07 AM] Exiting buildLineupServices(). SUCCESS.
[3/25/2021 12:00:07 AM] Entering getAllScheduleEntryMd5s() for 14 days on 616 stations.
[3/25/2021 12:02:10 AM] [ INFO] Successfully retrieved Md5s for 357 station's daily schedules. (0:00:02:02.9622244 /   532.248 KB)
[3/25/2021 12:07:10 AM] [ INFO] SD API WebException Thrown. Message: The operation has timed out , Status: Timeout . Trying again.
[3/25/2021 12:12:10 AM] [ INFO] SD API WebException Thrown. Message: The operation has timed out , Status: Timeout . Trying again.
[3/25/2021 12:12:10 AM] [ERROR] Failed to complete request. Exiting
[3/25/2021 12:12:10 AM] [ERROR] Did not receive a response from Schedules Direct for 357 station's daily schedules. (0:00:10:00.0234984)
[3/25/2021 12:12:10 AM] [ERROR] Problem occurred during getMd5ScheduleEntries(). Exiting.
[3/25/2021 12:12:10 AM] [ERROR] Failed to create MXF file. Exiting.
[3/25/2021 12:12:10 AM] [ INFO] epg123 update execution time was 00:12:09.2049562.
[3/25/2021 12:12:11 AM] ===============================================================================
[3/25/2021 12:12:11 AM]  Beginning epg123 client execution. version 1.6.3.0
[3/25/2021 12:12:11 AM] ===============================================================================
[3/25/2021 12:12:11 AM] [ INFO] Beginning epg123 client execution. 2021-03-25 07:12:11Z
[3/25/2021 12:12:11 AM] [ INFO] Import: True , Match: True , NoLogo: False , Force: False , ShowProgress: False , NoGC: False , NoVerify: False , Verbose: False
[3/25/2021 12:12:12 AM] Entering ImportMxfFile() for file "c:\programdata\garyan2\epg123\output\epg123.mxf".
[3/25/2021 12:14:02 AM] [ INFO] Successfully imported .mxf file into Media Center database. Exit code: 0
[3/25/2021 12:14:02 AM] Exiting ImportMxfFile(). SUCCESS.
[3/25/2021 12:14:02 AM] Entering VerifyLoad()
[3/25/2021 12:15:21 AM] [ INFO] Checked 197050 entries and corrected 0 of them.
[3/25/2021 12:15:21 AM] Exiting VerifyLoad()
[3/25/2021 12:15:25 AM] [ INFO] Completed the automatch of lineup stations to tuner channels.
[3/25/2021 12:15:26 AM] [ INFO] Completed lineup refresh.
[3/25/2021 12:15:26 AM] Entering RunWmcIndexTask(PvrScheduleTask)
[3/25/2021 12:15:26 AM] [ INFO] SUCCESS: Attempted to run the scheduled task "Microsoft\Windows\Media Center\PvrScheduleTask".
[3/25/2021 12:15:26 AM] [ INFO] Successfully started the PvrScheduleTask task. Exit code: 0
[3/25/2021 12:15:26 AM] Exiting RunWmcIndexTask(PvrScheduleTask). SUCCESS.
[3/25/2021 12:15:26 AM] Entering RunWmcIndexTask(ReindexSearchRoot)
[3/25/2021 12:15:26 AM] [ INFO] SUCCESS: Attempted to run the scheduled task "Microsoft\Windows\Media Center\ReindexSearchRoot".
[3/25/2021 12:15:26 AM] [ INFO] Successfully started the ReindexSearchRoot task. Exit code: 0
[3/25/2021 12:15:26 AM] Exiting RunWmcIndexTask(ReindexSearchRoot). SUCCESS.
[3/25/2021 12:15:28 AM] [ INFO] MXF file was created on 3/21/2021 3:05:43 PM
[3/25/2021 12:15:28 AM] [ERROR] The MXF file imported is 81.16 hours old.
[3/25/2021 12:15:30 AM] [ INFO] Completed EPG123 client execution.
[3/25/2021 12:15:30 AM] [ INFO] EPG123 client execution time was 00:03:19.2546155.
[3/26/2021 12:00:01 AM] ===============================================================================
[3/26/2021 12:00:01 AM]  Beginning epg123 update execution. version 1.6.3.0
[3/26/2021 12:00:01 AM] ===============================================================================
[3/26/2021 12:00:01 AM] [ INFO] Beginning EPG123 update execution. 2021-03-26 07:00:01Z
[3/26/2021 12:00:01 AM] [ INFO] DaysToDownload: 14 , TheTVDBNumbers : True , PrefixEpisodeTitle: False , PrefixEpisodeDescription : False , AppendEpisodeDesc: False , OADOverride : True , TMDbCoverArt: True , IncludeSDLogos : True , AutoAddNew: True , CreateXmltv: False , ModernMediaUiPlusSupport: False
[3/26/2021 12:00:02 AM] [ INFO] Token request successful. serverID: 20141201.web.1
[3/26/2021 12:00:03 AM] [ INFO] Status request successful. account expires: 2021-07-22T00:27:01Z , lineups: 1/4 , lastDataUpdate: 2021-03-26T02:35:03Z
[3/26/2021 12:00:03 AM] [ INFO] system status: Online , message: No known issues.
[3/26/2021 12:00:04 AM] [ INFO] Successfully retrieved TMDb configurations.
[3/26/2021 12:00:04 AM] [ INFO] Successfully requested listing of client lineups from Schedules Direct.
[3/26/2021 12:00:04 AM] Entering buildLineupServices() for 1 lineups.
[3/26/2021 12:00:05 AM] [ INFO] Successfully retrieved the station mapping for lineup USA-WA62449-X.
[3/26/2021 12:00:07 AM] [ INFO] Kicking off background worker to download and process 1 station logos.
[3/26/2021 12:00:07 AM] Exiting buildLineupServices(). SUCCESS.
[3/26/2021 12:00:07 AM] Entering getAllScheduleEntryMd5s() for 14 days on 616 stations.
[3/26/2021 12:02:06 AM] [ INFO] Successfully retrieved Md5s for 357 station's daily schedules. (0:00:01:58.1989145 /   532.255 KB)
[3/26/2021 12:07:06 AM] [ INFO] SD API WebException Thrown. Message: The operation has timed out , Status: Timeout . Trying again.
[3/26/2021 12:07:07 AM] [ INFO] SD API WebException Thrown. Message: The remote server returned an error: (502) Bad Gateway. , Status: ProtocolError
[3/26/2021 12:07:07 AM] [ERROR] Failed to complete request. Exiting
[3/26/2021 12:07:07 AM] [ERROR] Did not receive a response from Schedules Direct for 357 station's daily schedules. (0:00:05:01.4809963)
[3/26/2021 12:07:07 AM] [ERROR] Problem occurred during getMd5ScheduleEntries(). Exiting.
[3/26/2021 12:07:08 AM] [ERROR] Failed to create MXF file. Exiting.
[3/26/2021 12:07:08 AM] [ INFO] epg123 update execution time was 00:07:06.8550902.
[3/26/2021 12:07:08 AM] ===============================================================================
[3/26/2021 12:07:08 AM]  Beginning epg123 client execution. version 1.6.3.0
[3/26/2021 12:07:08 AM] ===============================================================================
[3/26/2021 12:07:08 AM] [ INFO] Beginning epg123 client execution. 2021-03-26 07:07:08Z
[3/26/2021 12:07:08 AM] [ INFO] Import: True , Match: True , NoLogo: False , Force: False , ShowProgress: False , NoGC: False , NoVerify: False , Verbose: False
[3/26/2021 12:07:09 AM] Entering PerformGarbageCleanup().
[3/26/2021 12:49:24 AM] [ INFO] Successfully completed garbage cleanup. Exit code: 0
[3/26/2021 12:49:24 AM] Exiting PerformGarbageCleanup(). SUCCESS.
[3/26/2021 12:49:26 AM] Entering ImportMxfFile() for file "c:\programdata\garyan2\epg123\output\epg123.mxf".
[3/26/2021 12:51:07 AM] [ INFO] Successfully imported .mxf file into Media Center database. Exit code: 0
[3/26/2021 12:51:07 AM] Exiting ImportMxfFile(). SUCCESS.
[3/26/2021 12:51:07 AM] Entering VerifyLoad()
[3/26/2021 12:52:04 AM] [ INFO] Checked 197050 entries and corrected 17 of them.
[3/26/2021 12:52:04 AM] Exiting VerifyLoad()
[3/26/2021 12:52:08 AM] [ INFO] Completed the automatch of lineup stations to tuner channels.
[3/26/2021 12:52:09 AM] [ INFO] Completed lineup refresh.
[3/26/2021 12:52:09 AM] Entering RunWmcIndexTask(PvrScheduleTask)
[3/26/2021 12:52:09 AM] [ INFO] SUCCESS: Attempted to run the scheduled task "Microsoft\Windows\Media Center\PvrScheduleTask".
[3/26/2021 12:52:09 AM] [ INFO] Successfully started the PvrScheduleTask task. Exit code: 0
[3/26/2021 12:52:09 AM] Exiting RunWmcIndexTask(PvrScheduleTask). SUCCESS.
[3/26/2021 12:52:09 AM] Entering RunWmcIndexTask(ReindexSearchRoot)
[3/26/2021 12:52:09 AM] [ INFO] SUCCESS: Attempted to run the scheduled task "Microsoft\Windows\Media Center\ReindexSearchRoot".
[3/26/2021 12:52:09 AM] [ INFO] Successfully started the ReindexSearchRoot task. Exit code: 0
[3/26/2021 12:52:09 AM] Exiting RunWmcIndexTask(ReindexSearchRoot). SUCCESS.
[3/26/2021 12:52:09 AM] [ INFO] MXF file was created on 3/21/2021 3:05:43 PM
[3/26/2021 12:52:09 AM] [ERROR] The MXF file imported is 105.77 hours old.
[3/26/2021 12:52:11 AM] [ INFO] Completed EPG123 client execution.
[3/26/2021 12:52:11 AM] [ INFO] EPG123 client execution time was 00:45:03.2917421.

User avatar
Gripweed

Posts: 103
Joined: Sun Feb 03, 2013 6:35 pm
Location: Scranton,PA_USA

HTPC Specs: Show details

#1816

Post by Gripweed » Sat Mar 27, 2021 4:02 am

Try changing the time for your guide update.
Do not choose the top of the hour, the bottom of the hour, and the quarter hours especially during the overnight hours around midnight +/- 4 hours. Plus on the East Coast and minus on the West. This seems to be the busiest times for SD and they are probably being overload.

NaivekGB

Posts: 2
Joined: Sat Mar 27, 2021 12:18 am
Location:

HTPC Specs: Show details

#1817

Post by NaivekGB » Sun Mar 28, 2021 2:36 am

Cool thanks! Changed it to a different time...

Post Reply