Introducing EPG123

An evolving, supported alternative to Rovi
Forum rules
★ Download the latest EPG123 here: http://epg123.garyan2.net <> Setup guide here: http://epg123.garyan2.net/downloads/epg123_Guide.pdf
Post Reply
User avatar
garyan2

Posts: 5123
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

Re: Introducing EPG123

#1801

Post by garyan2 » Fri Mar 20, 2020 2:09 am

glorp wrote:
Thu Mar 19, 2020 11:55 pm
I'm glad to see that the cache will finally be a single file. That should substantially improve the disk thrashing. Are there any instructions or concerns about upgrading? What happens to the tens of thousands of old cache files on an upgraded system and what do you suggest as memory requirements in order to see any performance gain?
Nothing special to do or any concerns. It's just the first time you run the new version, it will take a bit longer to create the new cache file and delete all the old ones.

Memory requirements will certainly be dependent on the number of stations and days you are downloading. With 680 stations and 15 days, with no cached files and creating an XMLTV, memory usage got up to 1650MB for my tests. The same number of stations and days, with all data cached and no XMLTV, required 870MB.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

glorp

Posts: 354
Joined: Sun Sep 23, 2012 2:54 pm
Location:

HTPC Specs: Show details

#1802

Post by glorp » Fri Mar 20, 2020 4:45 pm

Yup got rid of ~60,000 cache files for one ~140K JSON with the first run this morning. That should speed things up going forward and reduce disk thrashing substantially. Nice job. Didn't really pay attention to memory use since it's being run on the headless server but that has 16GB available so shouldn't be an issue.

glorp

Posts: 354
Joined: Sun Sep 23, 2012 2:54 pm
Location:

HTPC Specs: Show details

#1803

Post by glorp » Sun Mar 22, 2020 9:28 pm

My task run times have fallen from 12-15 minutes to under 2.

User avatar
garyan2

Posts: 5123
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#1804

Post by garyan2 » Sun Mar 22, 2020 10:29 pm

Yah, I really wish I had done this a long time ago. I had always imagined I'd have to learn to use SQL/SQLite or something, but once I sat down and really looked at it, my solution was simple and the efforts paid out in spades. It's also nice to have a single file that I can search through to find info for troubleshooting if need be.

I'm now researching a little into how to compress the file, but I think it might take too much memory (serializing the data to a memory stream, to then send to a compression stream and to a file stream), but I could probably get the cache file to 15% of its' current size with normal compression.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

User avatar
garyan2

Posts: 5123
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#1805

Post by garyan2 » Mon Mar 23, 2020 2:16 am

All right, I've got compression going on. The table below says it all. This is with maximum compression. I'm going to try and benchmark on some slow systems to see if there are any side effects.
cacheImprovements.PNG
cacheImprovements.PNG (6.85 KiB) Viewed 422 times
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

Space

Posts: 1988
Joined: Sun Jun 02, 2013 9:44 pm
Location:

HTPC Specs: Show details

#1806

Post by Space » Mon Mar 23, 2020 2:42 am

Personally I think it's fine the way it is. Not sure disk space is a huge concern for most people. The only reason I mentioned the disk space was due to the slack space and the efficiency aspect. The actual amount of space it took up was of little concern to me.

Plus I also like the idea that I can look through the cache file, although I probably will never do it again.

I'd rather have it run fast then save some disk space, but if it is a negligible increase in execution time then I guess it's fine.

Note that ages ago I did some tests with different compression algorithms for text (not binary) data and the ones that are better at text compression can be orders of magnitude better then generic compression algorithms that are designed for binary data.

Although, like I said, that was years ago, so compression may have improved since then.

EDIT:
Just looked back at my old test results and the best algorithm for compressing text at that time was PPMd with 7zip. Still looks like this is one of the best ones as far as speed and compression.

https://www.quora.com/What-is-the-best- ... -algorithm

Here's one of the notes I took:
gzip 16.697% of original size
7zip 2.727% of original size

I believe that gzip was using Lempel-Ziv and 7zip was using PPMd.

User avatar
IT Troll

Posts: 794
Joined: Sun Nov 27, 2011 9:42 am
Location: Edinburgh, UK

HTPC Specs: Show details

#1807

Post by IT Troll » Tue Mar 24, 2020 7:34 am

Oh interesting, I’ll have to get upgraded. I am guessing those running from SSD will not see much difference though?
Are you a Recorded TV HD user or want to give it a try? Check out the new community-made update; Recorded TV HD v2.1

User avatar
garyan2

Posts: 5123
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#1808

Post by garyan2 » Tue Mar 24, 2020 3:00 pm

Not quite as much, but I did see a difference. For an update run that was typically around 22s, I am now around 16s with the .20 version. This is on a circa '16 Mushkin SSD.
- Gary
Keeping WMC alive beyond January 2020. http://epg123.garyan2.net

Post Reply