EPG123 Log Parser

An evolving, supported alternative to Rovi
Forum rules
★ Download the latest EPG123 here: https://garyan2.github.io/ <> Setup guide here: https://garyan2.github.io/install.html
User avatar
StinkyImp

Posts: 669
Joined: Thu May 11, 2017 7:53 pm
Location:

HTPC Specs: Show details

EPG123 Log Parser

#1

Post by StinkyImp » Mon Dec 11, 2017 11:01 pm

Hello everyone!

I guess I'm getting too lazy, too old, eyesight going bad, or a combination of all of it. I know I can open the trace.log file and visually scan it or perform a search in notepad for errors and whatnot... But, it all comes back to lazy, old, failing eyesight (and probably other stuff as well).

Anyway, I decided to try and make it easy on myself because Gary has done such a wonderful job on his program that I don't find myself looking at the logs much at all. When I do, I find that they've grown pretty sizable and tedious for an old geezer to navigate. Besides, all I want to look at are the errors, warnings, and stats. So I wrote a little utility to help me with that.

It's called "EPG123 Log Parser" and it's a simple exe that can be dropped anywhere. I dropped it in a folder under my EPG123 directory called "Other Stuff".

Image

The first time it's run, it will create an ini file (saving file names and locations) in the same directory that it's run from. It will also create 2 dll files that give it access to:

wwsop34i.dll - Shell operations (for file operation capabilities)
WBDIB44I.DLL - WIL Interpreter (Interprets the source instructions)

These can be safely deleted and will be created each time the parser is run.

Image

The first time It's run it needs to know where the EPG123 log file is located. I had multiple log files, so I just selected one of them.

Image

After that I created a simple interface that shows me the file that I'm working with and then gives me options for what I'd like to see.

Image

Errors - Creates a concise listing of errors.
Warnings - Creates a concise listing of warnings.
Stats - Creates a concise listing of stats.
Full Log - Opens the currently selected log file.
Select Different Log File - I had multiple log files, so I needed a way to select a different one!
Exit - ...

When a button is pressed (for example Errors) the program will open the selected log file in READ ONLY mode. As it analyzes each line of the log, it will copy only the relevant lines to a new file. When it has analyzed each line in the log, it will open a new Notepad document.

Image

It creates these "trash" files in the same directory as the log file that was selected and names them "OK_TO_DELETE_XXXXXXOutput.txt". Where "XXXXXX" is Error, Stats, or Warning as applicable.

Image

And as it says, they're OK to delete! All in all it creates 6 files (3 text files, 2 dll's, and an ini file) that can be easily deleted. It doesn't write anything to the registry and deleting the exe is all that's needed to "uninstall" it.

I've successfully run it in Windows 10(x64) and Windows 7 Pro(x64).

stuartm

Posts: 721
Joined: Mon Nov 05, 2012 8:05 pm
Location: Longmont, CO

HTPC Specs: Show details

#2

Post by stuartm » Tue Dec 12, 2017 1:41 am

If you want to save yourself from ever having to look in your logs you can also follow the instructions I posted here:
viewtopic.php?f=99&t=10984
and get an e-mail if EPG123 has thrown an error or warning.

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#3

Post by garyan2 » Tue Dec 12, 2017 5:07 am

This is pretty cool, though I may see something different than everyone else.

What I see is a system that has been online at least 10 months and, regardless of cause, has only had 9 days that may not have received an update.

@StinkyImp
Did you intend to attach the utility, or are you still polishing it up? I wouldn't mind scanning my server to see what it says, though I did a complete rebuild about 6 months ago.

EDIT: I'll actually revise that number to 5 days in the last 10 months and 2 of those is SD down for maintenance and a recording in progress.
EDT2: Well, shoot ... it was only for 6 months, not 10.
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

User avatar
StinkyImp

Posts: 669
Joined: Thu May 11, 2017 7:53 pm
Location:

HTPC Specs: Show details

#4

Post by StinkyImp » Tue Dec 12, 2017 11:52 am

garyan2 wrote:This is pretty cool, though I may see something different than everyone else.

What I see is a system that has been online at least 10 months and, regardless of cause, has only had 9 days that may not have received an update.

@StinkyImp
Did you intend to attach the utility, or are you still polishing it up? I wouldn't mind scanning my server to see what it says, though I did a complete rebuild about 6 months ago.

EDIT: I'll actually revise that number to 5 days in the last 10 months and 2 of those is SD down for maintenance and a recording in progress.
EDT2: Well, shoot ... it was only for 6 months, not 10.
Thanks garyan2! I built this system about 6 months ago, so that's all the logs I have. I should probably pull the logs off my old HTPC and see what they show!

Your program is so rock solid that the errors and whatnot are pretty few and far between. I just wanted a way to look at only those. I also didn't know whether anyone (especially you) would be offended by it. Therefore this post was just a "feeler". I figured if anyone was opposed to it, I'd keep it to myself, otherwise I'd share it with anyone that thought it might be useful to them.

Polished? I'm like anyone else. I see things in it I'd like to improve, but will I? It depends... I'm like you though, I wanted to see every error listed by itself in sequential order so I could have an overview. If you'd like I can PM you a copy with the source so you can verify that it won't affect your program or logging in any way.

User avatar
garyan2

Posts: 7438
Joined: Fri Nov 27, 2015 7:23 pm
Location:

HTPC Specs: Show details

#5

Post by garyan2 » Wed Dec 13, 2017 7:08 am

Certainly not offended... the log file is there first for a troubleshooting aid, but I also use it to measure how well things are performing. Kind of a health check on my setup. By all means, have fun with it and go as far as you want.

If you can, try to ensure that your I/O access to the file does not lock the file (read only, shared?). If your program has the file locked while epg123 tries to write to it, well nothing bad will happen; it will just fail to write to the file. Of course, the user could simply make sure they didn't run your program during an update. :)
- Gary
Keeping WMC alive beyond January 2020. https://garyan2.github.io

User avatar
StinkyImp

Posts: 669
Joined: Thu May 11, 2017 7:53 pm
Location:

HTPC Specs: Show details

#6

Post by StinkyImp » Thu Dec 14, 2017 3:40 pm

Hmmm... For the past two days I've tried to PM you a copy of this. Yesterday I was getting "The connection to the server was reset while the page was loading." error. So this morning I tried changing extensions, zipping, etc. The source is a Notepad txt file and it returned "The extension txt is not allowed." in addition to the "server reset" error.

What kind of attachments are allowed to PM's? Maybe it's a size limit, but the text document is only ~12KB.

Sammy2

Posts: 1708
Joined: Fri Aug 24, 2012 7:35 pm
Location:

HTPC Specs: Show details

#7

Post by Sammy2 » Fri Dec 15, 2017 2:16 pm

Will it parse channel number changes, removals and additions?
Thanks

Sent from my SM-G930P using Tapatalk

User avatar
StinkyImp

Posts: 669
Joined: Thu May 11, 2017 7:53 pm
Location:

HTPC Specs: Show details

#8

Post by StinkyImp » Sat Dec 16, 2017 11:43 am

Sammy2 wrote:Will it parse channel number changes, removals and additions?
Thanks
I'm going to rewrite this because I found that I wanted other information that I didn't include in the original version, like "execution times". When I was adding in that functionality I realized that the utility was becoming larger and losing efficiency.

I'd like to look at other things too (like channel number changes, etc)... So I'm going to scrap the version I was building and start from square one. Right now the things it looks for are based on [BRACKET] tags that garyan2 uses to make certain things pop. I envision a tool that will have a drop down of common queries, but will also allow a custom search string to be used.

So in response to Will it parse channel number changes, removals and additions?... Not at the moment. The next version "should" be able to pull out only the information you want to see.

User avatar
StinkyImp

Posts: 669
Joined: Thu May 11, 2017 7:53 pm
Location:

HTPC Specs: Show details

#9

Post by StinkyImp » Sun Dec 17, 2017 7:08 pm

StinkyImp wrote:
Sammy2 wrote:Will it parse channel number changes, removals and additions?
Thanks
I'm going to rewrite this because I found that I wanted other information that I didn't include in the original version, like "execution times". When I was adding in that functionality I realized that the utility was becoming larger and losing efficiency.
ADMIN PLEASE HELP! Every time I try to upload this utility to a post, or to a PM, I get the "Connection reset by server" error. Can you assist me with this problem? I've tried zipping, changing the extension, etc...


ALPHA VERSION
Interface NEW.png
Things to know:
  • 1. I was happily generating reports with this when (after awhile) I realized I had 25+ Notepad documents open. Therefore... This utility is very Notepad'centric. It checks for the existence of any open Notepad instances and won't continue processing until they are closed.
  • 2. On initial startup it will check to see if you have any Notepad instances open and will give you only ONE chance to close them before it shuts itself down.
  • 3. Upon your first use, you will be presented with a file browser to select the location of the log file you'd like to analyze.
  • 4. This utility never affects your "live" log file. It only works with a temporary copy.
  • 5. (Suggestion) Create a directory to stick this in. Then to uninstall, all you need to do is delete that directory.
  • 6. This was compiled and tested on Win10(x64) and Win7(x64). I'm not sure if it will run on 32 bit systems (or Win8) because I don't have one.
  • 7. TO UNINSTALL - If you used the suggestion above, simply delete the directory you created. This utility is completely portable and makes absolutely no changes to your system.
Buttons, fields, and their functions:

Section (Working Log File)
  • Select Different Log File - If you have multiple log files, you can select a new one to work with.
  • Open Working Log - Opens a temporary copy of the current log.
Section (Select Search Criteria)
  • Predefined Search Terms (Field) - This is a list of common terms found in EPG123.
  • Add Terms to List - Allows you to add terms to the predefined list. This is a work in progress. For now it only saves the changes for the current session.
  • Generate Predefined - Performs a search query and returns a summary report.
  • Enter Custom String (Field) - You can type (or copy/paste) a custom search word or phrase to search for.
  • Generate Custom - Performs a search query and returns a summary report.
Open Program Directory - Opens the directory where all the utility's files are kept. Convenient for transparency and uninstalling.
Delete Temp Files - Allows you to delete all the temporary files that you generate except the temporary copy of the log file. It uses that to work with.
Exit - Will perform cleanup operations (removes all temporary files, including the temporary log file) and cleanly exits the utility.

Section (Credits) - Tries to avert any hassles for garyan2 but also points potential users to his EXCELLENT program.

User avatar
Scallica

Posts: 2797
Joined: Mon Jun 06, 2011 7:09 pm
Location: USA!

HTPC Specs: Show details

#10

Post by Scallica » Sun Dec 17, 2017 9:22 pm

StinkyImp wrote:ADMIN PLEASE HELP! Every time I try to upload this utility to a post, or to a PM, I get the "Connection reset by server" error. Can you assist me with this problem? I've tried zipping, changing the extension, etc...
No need to shout :-) How large is the file?
HTPC Enthusiast / Forum Moderator - TGB.tv Code of Conduct

User avatar
StinkyImp

Posts: 669
Joined: Thu May 11, 2017 7:53 pm
Location:

HTPC Specs: Show details

#11

Post by StinkyImp » Sun Dec 17, 2017 9:57 pm

Scallica wrote:
StinkyImp wrote:ADMIN PLEASE HELP! Every time I try to upload this utility to a post, or to a PM, I get the "Connection reset by server" error. Can you assist me with this problem? I've tried zipping, changing the extension, etc...
No need to shout :-) How large is the file?
ROFL! That wasn't so much shouting as it was to be easily noticed. ;)

Zipped it's 574 KB. But... I tried to attach a 12 KB text file the other day and it failed too.

User avatar
Scallica

Posts: 2797
Joined: Mon Jun 06, 2011 7:09 pm
Location: USA!

HTPC Specs: Show details

#12

Post by Scallica » Mon Dec 18, 2017 1:32 am

Test file.
Attachments
test-file.zip
Zipped Text File 39KB
(323 Bytes) Downloaded 56 times
HTPC Enthusiast / Forum Moderator - TGB.tv Code of Conduct

User avatar
StinkyImp

Posts: 669
Joined: Thu May 11, 2017 7:53 pm
Location:

HTPC Specs: Show details

#13

Post by StinkyImp » Mon Dec 18, 2017 9:08 am

The attachment test-file.zip is no longer available
Connection Reset 01.png
I'm trying to upload EPG123 Log Parser.zip
575 KB and receive the following error every time.
Attachments
Connection Reset.png

User avatar
StinkyImp

Posts: 669
Joined: Thu May 11, 2017 7:53 pm
Location:

HTPC Specs: Show details

#14

Post by StinkyImp » Mon Dec 18, 2017 9:20 am

The attachment test-file.zip is no longer available
Connection Reset Chrome.png
Chrome error message
The previous post was in Firefox. This post is using Chrome. I can add your file and png files without any issues. Every time I try to add "EPG123 Log Parser.zip" I get the server reset error. Are there restrictions in place at my user level?

User avatar
StinkyImp

Posts: 669
Joined: Thu May 11, 2017 7:53 pm
Location:

HTPC Specs: Show details

#15

Post by StinkyImp » Mon Dec 18, 2017 9:42 am

The button [Add Terms to List] has been modified to allow changes to be persistent/permanent.

If the file named "EPG123 Predefined Terms.txt" in the program directory is deleted, then this utility will rebuild it using the default search terms and any changes will be lost.

User avatar
Scallica

Posts: 2797
Joined: Mon Jun 06, 2011 7:09 pm
Location: USA!

HTPC Specs: Show details

#16

Post by Scallica » Mon Dec 18, 2017 1:43 pm

Ahh, we don't allow EXE files. The ZIP contains an EXE file; that's why it's failing to upload. Rename the EXE to EPG_Log_Parser.zip. Then when a user downloads it, they can rename it back to .exe.
HTPC Enthusiast / Forum Moderator - TGB.tv Code of Conduct

User avatar
StinkyImp

Posts: 669
Joined: Thu May 11, 2017 7:53 pm
Location:

HTPC Specs: Show details

#17

Post by StinkyImp » Mon Dec 18, 2017 1:47 pm

Scallica wrote:Ahh, we don't allow EXE files. The ZIP contains an EXE file; that's why it's failing to upload. Rename the EXE to EPG_Log_Parser.zip. Then when a user downloads it, they can rename it back to .exe.
I'll give that a shot. The extension on this file needs to be changed from .zip to .exe.
EPG123 Log Parser.zip

User avatar
StinkyImp

Posts: 669
Joined: Thu May 11, 2017 7:53 pm
Location:

HTPC Specs: Show details

#18

Post by StinkyImp » Sat Dec 23, 2017 11:26 pm

I've been working on this thing until my eyes are crossed. This is an entirely different utility based on garyan2's input and will look for trace logs created by EPG123. It will merge together up to three log files and use that to work with. If EPG123 has just one or two files, that's what it will use. The three logs that were created on my system spanned from February 5, 2017 through today and has close to 22,000 lines of data!

This version has a ton of options but still only works off "copies" of the EPG123 log file(s). I've removed the Notepad checking because it created too much overhead and well... the user can close any instances that are created. This utility will give an occasional reminder to close them, but that's up to the user.

Here's the current interface.
The attachment Log Parser Interface 01.png is no longer available
and here's a sample report.
Log Parser Interface 01.png
It took 13 seconds to extract the 14 lines that included the search term "client execution time" between April 1st and April 15th by going through almost 22,000 lines of data.

As hard as I tried (putting everything about 5 to 6 directories deep on a USB drive) I couldn't perform a query that took even a minute to perform. The worst timed query was 51 seconds.

NOTE: "client execution time" does not equal "Client Execution Time". The queries are case sensitive.
TIP: Use the "Open Working Log" button to open the full log and copy/paste your search criteria. If it's something you'll use consistently, add it to the predefined list.

To reiterate, it's a simple exe that can be dropped anywhere. I dropped it in a folder under my EPG123 directory called "Log Parser". So to completely uninstall it I can just delete that directory. Nothing is ever written to the registry (or anywhere else) and it keeps itself contained. Change the "zip" extension to "exe" and that's it!
Report Example 01.png

Alan G

Posts: 89
Joined: Mon Feb 22, 2016 8:27 pm
Location: Bethesda, Maryland

HTPC Specs: Show details

#19

Post by Alan G » Sun Dec 24, 2017 1:09 pm

Cool utility. I'll give it a look but since I just rebuilt my HTPC with a more modern MoBo and CPU 10 days ago I don't have much of a log to look at. Probably going forward it will be useful. thanks for the work on this, when one has a huge log, finding stuff can be a real drag.

User avatar
StinkyImp

Posts: 669
Joined: Thu May 11, 2017 7:53 pm
Location:

HTPC Specs: Show details

#20

Post by StinkyImp » Sun Dec 24, 2017 4:48 pm

Alan G wrote:Cool utility. I'll give it a look but since I just rebuilt my HTPC with a more modern MoBo and CPU 10 days ago I don't have much of a log to look at. Probably going forward it will be useful. thanks for the work on this, when one has a huge log, finding stuff can be a real drag.
Hi Alan.

Thanks for your kind words! This is a work in progress. One of the reasons I built this utility is for the exact reason you stated. Garyan2 has designed EPG123 to have up to three log files. When the trace.log file reaches a certain size, he timestamps it, backs it up, and then creates a new trace.log. He does this twice and then deletes the oldest backup. Therefore, you could conceivably have one trace.log and two timestamped backups.

This utility attempts to merge these log files and then give the user various tools to analyze it. As of this morning my "merged log" starts in early February and includes 21,999 (almost 20,000) lines of data.

Attached is a maintenance update that adds some functionality or corrects some issues. I think I'm the only one really using it right now and I find it hard to test every scenario there might be so... any feedback would be useful.

Changes in this version - 1.5.1 (ALPHA)
  • 1. Creates a directory under Epg123 called "Log Backups". I did this so that in the future I could manually merge the oldest log backups (or create a routine that recognizes and automatically merges the oldest logs) and maintain a huge history that could conceivably span years (if not decades).
  • 2. Fixed a bug concerning quotation marks that, on rare occasions, could hang the utility.
  • 3. Report enhancements.
  • 4. Better error handling and code refinement.
  • 5. Minor stuff that was fixed on the fly.
EPG123 Log Parser.zip

Post Reply