High Memory Usage?

marvin-miller

Posts: 221
Joined: Tue Sep 18, 2012 9:50 pm
Location:

HTPC Specs: Show details

High Memory Usage?

#1

Post by marvin-miller » Thu Dec 01, 2016 11:28 pm

Hi Folks;

I'm running Windows 7 x64 and Media Center with a Hauppauge Colossus.

Recently I installed two 8TB drives to store my recorded content. One for Series recordings and one for Movies. Both are getting low on space so I'll be adding some more.

I have 8GB of memory installed and I'm seeing periodic paging to disk with a corresponding system slow down. Basically memory usage hits 7GB at times. The system is basically a dedicated DVR/workstation. Nothing I do uses much memory at all. Even with everything closed off other then Task Manager I still see periods of memory usage hitting 7GB of the 8 installed.

Normally, the system uses +/- 2GB. So something is going on in the background that causes memory usage spikes to 7GB and a lot of paging. After some time it will drop back down again. I am quite certain it's Media Center doing something in the background and I often wonder if it's related to indexing or something. I have;

2,544 individual series recordings and...
562 Movies recorded - both on separate 8TB drives.

When I check the task manager I can't see where the memory is going as it does not seem to show it. It does this with any new install, and will do it with virtually nothing running at all.
The paging/slowness is exacerbated by the system disk being an old RAID 0 set (disks about 10 years old now!). I can work around it by putting the pagefile on an SSD and that makes it livable. Also, another thing I found was moving the eHome directory to an SSD makes things noticeably better. Since then the SSD's were pulled and re-allocated to other systems.

I am curious though, it really does seem to be Media Center that's causing the memory usage and it is intermittent. It will happen several times a day. It's as if it's spawning a process and that process is causing high memory usage. Anyone seeing the same or have seen something similar? :wtf:

marvin-miller

Posts: 221
Joined: Tue Sep 18, 2012 9:50 pm
Location:

HTPC Specs: Show details

#2

Post by marvin-miller » Fri Dec 02, 2016 12:32 am

The high memory usage seems to be coming, in part, from Search Indexer. I can typically trigger it by firing up Media Center and then clicking on the Movies section.

This seems a bit odd as I would have thought it would build an index and then leave it alone unless a movie was added to it. Either way, I think the issue lies in that area....

marvin-miller

Posts: 221
Joined: Tue Sep 18, 2012 9:50 pm
Location:

HTPC Specs: Show details

#3

Post by marvin-miller » Fri Dec 02, 2016 7:46 am

I'm no longer sure what's going on. Re-start the computer, don't touch Media Center at all, wait a while and then, all of a sudden memory usage spikes to 7+ out of 8 Physical GB's installed. Can't seen to get my head around it.

I did see a lot of other posts from people about high memory usage and WMC.

Any others out there seeing the same?

DanH

Posts: 57
Joined: Wed Jun 22, 2011 1:19 pm
Location:

HTPC Specs: Show details

#4

Post by DanH » Fri Dec 02, 2016 6:34 pm

You can try to disable the Windows Search service. I do not know if this will effect cataloging or searching.

I recommend that you move your system drive to SSD and this issue will likely go away. The significantly faster drive access of the SSD makes a huge difference. It will not eliminate the service using the memory but the spike time will be very short and should not be noticeable.
Was Danh_HP_m376n

marvin-miller

Posts: 221
Joined: Tue Sep 18, 2012 9:50 pm
Location:

HTPC Specs: Show details

#5

Post by marvin-miller » Fri Dec 02, 2016 7:53 pm

Yeah, I tried that. Disabling the service doesn't do anything. Research from MS seems to indicate that in some instances shutting off the service won't cause it release memory. Disabling the service and doing a shut down/restart hasn't been tested yet.

Thing is, I'm not convinced it's the search service. From what I can see, the search service indexes the guide so that you can do keyword searches.

The index is only about 70 megs and it's been deleted and re-built.

Why do I get the feeling that Media Center probably has a big memory leak and that Microsoft will disavow any knowledge of it? (and therefore, not address it).

Unfortunately, it doesn't seem like too many people are using it anymore. I'd love to sample a good cross-section of Media Center users to see if they are seeing high memory usage as well.

stuartm

Posts: 721
Joined: Mon Nov 05, 2012 8:05 pm
Location: Longmont, CO

HTPC Specs: Show details

#6

Post by stuartm » Fri Dec 02, 2016 8:57 pm

I wonder if it's due to Media Center caching things related to your large number of saved recordings. I have not seen any evidence of high memory usage in normal usage of WMC. Although I haven't looked since I am not seeing any problems. I probably have a few hundred recordings saved on 4TB of storage. One odd event I experienced recently though was I deleted a hundred or so recordings in one session (one after another) via the WMC GUI and the system got extremely slow (i.e. acting like out of memory) and I ended up having to reboot to clear things up. This is with 16GB of memory but 9GB is dedicated to a ramdisk for live TV buffering.

User avatar
d00zah

Posts: 242
Joined: Fri Nov 07, 2014 7:20 pm
Location:

HTPC Specs: Show details

#7

Post by d00zah » Fri Dec 02, 2016 9:01 pm

Marvin, I began having issues with WMC performance when my movie collection was about the same size as yours... & it'll only get worse. Everything I read pointed to poor design of WMC database management which, obviously, was unlikely to ever be fixed. I suggest looking into alternate media servers with WMC support. I looked at several a few years back & decided on MediaBrowser3 (now Emby), but there are others with various features & associated costs. Take a look around.

Emby has a server component which allows collection management from your favorite browser & a WMC client plug-in which can replace the native WMC movie strip. These are FREE. Their forum/online WIKI are a good source of info.

RyC

Posts: 724
Joined: Tue Aug 21, 2012 10:21 pm
Location:

HTPC Specs: Show details

#8

Post by RyC » Fri Dec 02, 2016 9:35 pm

I'm pushing almost 4000 recordings and I haven't noticed any issues with 8gb of RAM, however, I don't use it for anything other than WMC


Sent from my iPhone using Tapatalk

User avatar
d00zah

Posts: 242
Joined: Fri Nov 07, 2014 7:20 pm
Location:

HTPC Specs: Show details

#9

Post by d00zah » Fri Dec 02, 2016 10:04 pm

Out of curiosity, do your libraries include any metadata (i.e.YAMMM, etc.)? Even with 8GB RAM/quad core i5, adding new content was ponderous, as Marvin described. With library management offloaded, I find the experience MUCH improved & intend to stick w/ WMC7/Emby indefinitely. YMMV

marvin-miller

Posts: 221
Joined: Tue Sep 18, 2012 9:50 pm
Location:

HTPC Specs: Show details

#10

Post by marvin-miller » Fri Dec 02, 2016 10:54 pm

stuartm wrote:I wonder if it's due to Media Center caching things related to your large number of saved recordings. I have not seen any evidence of high memory usage in normal usage of WMC. Although I haven't looked since I am not seeing any problems. I probably have a few hundred recordings saved on 4TB of storage. One odd event I experienced recently though was I deleted a hundred or so recordings in one session (one after another) via the WMC GUI and the system got extremely slow (i.e. acting like out of memory) and I ended up having to reboot to clear things up. This is with 16GB of memory but 9GB is dedicated to a ramdisk for live TV buffering.
Interesting...I think you're experiencing some of this issue when you said you used the GUI to delete a lot of recordings in one session. This is the kind of behavior I run into all the time until the system bogs down and sits on the page file for protracted periods.

Keep in mind, I have two data disks. One for Series and one for Movies. If I remove both from the Media Center library nothing changes. :crazy: That should, in theory, strip Media Center down to just watching the Recorded TV folder which never contains anything but a single recording (usually). But, I'm encouraged by your report about the system getting slow. I can almost garantee you that if you had task manager up you would have seen all available memory being used up as it does with mine.
d00zah wrote:Marvin, I began having issues with WMC performance when my movie collection was about the same size as yours... & it'll only get worse. Everything I read pointed to poor design of WMC database management which, obviously, was unlikely to ever be fixed. I suggest looking into alternate media servers with WMC support. I looked at several a few years back & decided on MediaBrowser3 (now Emby), but there are others with various features & associated costs. Take a look around.

Emby has a server component which allows collection management from your favorite browser & a WMC client plug-in which can replace the native WMC movie strip. These are FREE. Their forum/online WIKI are a good source of info.
I'm going to look into this shortly - thank you VERY much for the confirmation as well as what sounds like the solution! :thumbup:
RyC wrote:I'm pushing almost 4000 recordings and I haven't noticed any issues with 8gb of RAM, however, I don't use it for anything other than WMC
That's a tough one for me to swallow as I can't help but think this is not a unique issue. However, I do think it depends on how the data is organized. On my Series drive every Series is in it's own folder. So when I click on Videos I get a list of about 30 folders and then drill down into the one I want. IE, Go into the NCIS folder and then all the NCIS episodes show up. I never seen to have delays in the Series section (Videos) in Media Center. That always responds well unless it has to generate 100 icons or something, but that's to be expected.

It seems to be the Movies drive where all the issues come from. The Movies drive does not have any folder structure. It's just Movies dumped into the root of the drive. Approx 600 of them. When I click on Movies in the GUI it always pauses, you get the round circle for some time (sometimes up to a minute or more) and then eventually they all show up. So this is where I tend to experience the delays when using Media Center.

Also, when in Movies, if I delete one, then everything pauses for some time. The GUI will not be responsive until it's actually deleted it. Also, they often still show up even though they were deleted! The solution seems to be to exit Movies and then go back into Movies and then wait for the system (the busy circle) until it becomes responsive again. At that time, usually the Movie will not show up but instead you will see a file called something like $103833. If I play that oddly named file I find out it's the movie I just deleted.

So it seems to take a lot of time to work with the Movies directory. When a Movie gets added to it, it also does the same thing. You go into Movies, you have to wait for the circle, the system becomes unresponsive, until, finally, it does whatever it's doing. At that point it becomes responsive again. It very much seems that going into Movies, when a new one has been added, triggers some kind of indexing and until that's done, it's going to the page file. Same thing when deleting one. It seems to 'notice' and then re-index (or something like that) and then you have to wait until it's done.

Here's something really odd. I had recorded two instances of Stargate - The Ark of Truth. So I deleted the oldest version from Movies using the GUI. It kept showing both of them still being there in the list. I then closed off Media Center, went back in, went into Movies, and they are both still there. I then went to the hard drive using Explorer and manually deleted it. I then emptied the Recycle Bin. I then went back in to Media Center/Movies and guess what? They are both still listed. So I thought perhaps the index was not up to date and it required time. I then clicked on each of them, and THEY BOTH PLAYED. Both recordings. They were different at the start, one had a piece of commercial in it and the other did not. I then went to the Movies drive with a DOS prompt and hit Attrib thinking that perhaps the file was hidden. Nothing was showing. I have no idea how it could play a manually deleted file regardless of whether it's in the index or not. I looked for that file everywhere - it did not exist.
d00zah wrote:Out of curiosity, do your libraries include any metadata (i.e.YAMMM, etc.)? Even with 8GB RAM/quad core i5, adding new content was ponderous, as Marvin described. With library management offloaded, I find the experience MUCH improved & intend to stick w/ WMC7/Emby indefinitely. YMMV
I don't know what YAMMM is but all the files are natively recorded by the DVR in H.264/WTV files so they contain full .WTV metadata.

I'm going to try one more thing just to be sure. I've removed all Libraries from Media Center (again) so as far as it knows, all it should be watching it the Recorded TV folder. I'll re-start the system again and then watch for excess memory usage. I'm not sure if I've done that or not but I would like to nail it down as well as it should remove indexing by MCE from the equation entirely (I think!).

But I'm pretty certain I will be taking the Emby path....as I really think this is a Media Center design issue. It would just be nice to nail down what part of it is responsible.

User avatar
d00zah

Posts: 242
Joined: Fri Nov 07, 2014 7:20 pm
Location:

HTPC Specs: Show details

#11

Post by d00zah » Fri Dec 02, 2016 11:13 pm

YAMMM (Yet Another Media Meta Manager) is/was exclusive to movies... Michael Welter (sp?) wrote it as a plug-in to WMC to download description/cover art for the movie library. He always 'threatened' TV functionality, but it don't recall it ever materializing.

marvin-miller

Posts: 221
Joined: Tue Sep 18, 2012 9:50 pm
Location:

HTPC Specs: Show details

#12

Post by marvin-miller » Fri Dec 02, 2016 11:20 pm

Na, don't have it. In fact, I have almost nothing installed as I stripped even the Logo grabber program from it. It's MCE and EPG123 and the Collossus - that's about it. I even took out PlayReady.

I just finished removing the Movies and Series libraries from Media Center and then re-started. Without even going into Media Center the memory usage is still very high. I have no clue as to what's going on anymore :wtf:

Perhaps disable Media Center service and then re-start and see if it occurs again. If so, then it should be isolated from Media Center?

marvin-miller

Posts: 221
Joined: Tue Sep 18, 2012 9:50 pm
Location:

HTPC Specs: Show details

#13

Post by marvin-miller » Sat Dec 03, 2016 12:16 am

The plot thickens.....

With Media Center Scheduler Service and Receiver Service set to disabled, re-starting the computer, and then letting it sit for 10 or 15 minutes, it starts eating memory again. It quickly goes up to 6 or 7 gigs. Typical consumption is less then 2.

My guess is that shutting down those two services kills Media Center. If correct, I wonder if I'm seeing two different issues.....

User avatar
Crash2009

Posts: 4357
Joined: Thu May 17, 2012 12:38 am
Location: Ann Arbor, Michigan

HTPC Specs: Show details

#14

Post by Crash2009 » Sat Dec 03, 2016 7:19 am

marvin-miller wrote:
stuartm wrote:I do think it depends on how the data is organized. On my Series drive every Series is in it's own folder. So when I click on Videos I get a list of about 30 folders and then drill down into the one I want. IE, Go into the NCIS folder and then all the NCIS episodes show up. I never seen to have delays in the Series section (Videos) in Media Center. That always responds well unless it has to generate 100 icons or something, but that's to be expected.
Mine is setup differently. Everything is recorded in the Recorded Tv folder. Just one big lump of recordings. Series are not moved and placed in a folder of their own....I let WMC organize everything. Movies are moved automatically with MoveMovies and a folder is created for each movie. YAMM scans the new movie folder and places the MetaData in the folder.

When I open WMC and go to the Movie Section, all the movies have their Cover Poster and synopses.

When I open Recorded TV, I use the built in organizer. Click on Title and the Series appear with the number of episodes.

I learned long ago that WMC has great organizational skills....Where she has trouble is when we start organizing things for her. Movies should be the only types of recordings placed in their own folder.

http://www.thegreenbutton.tv/forums/vie ... 184#p39184

marvin-miller

Posts: 221
Joined: Tue Sep 18, 2012 9:50 pm
Location:

HTPC Specs: Show details

#15

Post by marvin-miller » Sat Dec 03, 2016 8:02 am

In my case Recorded TV is temporary storage (usually only one file) and then the commercials get cut and a new file, with a new name (airdate & Epdisode.wtv) then gets placed in the proper series folder. If it's a Movie, the commercials get cut and it gets renamed title - airdate and then gets dumped on the Movies drive. It's done using VAP which calls Video Redo and makes use of Comskip. It's very tidy and very fast.

I just wiped the hard drive and re-installed everything. So far I'm seeing some spikes in memory but it's too early to say for sure. I have a feeling hardware will be needed to really case this one....

Perhaps a dedicated SSD for the eHome diretory, certainly one for the boot drive and perhaps one more for the Recorded TV folder. Add in 16 gigs of memory (to start) and maybe that will help the old girl out :)

User avatar
DavidinCT

Posts: 1556
Joined: Mon Feb 13, 2012 3:45 pm
Location:

HTPC Specs: Show details

#16

Post by DavidinCT » Sat Dec 03, 2016 5:53 pm

Tweak the Indexer for Media Center. If this is a dedicated Media Center machine. Change the index to only Windows folder and all your media Center folders (Windows\ehome, Programdata\Microsoft\ehome, Movie/tv folders, etc). Just take a screen shot BEFORE so you can re add things if you run into issues later.

Then go Advanced and select "delete and rebuild index" this will remove it and start all over, If you go back to Indexing options, it will show it's status right at the top (and will say Indexing complete when finished)

After complete, reboot the computer and try again. See if this comes back. I remember one of my past Windows 7 HTPC WMC machine was dealing with the same issue, after tweaking indexing, memory usage dropped BIG TIME.

Can never forget when Microsoft added "Search" to WIndows XP.... It took so much resources the computer was almost unusable when indexing.

Edit: For the HECK of it, I rebuilt my Index after posting this, and while I was doing it, I had the task manager up and watched the CPU and Memory SPIKE while it was indexing....dam..
-Dave
Twitter @TheCoolDave

Windows Media Center certified and WMC MVP 2010 - 2012

marvin-miller

Posts: 221
Joined: Tue Sep 18, 2012 9:50 pm
Location:

HTPC Specs: Show details

#17

Post by marvin-miller » Sat Dec 03, 2016 9:24 pm

Been there done that :( I spent a lot of time on the index yesterday and came to the conclusion that wasn't the culprit.

I formatted the drive last night and re-installed the basics. It's too early to say because some of the Windows Updates are still coming in. But it's back up and running and it's fresh. Of course, I lost my recording history but....it's fresh. I'll know as soon as the updates are all in and it's settled down if it made any improvement.

The system was very clean prior to the wipe....

marvin-miller

Posts: 221
Joined: Tue Sep 18, 2012 9:50 pm
Location:

HTPC Specs: Show details

#18

Post by marvin-miller » Sun Dec 04, 2016 3:10 am

You know what I'm finding really weird? I can't seem to find anyone or any posts on the 'net that relate to actually tracking down where memory is going. I'm seeing tons of posts like mine which say words to the effect of;

My system is using 90% of the available physical memory - but nothing odd is showing up in Task/Process Manager.

Without exception every response seems to be determine the PID of the offending process (dependant on being able to see it in process monitor, which I can't) and then use tools like RAMMap or similar to see what is using the memory.

When I look at the Processes tab in Task Manager the highest one is something like 300 megs of memory with all the rest being much smaller. Yet 82% to 90% of the 8GB of Physical Memory is use.

When I go to file summary in Rammap I can see that something is accessing the series episodes and that each one is taking 192k of memory - and there is a lot of them. I guess I'm just bitching but it sure would be nice to see a process, any process, on the Processes tab of Task Manager showing something to the extent that it's using 3 GB of memory. At least then I could definitively pin it down to exactly where the memory is going.

I can see that about 300 megs is being used by the Windows update dB file. This is because I'm still waiting for it to determine all the updates required.

The good news (right now) is that even with 82% Physical Memory usage (6.57 of 8GB) it's not paging. What I'm seeing is that as work load increases memory is being released - which is as it should be.

Hopefully when all the updates are in it will still act that way....

marvin-miller

Posts: 221
Joined: Tue Sep 18, 2012 9:50 pm
Location:

HTPC Specs: Show details

#19

Post by marvin-miller » Mon Dec 05, 2016 11:34 pm

Just an update in the event that others run into this same issue....

All the updates are now installed so it's fully patched and I've been able to observe it's behavior. With the Windows Search service disabled, and the machine re-started, it will still use up to 92% of the available Physical Memory.

As for what is using that memory, nothing will show it. The memory will be released after some time. It is related to Media Center. At one point, the system was using all the memory and I forced a dismount of the Series disk (for a chkdsk). As soon as that disk was forcibly dismounted the memory usage dropped to nearly nothing. So this was indicative of an issue. This was with the Windows Search service disabled and not running.

My conclusion is that it's Media Center that's using the memory. Not the Windows Search service. As to why it does not show up in any monitoring/diagnostic software, I couldn't say.

The best way (that I've found) to see what Media Center is really doing is to watch the disk access on the system. When I go into the Series drive you can see the wmplayer.exe service looking at each of the files. This appears to be due to Media Center generating thumbnails. Once this is done I then see the same behavior from a System thread. I do not know what that one is doing.

Surprisingly, I don't see that behavior on the Movies disk. I think the reason for the pause when selecting the Movies disk is because the system is loading up a dB of the movies which is quite large.

My conclusion is that there is a problem with Media Center in that it will not report how much memory it is using. This makes diagnostics very hard. You have to infer what it's doing by watching the disk access. Not knowing the underlying design of Media Center also makes this difficult as you have to infer what it's doing.

But....I'm pretty sure that most all of these issues can be resolved by adding hardware in a smart manner. Change the boot drive over to an SSD will help immensely when it's paging to disk. Put the pagefile on another SSD will make it even better. Keep the eHome directory on it's own SSD. In short, separate as much of it as possible onto separate SSD's. Double the installed memory to 16 GB.

Pretty sure that will case it completely and make it fly. It's a work-around but without being able to see specifically where the memory is going, that's all a guy can do. I think :lolno: Anyway, I did put eHome on it's own disk (platter) and that has helped. Looks like it needs more money thrown at it.

User avatar
Crash2009

Posts: 4357
Joined: Thu May 17, 2012 12:38 am
Location: Ann Arbor, Michigan

HTPC Specs: Show details

#20

Post by Crash2009 » Thu Dec 08, 2016 1:29 am

Where is the recorded TV folder? Try setting it to the series drive.

Post Reply