Utilizing your Processor

[Mindweaver]

I've been working on a theory of mine after hearing a big question from WCG members, "How can I run more work units?". In essence one WU for each core.. Correct?... No... for example HT (Hyper-Threading) which consists of Virtual cores.

Now the bigger question, "what if my processor doesn't support HT? ". Now back to my theory.. and that is, "Run a virtual pc/server ." I started testing this on 6/25/09 with my E8400 @ 4ghz and the results are nice.

Don't get to excited each virtual pc/server only represents one core, but with a capable processor you can run more virtual pc/servers. So, with the fundamental nature of a processor with multiple cores you can run as many virtual pc/servers as your core number with small to medium impact." i7" users should be able to use HT cores which have the newest more efficient HT.

Now, let's see if your processor qualifies? Intel users need to have VT (Virtualization Technology) and AMD users need to have AMD-V. If you do, then verify that it is turned on in bio's. And remember we all know more RAM the better.

Next what you will need is software to run your virtual PC/server (example: Windows Virtual PC, Windows Virtual 2006/2007, and VMWare.). Obviously, Windows Virtual PC only runs Windows OS's, but there are other software like VMWare which run Windows, Mac, and Linux. You can find others just use google or some other search engine to find them.

Well, what are you waiting for go create your virtual server! Hehehe... I've listed some results from my E8400 with VT below. Just realize from the results below the PC it self ran longer than the Virtual server. After doing a little math you can see the numbers are close give or take 200 or 300 points. The Virtual server returns around 420 to 460 points every 2 to 2.5 hours @ 60% CPU.

E8400 (60% to 95% CPU)

Total Run Time
Statistics Date (y:d:h:m:s) Points Generated Results Returned
6/28/2009 0:000:06:02:52 1,376 2
6/27/2009 0:001:10:11:17 6,326 12
6/26/2009 0:001:04:05:47 5,623 9
6/25/2009 0:001:00:12:57 4,808 8

Virtual PC/server (60%)

Total Run Time
Statistics Date (y:d:h:m:s) Points Generated Results Returned
6/28/2009 0:000:04:08:47 648 1
6/27/2009 0:000:06:23:32 1,009 2
6/26/2009 0:000:10:05:34 1,738 4
6/25/2009 0:000:07:23:18 1,260 3

I'm curious to see other members results! Please post. thanks!

 

[PaulieG]

This is a great idea, though I've never set up a virtual server before, and have no idea how to do it...

 

[Mindweaver]

Simple, if you have ever formated your hard drive and re-installed your OS your good to go. Just down load the software (MS Virtual PC is free) and install. Then just start the software and create new virtual PC/server. Remember it will take up space on your hard drive. I would suggest creating them no bigger than 4.7gb's with Windows virtual PC. So, you can back them up on a standard dvd.

If you use VMWare you can go bigger, because it creates them in 2gb blocks for backing up and moving.

 

[FordGT90Concept]

I don't get it. Virtual PCs allows you to run more instances of the application but it also adds a lot of overhead. Are you not better off just running the two instances your processor supports and eliminating the overhead?

Yeah, a Virtual PC means more work units but at the end of the day, it doesn't necessarily mean more work done. Remember, there is about a week-long deadline on all work so it is a good idea to get it done almost as fast as you receive it.

 

[Mindweaver]

I don't get it. Virtual PCs allows you to run more instances of the application but it also adds a lot of overhead. Are you not better off just running the two instances your processor supports and eliminating the overhead?

Yeah, a Virtual PC means more work units but at the end of the day, it doesn't necessarily mean more work done. Remember, there is about a week-long deadline on all work so it is a good idea to get it done almost as fast as you receive it.

Very good question. That can be said about i7's with HT enabled, but they are returning work just fine... My E8400 has been returning work just fine.. Almost as if it were a triple core. If you bought a processor with VT or AMD-V than why not use it? So far, my Virtual machine is out performing my single core processors.

It's returning almost 2000 points (4 results) every 10 hours @ 60% CPU usage with out taking anything away from both cores.

 

[HammerON]

This sounds interesting. I have a E8500 and was wondering how I could get more out of it for crunching

 

[FordGT90Concept]

VT and AMD-V are simply instructions to handle virtualization on a lower, machine level rather than being purely software. It boosts virtualization performance (by completely virtualizing the OS) but it is still no faster than just using the core directly.

I see what you are aiming for--HT on non-HT processors but, Hyperthreading is an architectual feature. Netburst-based processors have long pipes compared to non-Netburst processors. Because some stages of a pipe can be used while others are busy, the route that extra work in through a logical core. At least that was the implementation in Pentium 4. In Core i7, the line between physical and logical is pretty much gone. Every core is fully capable of handling two threads and which thread ends up where really makes no difference. I mean, when my Core i7 was reset, all 8 work units finished up at about the same time demonstrating consistency across the physical/logical cores. I'm going on a tangent here...

The point mean is that if you try to run two threads on the same core at the same time when the core can only handle one thread at a time does not provide any benefit. In your example, 60-95% on your dual core means 100% of core 0 and 20%-90% of core 1. 60% on your virtual pc means 60% of core1 which adjusts. Your virtual pc host software will mask the other 20%-90% that's already there and WCG will respond to the 60% workload already present meaning it will only use 0-30%. In the end, it looks something like this...
core 0: 100% WCG
core 1: 60% Virtual PC, 0-30% WCG

There's a net loss here because of the virtual PC overhead. Virtualization doesn't mean the core can handle more than one thread a time so there is nothing to be gained from it.


Is this computer left on 24/7? The best way to test is to take a three, five, or seven day average with the virtual machine running and do another three, five, or seven day average with the virtual machine disabled. Compare the results and see what averaged better. I also recommend turning it up to 100% for this test to prevent throttling (which is never good).

 

[Mindweaver]

Wow I was going to bed ford.. You make a very good point. I understand the concept of HT versus VT. I was just pointing out that they both are virtual tech.

I've listed the results from my E8400 @ 4ghz with cpu 60 to 95%. From the 25th till now has the Virtual server running on it. Everything before is just the 2 cores returning work.

Total Run Time
Statistics Date (y:d:h:m:s) Points Generated Results Returned
6/28/2009 0:000:06:02:52 1,376 2
6/27/2009 0:001:10:11:17 6,326 12
6/26/2009 0:001:04:05:47 5,623 9
6/25/2009 0:001:00:12:57 4,808 8
6/24/2009 0:000:12:40:41 2,263 4
6/23/2009 0:000:20:00:57 3,906 7
6/22/2009 0:001:01:57:47 4,858 7
6/21/2009 0:000:14:30:39 2,787 4
6/20/2009 0:001:09:52:45 7,041 10
6/19/2009 0:001:03:14:42 5,549 7
6/17/2009 0:000:02:56:47 578 1
6/16/2009 0:000:12:54:08 2,446 5
6/15/2009 0:000:11:02:40 2,026 4
6/14/2009 0:001:00:21:30 4,740 7

See first post for Virtual Results.

 

[Mindweaver]

Don't count the 28th's results yet.. After the last update it had 8 results finished but not returned.. since then they have returned... I'll post results tomorrow. Nite all!

 

[Mindweaver]

Remember, the result numbers and points need to be divided in half to compare with the Virtual PC. Now I'm going to bed..lol

 

[FordGT90Concept]

Remember, the result numbers and points need to be divided in half to compare with the Virtual PC. Now I'm going to bed..lol

Huh? If these are device statistics, why would you have to divide it in half? The Virtual PC inflates the runtime value, is that what you mean?

What I am looking at now is not consistent enough to be scientific. Run times are all over the place yet, the points rewarded are relatively constant with the run times throughout. Without knowing exactly how long the computer was running WCG projects on those days, it's hard to be certain of anything.

Even my server which has been running 24/7 for the past 19 days is all over the chart with runtimes...

Statistics Date | Total Run Time | Points Generated | Results Returned
6/28/09 | 0:007:04:02:59 | 14,300 | 27
6/27/09 | 0:006:17:34:44 | 12,526 | 25
6/26/09 | 0:009:10:53:20 | 18,185 | 36
6/25/09 | 0:007:23:39:32 | 15,386 | 29
6/24/09 | 0:009:01:55:01 | 17,319 | 33
6/23/09 | 0:008:00:55:06 | 14,831 | 29
6/22/09 | 0:008:08:39:18 | 16,303 | 29
6/21/09 | 0:006:16:05:54 | 12,675 | 25
6/20/09 | 0:007:10:42:15 | 14,590 | 28
6/19/09 | 0:007:20:26:32 | 14,722 | 30
6/18/09 | 0:007:19:17:17 | 15,176 | 27
6/17/09 | 0:007:09:12:30 | 14,946 | 27
6/16/09 | 0:010:00:49:31 | 18,483 | 37
6/15/09 | 0:007:07:49:08 | 13,821 | 24
6/14/09 | 0:008:22:02:44 | 17,122 | 31
6/13/09 | 0:008:21:24:51 | 16,884 | 32
6/12/09 | 0:005:12:16:36 | 10,452 | 19
6/11/09 | 0:008:03:46:14 | 15,137 | 27
6/10/09 | 0:009:06:36:02 | 17,025 | 28

It should be 8 days on average because it is 8 cores running all day every day.


But, let's just say (rather, stating a fact) were not going to get anything more consistent just because we are dealing with non-uniform workloads (not to mention points are rewarded on the day the work is verified good and not when it is turned in). That said, we look at the 27/26th compared to the 20/19th. The runtimes are similar albeit slightly in favor of the virtual PC and the subesequent scores are very close. Going just by those data points, I'd say it isn't making any difference.


With it set at 60%, it is virtually impossible to draw any conclusions.

 

[Mindweaver]

Huh? If these are device statistics, why would you have to divide it in half? The Virtual PC inflates the runtime value, is that what you mean?

The last table of results are from the non-virtual environment which has two cores. Since it has two cores you would divide the results in half to receive a number for each core then compare with the virtual numbers. I realize that each core works on it on result.. so if you have a dual core and the results are odd... just round up or round down.. I would round up and then divide by two.

 

[FordGT90Concept]

Per core really doesn't matter. The important thing to compare is the machine output. All virtual machine does is make it work on three at a time instead of two.

HT isn't virtual technology, it is a second physical pipe to the core (can process two threads simutaneously instead of one).

 

[Mindweaver]

Per core really doesn't matter. The important thing to compare is the machine output. All virtual machine does is make it work on three at a time instead of two.

HT isn't virtual technology, it is a second physical pipe to the core (can process two threads simutaneously instead of one).

So, everything that you have said so far you should turn off your HT. The fact is that your i7 only has 4 physical cores no matter how many extra pipes it has going to it. By adding the extra pipe to each core enables OS to think it has 8 logical cores. This means that only Four processor is physically present but the operating system sees eight virtual processors, and shares the workload between them.

Intel only claims 15 to 30% increase in performance using HT. And yes your i7 creates overhead with HT enabled. With the Virtual PC I never said this would speed up your processor or double your output, but i have seen an increase in results returned.

In the end it's all about the number of results that find a cure correct? So far, I can see the increase of results returned from my E8400. Your one concern was the virtual pc returning work in time. As of right now it does not have any errors or work pushed aside, because it did not reach them in time.

Please do one favor for me before you post your rebuttal. Try it... Stop reading other peoples work, and test it out for yourself. I was sceptical at first too.. but I tried it. Ford I would really like to see some results from your i7 if possible. I work with virtual OS's day in and day out. I think it's a fantastic piece of software that's able to take advantage of VT, and AMD-V technology.

I'm just trying to find a cure.... Will you help?

 

[FordGT90Concept]

I guess I'm going to have to paint a picture....

The only way to tap 100% power of a Pentium 4 or Core i7 is to use Hyper-Threading. If you have them disabled, large portions of the core go unused (this is clearly demonstrated by how much heat Pentium 4 and Core i7 produce when Hyper-Threading is enabled).

Intel only claims 15 to 30% increase in performance using HT. And yes your i7 creates overhead with HT enabled. With the Virtual PC I never said this would speed up your processor or double your output, but i have seen an increase in results returned.

Overhead is near non-existant with Hyper-Threading enabled. The biggest loss was -2.7% in World in Conflict (namely because it is single threaded). Gains are as high as 34.1% in some synthetic benchmarks.

Ford I would really like to see some results from your i7 if possible. I work with virtual OS's day in and day out. I think it's a fantastic piece of software that's able to take advantage of VT, and AMD-V technology.

My Core i7 computer goes off every night so it is not a good for testing throughput; moreover, the WCG client running often has problems (failing to submit completed work, failing to work on existing tasks, failure to download new work).

I am also running Windows XP which doesn't support Intel VT.

 

[Steevo]

Close, but the HT from the netburst days was a poor excuse for cache misses and a long ass pipeline made up to be a great thing. In those days AMD could say real men use cores.


However it did improve performance in multitasking with things like F@H. But fell flat when using it for other things that required CPU cycles and real time data, basicly single threaded apps took a fall.

 

[Mindweaver]

I guess I'm going to have to paint a picture....
http://img.techpowerup.org/090630/threading444.png

The only way to tap 100% power of a Pentium 4 or Core i7 is to use Hyper-Threading. If you have them disabled, large portions of the core go unused (this is clearly demonstrated by how much heat Pentium 4 and Core i7 produce when Hyper-Threading is enabled).



Overhead is near non-existant with Hyper-Threading enabled. The biggest loss was -2.7% in World in Conflict (namely because it is single threaded). Gains are as high as 34.1% in some synthetic benchmarks.



My Core i7 computer goes off every night so it is not a good for testing throughput; moreover, the WCG client running often has problems (failing to submit completed work, failing to work on existing tasks, failure to download new work).

I am also running Windows XP which doesn't support Intel VT.

Why do you insist on telling me the work on one logical core is divided using a virtual PC. I UNDERSTAND THAT.... And you don't have to paint me a picture.. Should I paint you one?... I know HT is more efficient than using VT, but if you don't have it but want to run more work units at a time.. Then using Virtual PC/Server is your best alternative...

I never stated that by running a virtual server it would perform as well as having HT. I said it would give you the ability to run more work units...

Why don't you try putting more effort in returning valid work units than bantering my thread? Good day Sir! I said Good day!...

 

[Mindweaver]

Windows Virtual pc 2007 will run on windows XP. See link below. This version will slow older pc's down some... but nice to have for testing.

System Requirements

* Supported Operating Systems: Windows Server 2003, Standard Edition (32-bit x86); Windows Server 2003, Standard x64 Edition; Windows Vista Business; Windows Vista Business 64-bit edition; Windows Vista Enterprise; Windows Vista Enterprise 64-bit edition; Windows Vista Ultimate; Windows Vista Ultimate 64-bit edition; Windows XP Professional Edition ; Windows XP Professional x64 Edition ; Windows XP Tablet PC Edition

An x64-based or an x86-based computer with a 400 MHz or faster (1 GHz recommended) processor with L2 cache

Processor: AMD Athlon/Duron, Intel Celeron, Intel Pentium II, Intel Pentium III, Intel Pentium 4, Intel Core Duo, and Intel Core2 Duo

RAM: Add the RAM requirement for the host operating system that you will be using to the requirement for the guest operating system that you will be using. If you will be using multiple guest operating systems simultaneously, total the requirements for all the guest operating systems that you need to run simultaneously.

Available disk space: To determine the hard disk space required, add the requirement for each guest operating system that will be installed.

Virtual PC 2007 runs on: Windows Vista™ Business; Windows Vista™ Enterprise; Windows Vista™ Ultimate; Windows Server 2003, Standard Edition; Windows Server 2003, Standard x64 Edition; Windows XP Professional; Windows XP Professional x64 Edition; or Windows XP Tablet PC Edition


http://www.microsoft.com/downloads/details.aspx?FamilyId=04D26402-3199-48A3-AFA2-2DC0B40A73B6&displaylang=en

 

[FordGT90Concept]

I never stated that by running a virtual server it would perform as well as having HT. I said it would give you the ability to run more work units...

I'm stating it makes no difference. There are two huge x-factors in your current testing: WCG isn't running 100% idle (very important) and your day to day runtime isn't constant.

 

[Disparia]

I have no doubt that your output has increased from what I've read. Yyou weren't fully utilizing your CPU from the get-go (60%). You then start a VM with the WGC client, it's now sucking up those spare cycles, no?

But I do applaud you, and anyone who would take the time to research for the benefit of others. Which is why we'd all like to help your research be as scientific and precise as possible.

 

[Mindweaver]

I have no doubt that your output has increased from what I've read. Yyou weren't fully utilizing your CPU from the get-go (60%). You then start a VM with the WGC client, it's now sucking up those spare cycles, no?

But I do applaud you, and anyone who would take the time to research for the benefit of others. Which is why we'd all like to help your research be as scientific and precise as possible.

True, but when the computer is at 60% the VM ware is turned off... The E8400 is my main personal rig... When my nephew comes over to play games I turn off vm, and set wcg to 60%. When ever vm is running the physical pc is set to 95% and vm is set to 60%. I would run vm higher, but I don't want it over stating the physical pc.

Do you think by setting WCG to 100% utilizes your processor 100%? Why does your mouse and keyboard respond? How can you open other apps? Don't tell me WCG backs off when these other task are applied.. If so, why have an option to turn off or back off when the user is present? When your pc becomes unresponsive and the little red light on the front of your case is solid, that is when your pc is being taxed at 100%.

Given the nature of Window's, the kernel only hands out 85% of your processor and holds back 15% for other duties when windows shows performance at 100%. How is your pc responsive? That's the kernel doing its job. And I'm not stating that Window's kernel is 100% efficient.. But, I still think we can harness that 15%. I'm just trying to manipulate it.

Why put better compound between your heatsink and processor?.. Not for it to do all of your cooling but to shave a little off... If you didn't buy that better compound then your most likely not going to try this.. For me I did! Thanks Jizzler for your response! I like your approach! Wow, my 2.734 posts a day is turning into 3.021...hehehe

 

[Mindweaver]

I'm stating it makes no difference. There are two huge x-factors in your current testing: WCG isn't running 100% idle (very important) and your day to day runtime isn't constant.

Non sequitur... Carpe diem!

 

[Disparia]

I think I have a clearer picture of what you're trying to accomplish.

My first thought would be to work on a little scheduled script to increase priority of the unit processes (low by default), for a longer time slice and less interrupts. It's first as I'm trying to think of solutions that don't involve a VM (overhead) and the maintenance of said VM. It's great for Folding as the Linux client stomps all over the Windows client, so overhead be damned, but in this case... having a hard time seeing a gain from it.

One of my Q6600's isn't being used by anyone, so that seems like the best place for some long-term testing without the user variable.

 

[Mindweaver]

I think I have a clearer picture of what you're trying to accomplish.

My first thought would be to work on a little scheduled script to increase priority of the unit processes (low by default), for a longer time slice and less interrupts. It's first as I'm trying to think of solutions that don't involve a VM (overhead) and the maintenance of said VM. It's great for Folding as the Linux client stomps all over the Windows client, so overhead be damned, but in this case... having a hard time seeing a gain from it.

One of my Q6600's isn't being used by anyone, so that seems like the best place for some long-term testing without the user variable.

Great Idea! Script or bat file will work great. VM does have overhead but on a dedicated cruncher should not be as big a factor. As for maintenance just install the os and wcg and keep it under a single layer DVD for backing up. If something goes wrong with VM just delete the Virtual Harddrive and restore it from the DVD.

Yes, linux is great.. I first started using Unix years ago which later became linux. I still have the 20 book shelf boxes the last version I have of unix. That's when Instruction books where books.. hard back..lol full size..lol One HD3.5 per box..lol The last system it ran on was a RS6000 Computer.. Not Mirco-computer.. hehehe... with over 60 terminals pc's...

I'm getting of topic... Sorry... stealing my on thread...

 

[FordGT90Concept]

Do you think by setting WCG to 100% utilizes your processor 100%? Why does your mouse and keyboard respond? How can you open other apps? Don't tell me WCG backs off when these other task are applied.

This is hard to explain...so another picture is in order!

I play games with WCG running 100% all the time. WCG runs at low priority which means it only uses up to 100% of idle clocks. For instance, most games are single-threaded which means they only use 13.5% of the CPU. WCG, therefore, will use the other 86.5%. I can't tell it is working unless I open up task manager.

Video cards aren't so good with thread priority...


This is a huge discussion topic, really. Your PC will come to a crawl if the the threads that are using 100% of your processor power are at normal or above. Why does it cause problems? Because the OS runs at normal. Problems start arising when the OS gets bumped. Idle/Low priority means WCG can't cause that. The OS will beat WCG over the head if it tries. XD

Non sequitur... Carpe diem!

That's...not english.