techPowerUp! Forums

Go Back   techPowerUp! Forums > Other > World Community Grid (WCG)

Reply
 
Thread Tools
Old Nov 30, 2011, 10:19 PM   #1
BinaryMage
500 Posts
 
BinaryMage's Avatar
 
Join Date: Feb 2011
Posts: 606 (0.72/day)
Thanks: 850
Thanked 457 Times in 266 Posts

System Specs

WCG GPU Client Discussion Thread

Announced last September, WCG is planning to release a GPU application, initially for the Help Conquer Cancer project.

Quote from forums:

Quote:
Hello WCGrid members,

The September 2011 update to the Help Conquer Cancer project has been posted. In this update, we announce work presented earlier this year at the High-Performance Computing Symposium in Montreal: the development of an OpenCL (GPU) implementation of HCC. This GPU version is currently running in our lab. We are pleased with its performance, and anticipate its eventual launch on the World Community Grid.

Please be patient in the meantime. It will take time. A large-scale launch on a new hardware platform is tricky business for the World Community Grid staff, especially while juggling every other active project and new projects emerging from the pipeline.

Thank you to everyone for your enthusiasm and interest in HCC, and your contribution of CPU cycles that make this and other projects possible.

Christian A. Cumbaa
Research Associate, Ontario Cancer Institute
University Health Network
Timing-wise (thank you KieX) a beta period is slated to start sometime in January 2012. (If you want to participate, you'll need to opt-in on the WCG website)

If this goes through as planned, our whole crunching strategy will likely change. GPUs are far, far more efficient than CPUs, FLOPS-wise. (Floating Point Operations per Second) For example, an overclocked i7 2600k, costing about $310, puts out about 14 GFLOPS. (reference) A videocard at the same price point, the HD 6970, puts out 2700 GFLOPS - 192 times what the i7 does. Now, GFLOPS do not directly translate into BOINC crunching power - but they are related. An i7 puts out maybe 5,000 PPD on an average BOINC project. The 6970 will do 250,000 PPD - a 50-fold increase. (source) Now, these GPU credit numbers are not from WCG, and the actual numbers from WCG GPU crunching will probably differ slightly. But nonetheless, crunching with a videocard instead of a CPU will likely be more efficient by many orders of magnitude. The official supported list of cards is here, but any AMD card supporting OpenCL or nVidia card supporting CUDA should work.

Now, many of you are probably frustrated right now. Your 2600k you just bought for crunching - you now think was maybe not the best investment. I like to think of it like this. We crunch to help people. Not to win contests. Not to accumulate points. Your i7 isn't going to do any less work. It's not going to help people any less. But hopefully using videocards in the future will allow us to help people more. If the GPU application is, say, 40 times more efficient than the CPU one (in tasks finished/day at a certain price point), then that means, if we switch to buying GPUs instead, the project can obtain hopefully useful data much, much faster than it could otherwise. More efficient ways to help people? Sounds good to me, and that's the bottom line.
__________________
Donate your idle CPU/GPU cycles and help save lives - crunch and fold for TechPowerUp!


Last edited by BinaryMage; Dec 1, 2011 at 12:30 AM.
BinaryMage is offline  
Reply With Quote
The Following 8 Users Say Thank You to BinaryMage For This Useful Post:
Old Nov 30, 2011, 10:28 PM   #2
(FIH) The Don
Eligible for custom title
 
(FIH) The Don's Avatar
 
Join Date: Aug 2009
Location: Viking Land
Posts: 5,406 (3.91/day)
Thanks: 3,101
Thanked 2,962 Times in 1,834 Posts
Send a message via MSN to (FIH) The Don Send a message via Skype™ to (FIH) The Don

System Specs

this will be VERY interesting, may even make me invest in 2-3 more 5870s just for this, as they are dirt cheap now and will get cheaper
__________________
"there are 2 types of idiots. those who hate the new stuff to justify not being able to upgrade. and those who have no fucking clue about anything" -KieX
“So with this new technology my video card will now fart?” -TheMailMan78
(FIH) The Don is offline  
Reply With Quote
Old Nov 30, 2011, 10:31 PM   #3
Bow
1000 Posts
 
Bow's Avatar
 
Join Date: Jul 2008
Location: South Colton, New York
Posts: 1,653 (0.94/day)
Thanks: 755
Thanked 664 Times in 401 Posts

System Specs

Sweet put my 6950's to work
__________________
"I didn't want my Mom to know I was drag racing for 20 years so I told her I was in prison."
Bow is offline  
Crunching for Team TPU
Reply With Quote
Old Nov 30, 2011, 10:34 PM   #4
f22a4bandit
200 Posts
 
f22a4bandit's Avatar
 
Join Date: Sep 2010
Location: SATX
Posts: 462 (0.47/day)
Thanks: 210
Thanked 200 Times in 115 Posts
Send a message via AIM to f22a4bandit

System Specs

This is awesome news! I wonder how points would scale in crossfire/SLI?
__________________


HEATWARE
f22a4bandit is offline  
Reply With Quote
Old Nov 30, 2011, 10:35 PM   #5
(FIH) The Don
Eligible for custom title
 
(FIH) The Don's Avatar
 
Join Date: Aug 2009
Location: Viking Land
Posts: 5,406 (3.91/day)
Thanks: 3,101
Thanked 2,962 Times in 1,834 Posts
Send a message via MSN to (FIH) The Don Send a message via Skype™ to (FIH) The Don

System Specs

im thinking it will be like folding, where sli/cf has to be disabled
__________________
"there are 2 types of idiots. those who hate the new stuff to justify not being able to upgrade. and those who have no fucking clue about anything" -KieX
“So with this new technology my video card will now fart?” -TheMailMan78
(FIH) The Don is offline  
Reply With Quote
The Following 2 Users Say Thank You to (FIH) The Don For This Useful Post:
Old Nov 30, 2011, 10:40 PM   #6
KieX
2000 Posts
 
Join Date: Dec 2009
Location: A Coruña, Spain
Posts: 2,050 (1.64/day)
Thanks: 6,950
Thanked 2,643 Times in 1,136 Posts
Send a message via Skype™ to KieX

System Specs

Thanks for putting up the thread BinaryMage. Might wanna add the links from the Daily Numbers thread to your OP to keep it all in one place.

As far as information on start dates:
Quote:
At this time we are not prepared to give you a date for the start of GPU for HCC but we can tell you this:

1. At this time, the plan is to start Beta testing before the end of January, 2012. This may change as conditions arise.

2. Because of the large variety of processors, we will probably have a longer than average Beta test.
Source

Looks like it's still some time away from full implementation for end users. Anyone willing to participate in the BETA, will need to remember to opt-in for it on the WCG website.

Now.. the scoring system.. that's going to be interesting. According to WCG the current points are based on a 1 GigaFLOP machine, running full time, producing 100 units of credit in 1 day (~14Boinc points)

That would certainly entail a massive PPD difference over anything a CPU can produce (GPU vs CPU GFlops). And I would not imagine they'll want to go the route of F@H and start differentiating points

A lot of informations still to come, but looks very promising.

http://boinc.berkeley.edu/flops.php
__________________
KieX is offline  
Reply With Quote
The Following 3 Users Say Thank You to KieX For This Useful Post:
Old Nov 30, 2011, 10:43 PM   #7
f22a4bandit
200 Posts
 
f22a4bandit's Avatar
 
Join Date: Sep 2010
Location: SATX
Posts: 462 (0.47/day)
Thanks: 210
Thanked 200 Times in 115 Posts
Send a message via AIM to f22a4bandit

System Specs

Besides the PPD output (speculation at this point) another great thing about using video cards is their universality. You don't have to worry about new sockets, it's just plug-and-play. I know our video cards aren't stressed that much anyway unless we're gaming, editing, etc., so this just makes using your machine that much easier.
__________________


HEATWARE
f22a4bandit is offline  
Reply With Quote
Old Nov 30, 2011, 10:49 PM   #8
Chicken Patty
WCG Team Captain
 
Chicken Patty's Avatar
 
Join Date: Nov 2007
Location: Miami, Florida
Posts: 23,895 (11.92/day)
Thanks: 15,156
Thanked 12,315 Times in 7,779 Posts
Send a message via AIM to Chicken Patty Send a message via MSN to Chicken Patty

System Specs

F22 has a very good point. A lot easier to stack up points with the GPU's than it is with CPU's.
__________________
Help save lives, CRUNCH and/or FOLD for Team TPU and make a difference!
Crunching with a 2600K, i7-950, FX-8120, Radeon 6850 (2 WU's), Radeon 7770 (4 WU's)

"AlienIsGOD - Awe!!! CP has grown from a small chicken nugget to the IMMENSE PATTY and WCG Team leader he is today"
Chicken Patty is offline  
Crunching for Team TPU
Reply With Quote
Old Nov 30, 2011, 10:56 PM   #9
F150_Raptor
200 Posts
 
F150_Raptor's Avatar
 
Join Date: May 2011
Location: Connecticut
Posts: 345 (0.46/day)
Thanks: 1,217
Thanked 727 Times in 232 Posts

Your also going to have a hell of alot more heat and a higher power bill. Both of those were the reasons why I stopped using gpu's with F@H.
F150_Raptor is offline  
More than 25k PPD
Reply With Quote
The Following 4 Users Say Thank You to F150_Raptor For This Useful Post:
Old Nov 30, 2011, 10:58 PM   #10
Chicken Patty
WCG Team Captain
 
Chicken Patty's Avatar
 
Join Date: Nov 2007
Location: Miami, Florida
Posts: 23,895 (11.92/day)
Thanks: 15,156
Thanked 12,315 Times in 7,779 Posts
Send a message via AIM to Chicken Patty Send a message via MSN to Chicken Patty

System Specs

Quote:
Originally Posted by F150_Raptor View Post
Your also going to have a hell of alot more heat and a higher power bill. Both of those were the reasons why I stopped using gpu's with F@H.
That is very true. But I already fold as it is so for me it's not much of a big deal. Plus I have a fixed rate monthly. So unless it's a huge jump I don't think they'll tell me anything. Just bad for those who will get the big hit.
__________________
Help save lives, CRUNCH and/or FOLD for Team TPU and make a difference!
Crunching with a 2600K, i7-950, FX-8120, Radeon 6850 (2 WU's), Radeon 7770 (4 WU's)

"AlienIsGOD - Awe!!! CP has grown from a small chicken nugget to the IMMENSE PATTY and WCG Team leader he is today"
Chicken Patty is offline  
Crunching for Team TPU
Reply With Quote
Old Nov 30, 2011, 10:59 PM   #11
KieX
2000 Posts
 
Join Date: Dec 2009
Location: A Coruña, Spain
Posts: 2,050 (1.64/day)
Thanks: 6,950
Thanked 2,643 Times in 1,136 Posts
Send a message via Skype™ to KieX

System Specs

Quote:
Originally Posted by Chicken Patty View Post
F22 has a very good point. A lot easier to stack up points with the GPU's than it is with CPU's.
If the points are calculated the same way for GPU then yes.

At the moment it's only a small part of one project which will have support. Might be a lottery for the first Work Units. I'm feeling lukcy
__________________
KieX is offline  
Reply With Quote
The Following User Says Thank You to KieX For This Useful Post:
Old Nov 30, 2011, 11:00 PM   #12
f22a4bandit
200 Posts
 
f22a4bandit's Avatar
 
Join Date: Sep 2010
Location: SATX
Posts: 462 (0.47/day)
Thanks: 210
Thanked 200 Times in 115 Posts
Send a message via AIM to f22a4bandit

System Specs

Quote:
Originally Posted by F150_Raptor View Post
Your also going to have a hell of alot more heat and a higher power bill. Both of those were the reasons why I stopped using gpu's with F@H.
Great point, I don't know why I even thought of that! But for those like CP with a fixed electricity bill, it is a great advantage.
__________________


HEATWARE
f22a4bandit is offline  
Reply With Quote
Old Nov 30, 2011, 11:05 PM   #13
KieX
2000 Posts
 
Join Date: Dec 2009
Location: A Coruña, Spain
Posts: 2,050 (1.64/day)
Thanks: 6,950
Thanked 2,643 Times in 1,136 Posts
Send a message via Skype™ to KieX

System Specs

Quote:
Originally Posted by F150_Raptor View Post
Your also going to have a hell of alot more heat and a higher power bill. Both of those were the reasons why I stopped using gpu's with F@H.
Although that's also partly due to the bigadv bonus system, right? In F@H until recently an i7 could outdo a couple of GPUs.

I don't think WCG being IBM sponsored want to go down that route of varying bonuses, so it may be that GPU's really are better than CPU for this type of project.

From my GPU folding days I can certainly vouch for 500W+ for dual GPU setups being way less efficient than a good CPU. All speculation until there's any confirmation, of course. But here's to hoping
__________________
KieX is offline  
Reply With Quote
The Following User Says Thank You to KieX For This Useful Post:
Old Dec 1, 2011, 12:24 AM   #14
BinaryMage
500 Posts
 
BinaryMage's Avatar
 
Join Date: Feb 2011
Posts: 606 (0.72/day)
Thanks: 850
Thanked 457 Times in 266 Posts

System Specs

Quote:
Originally Posted by F150_Raptor View Post
Your also going to have a hell of alot more heat and a higher power bill. Both of those were the reasons why I stopped using gpu's with F@H.
True indeed. There are differences, though. First, BOINC assigns credit roughly based on how much data you crunched, while F@H does not necessarily. GPUs are far more efficient in FLOPS/watt than CPUs, so BOINC crunching with GPUs is more efficient in work/watt and credit/watt, while F@H crunching with GPUs is more efficient in work/watt but not credit/watt. In addition, F@H's GPU system is mainly build for CUDA, and doesn't use the more powerful AMD GPUs (speaking in compute power) as effectively as it could.
__________________
Donate your idle CPU/GPU cycles and help save lives - crunch and fold for TechPowerUp!

BinaryMage is offline  
Reply With Quote
The Following User Says Thank You to BinaryMage For This Useful Post:
Old Dec 1, 2011, 01:16 AM   #15
FordGT90Concept
"I go fast!1!11!1!"
 
FordGT90Concept's Avatar
 
Join Date: Oct 2008
Location: IA, USA
Posts: 10,578 (6.28/day)
Thanks: 1,755
Thanked 2,596 Times in 1,960 Posts

System Specs

The reason I switched from F@H to BOINC is because BOINC didn't use GPUs. The reason is simple: CPUs are more accurate. An incorrect value often gets caught immediately in a CPU where a GPU requires testing on a separate machine in order to verify it's correctness and still not with 100% certainty.

We're doing science here: slow, correct results are more desirable than fast, incorrect results.

If TPU starts adding GPU work to its project list, I'll have to stop running BOINC for TPU. I refuse to compromise science for an arbitrary, altogether meaningless number.
__________________
Golden Rule of Programming: Never assume.

try { SteamDownload(); }
catch (Steamception ex) { RageQuit(); }
FordGT90Concept is online now  
Crunching for Team TPU
Reply With Quote
Old Dec 1, 2011, 01:21 AM   #16
BinaryMage
500 Posts
 
BinaryMage's Avatar
 
Join Date: Feb 2011
Posts: 606 (0.72/day)
Thanks: 850
Thanked 457 Times in 266 Posts

System Specs

Quote:
Originally Posted by FordGT90Concept View Post
The reason I switched from F@H to BOINC is because BOINC didn't use GPUs. The reason is simple: CPUs are more accurate. An incorrect value often gets caught immediately in a CPU where a GPU requires testing on a separate machine in order to verify it's correctness and still not with 100% certainty.

We're doing science here: slow, correct results are more desirable than fast, incorrect results.

If TPU starts adding GPU work to its project list, I'll have to stop running BOINC for TPU. I refuse to compromise science for an arbitrary, altogether meaningless number.
Really? Where did you hear that? I did some quick Google-fu and came up with nothing...

And anyway, BOINC tasks for both GPUs and CPUs are double or triple-checked, with multiple computers performing the same tasks. I doubt BOINC would use GPUs if they didn't produce valid data.
__________________
Donate your idle CPU/GPU cycles and help save lives - crunch and fold for TechPowerUp!

BinaryMage is offline  
Reply With Quote
Old Dec 1, 2011, 01:51 AM   #17
ChaoticAtmosphere
1000 Posts
 
ChaoticAtmosphere's Avatar
 
Join Date: Oct 2008
Location: Look up. Look way up.
Posts: 1,977 (1.18/day)
Thanks: 2,717
Thanked 1,689 Times in 796 Posts

System Specs

Well electricity is no concern for me as it is included in my rent and it's why I have a 10,000BTU air conditioner in my room dedicated to cooling my rig.....almost like a server room

More good news, I plan on buying a second HD6870 in 2 weeks (Or maybe I should buy the bulldozer, hmmmm decisions decisions.....)
ChaoticAtmosphere is offline  
Reply With Quote
Old Dec 1, 2011, 02:52 AM   #18
twilyth
Guest
 
Posts: n/a (0/day)

Hell, I'll load up on 6970's or 7xxx's if each one can do more work than an overclocked 2600k.
 
Reply With Quote
Old Dec 1, 2011, 07:15 AM   #19
KieX
2000 Posts
 
Join Date: Dec 2009
Location: A Coruña, Spain
Posts: 2,050 (1.64/day)
Thanks: 6,950
Thanked 2,643 Times in 1,136 Posts
Send a message via Skype™ to KieX

System Specs

Quote:
Originally Posted by FordGT90Concept View Post
The reason I switched from F@H to BOINC is because BOINC didn't use GPUs. The reason is simple: CPUs are more accurate. An incorrect value often gets caught immediately in a CPU where a GPU requires testing on a separate machine in order to verify it's correctness and still not with 100% certainty.

We're doing science here: slow, correct results are more desirable than fast, incorrect results.

If TPU starts adding GPU work to its project list, I'll have to stop running BOINC for TPU. I refuse to compromise science for an arbitrary, altogether meaningless number.
Might want to take a look at the PDF with their initial findings: http://www.cs.toronto.edu/~juris/WCG/UPDATE-SEP2011.pdf

It includes a section on their findings with regards to accuracy. Like you said, GPU is always going to trade off some accuracy but it seems that the researchers find this to be a tolerable margin in order to run simultaneously with CPU WU.
__________________
KieX is offline  
Reply With Quote
The Following 2 Users Say Thank You to KieX For This Useful Post:
Old Dec 1, 2011, 04:46 PM   #20
Chicken Patty
WCG Team Captain
 
Chicken Patty's Avatar
 
Join Date: Nov 2007
Location: Miami, Florida
Posts: 23,895 (11.92/day)
Thanks: 15,156
Thanked 12,315 Times in 7,779 Posts
Send a message via AIM to Chicken Patty Send a message via MSN to Chicken Patty

System Specs

I won't argue FORD on the accuracy/GPU subject because he does have a valid point, but keep in mind that the higher than meaningless # is, the more research that is getting done.

Now is this just a personal thing that would not like to Crunch for a team that uses GPU's as well, or you refuse to crunch with GPU's? I mean, no one here can obviously hold you back, but as a consistant and dedicated cruncher you are, I'm sure no one here would like to see you go. I know I sure as heck wouldn't. Not only that, but you've been of great help to the community in other things not just the research.
__________________
Help save lives, CRUNCH and/or FOLD for Team TPU and make a difference!
Crunching with a 2600K, i7-950, FX-8120, Radeon 6850 (2 WU's), Radeon 7770 (4 WU's)

"AlienIsGOD - Awe!!! CP has grown from a small chicken nugget to the IMMENSE PATTY and WCG Team leader he is today"
Chicken Patty is offline  
Crunching for Team TPU
Reply With Quote
The Following 2 Users Say Thank You to Chicken Patty For This Useful Post:
Old Dec 1, 2011, 11:28 PM   #21
ChaoticAtmosphere
1000 Posts
 
ChaoticAtmosphere's Avatar
 
Join Date: Oct 2008
Location: Look up. Look way up.
Posts: 1,977 (1.18/day)
Thanks: 2,717
Thanked 1,689 Times in 796 Posts

System Specs

Hmmmmm, Maybe a bulldozer then.....
ChaoticAtmosphere is offline  
Reply With Quote
Old Dec 1, 2011, 11:37 PM   #22
(FIH) The Don
Eligible for custom title
 
(FIH) The Don's Avatar
 
Join Date: Aug 2009
Location: Viking Land
Posts: 5,406 (3.91/day)
Thanks: 3,101
Thanked 2,962 Times in 1,834 Posts
Send a message via MSN to (FIH) The Don Send a message via Skype™ to (FIH) The Don

System Specs

Quote:
Originally Posted by twilyth View Post
Hell, I'll load up on 6970's or 7xxx's if each one can do more work than an overclocked 2600k.
i hope that this gpu project wont have nvidia favored like it is in f@h

i hope for some equal competition in this, just for once
__________________
"there are 2 types of idiots. those who hate the new stuff to justify not being able to upgrade. and those who have no fucking clue about anything" -KieX
“So with this new technology my video card will now fart?” -TheMailMan78
(FIH) The Don is offline  
Reply With Quote
The Following 2 Users Say Thank You to (FIH) The Don For This Useful Post:
Old Dec 1, 2011, 11:43 PM   #23
Bow
1000 Posts
 
Bow's Avatar
 
Join Date: Jul 2008
Location: South Colton, New York
Posts: 1,653 (0.94/day)
Thanks: 755
Thanked 664 Times in 401 Posts

System Specs

Quote:
Originally Posted by (FIH) The Don View Post
i hope that this gpu project wont have nvidia favored like it is in f@h

i hope for some equal competition in this, just for once
__________________
"I didn't want my Mom to know I was drag racing for 20 years so I told her I was in prison."
Bow is offline  
Crunching for Team TPU
Reply With Quote
Old Dec 2, 2011, 12:27 AM   #24
Chicken Patty
WCG Team Captain
 
Chicken Patty's Avatar
 
Join Date: Nov 2007
Location: Miami, Florida
Posts: 23,895 (11.92/day)
Thanks: 15,156
Thanked 12,315 Times in 7,779 Posts
Send a message via AIM to Chicken Patty Send a message via MSN to Chicken Patty

System Specs

Quote:
Originally Posted by ChaoticAtmosphere View Post
Hmmmmm, Maybe a bulldozer then.....
I'm really dying to have some money left over and maybe build a BD rig. Just dying to see what it can do.
__________________
Help save lives, CRUNCH and/or FOLD for Team TPU and make a difference!
Crunching with a 2600K, i7-950, FX-8120, Radeon 6850 (2 WU's), Radeon 7770 (4 WU's)

"AlienIsGOD - Awe!!! CP has grown from a small chicken nugget to the IMMENSE PATTY and WCG Team leader he is today"
Chicken Patty is offline  
Crunching for Team TPU
Reply With Quote
The Following User Says Thank You to Chicken Patty For This Useful Post:
Old Dec 2, 2011, 05:36 AM   #25
BinaryMage
500 Posts
 
BinaryMage's Avatar
 
Join Date: Feb 2011
Posts: 606 (0.72/day)
Thanks: 850
Thanked 457 Times in 266 Posts

System Specs

Quote:
Originally Posted by (FIH) The Don View Post
i hope that this gpu project wont have nvidia favored like it is in f@h

i hope for some equal competition in this, just for once
The opposite will likely be true. AMD cards, by raw compute power, are way more powerful than comparatively priced nVidia ones, usually by about a factor of two. (For example, the HD 6990 has roughly 5 TFLOPs versus the GTX 590's 2.5) Folding@home, which was initially designed for CUDA (nVidia), hasn't ever really used AMD cards very efficiently. BOINC, however, does. Assuming this follows similiar trends as the other BOINC projects, you will get roughly double the crunching ability and thus credit from an AMD GPU versus a similarly priced nVidia one. Hope that cheers you up!
__________________
Donate your idle CPU/GPU cycles and help save lives - crunch and fold for TechPowerUp!

BinaryMage is offline  
Reply With Quote
The Following User Says Thank You to BinaryMage For This Useful Post:
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
2011 WCG Contest Discussion Thread KieX World Community Grid (WCG) 108 Sep 24, 2011 04:23 PM
AMD 6 Core WCG production discussion Paulieg World Community Grid (WCG) 51 May 1, 2010 05:34 PM
HD 5870 Discussion thread. a_ump AMD / ATI 610 Jan 13, 2010 06:38 PM
[Core i7] Watercooling discussion thread mudkip Overclocking & Cooling 256 Oct 16, 2009 05:35 AM
TPU WCG contest discussion thread² Paulieg World Community Grid (WCG) 92 Oct 9, 2009 06:34 PM


All times are GMT. The time now is 09:52 AM.


Powered by vBulletin® Version 3.8.6
Copyright ©2000 - 2013, Jelsoft Enterprises Ltd.
no new posts