Should we have a "benchmarks" type thread for LLMS?

[Easy Rhino]

I can't seem to find any good data that highlights tokens per second with local LLMS and graphics cards.

I am proposing something like a table that include, tpu username, graphics card and vram, graphics driver and version, platform (ollama, lm studio, etc), cpu, ram, operating system and version, model, tokens per second, date of benchmark.

Are there any other relevant fields to add? Are other people interested in this kind of information?

 

[Solaris17]

I was thinking about doing something like this so I’m for it.

 

[Easy Rhino]

I think the challenge will be verifying the results. People could just post crap because reasons. it would be neat if projects like ollama had a way to upload results to a database.

 

[Solaris17]

I think the challenge will be verifying the results. People could just post crap because reasons. it would be neat if projects like ollama had a way to upload results to a database.

I was just thinking that. I was going to maybe make something bundled that just runs the benchmarks, but I'm legit stupid when it comes to DB APIs. We should def think tank this though, I was going to start a thread of my own literally like last week in my case, specifically for the Intel cards using AI playground just for the lols, but I had my teeth yanked so im on hella drugs right now and Dont feel like the song and dance.

I think a good place to start is reading some of the GPU reviews where w1zz displays some AI stats to see what is important.

Off the top of my head (imo)

- Tokens total (some models and services charge by tokens total)
- Tokens/s (tokens /s determines readability, and weather your talking to an adult at normal speech cadence or someone that barely knows how to read.)
- Time to completion
- VRAM Usage

I was also thinking "Accuracy" In my case I was going to throw it a known problem, to make a script I already know how to make as a human, and see how well it does in replicating it or something that works, in a vanilla sense. IE: not telling it to iterate or improve my own.

Accuracy is a little harder though and I wasd in the middle of thinking about it when I started taking pain killers. Love the idea though.

 

[KLMR]

Local LLMs should be the way to go in most cases. TPU should encourage local testing on non-workstation PCs.

 

[Solaris17]

Local LLMs should be the way to go in most cases. TPU should encourage local testing on non-workstation PCs.

I dont think any of the testing that is done in the GPU reviews is on workstation PCs. We actually dont do any testing on workstation class gear since forever. I think Easy is talking about member contributed here, not like a new way to review.

 

[KLMR]

I dont think any of the testing that is done in the GPU reviews is on workstation PCs. We actually dont do any testing on workstation class gear since forever. I think Easy is talking about member contributed here, not like a new way to review.

My bad, what I meant is TPU is the perfect place to do tests of local LLMs for non-workstation hardware.

Smaller sites haven't the resources and the variety of hardware TPU has.

For workstation class hardware servethehome, level1techs or maybe the articles from puget. I must say its difficult to find good "workstation" review places or at least as "wide" as TPU. Since brands don't usually want their hardware to be exposed (hp, dell, etc.) and non-branded workstations is such a niche market.

 

[Solaris17]

My bad, what I meant is TPU is the perfect place to do tests of local LLMs for non-workstation hardware.

Smaller sites haven't the resources and the variety of hardware TPU has.

For workstation class hardware servethehome, level1techs or maybe the articles from puget. I must say its difficult to find good "workstation" review places or at least as "wide" as TPU. Since brands don't usually want their hardware to be exposed (hp, dell, etc.) and non-branded workstations is such a niche market.

I always dreamed that one day TPU would expand there review market; I would jump on signing up to do WS/Enterprise gear. Maybe someday.

As far as doing more AI stuff; I hope given its proliferation more tests will be included during things like GPU reviews, that fact that it was even added at all is a good sign, maybe one day there will be more targeted tests in an official capacity from w1zz.

We're just peons though. So humble threads is the best we can do.

 

[Makaveli]

Good Idea I would participate.

 

[igormp]

Something like Vlad's benchmark for SD would be pretty handy and doable:

SD WebUI Benchmark Data

SD WebUI Benchmark Data; Author: Vladimir Mandic <https://github.com/vladmandic>

vladmandic.github.io

Maybe use llama-bench as the baseline and build a DB on top of it. Don't even need to get fancy with it, any nosql DB or just your regular rdbms (if it's already in place) will work.

 

[bug]

I can't seem to find any good data that highlights tokens per second with local LLMS and graphics cards.

I am proposing something like a table that include, tpu username, graphics card and vram, graphics driver and version, platform (ollama, lm studio, etc), cpu, ram, operating system and version, model, tokens per second, date of benchmark.

Are there any other relevant fields to add? Are other people interested in this kind of information?

All content on TPU is static (i.e. no animation, no dynamic HTML). How would you make a table with that many columns useful/readable?

 

[Easy Rhino]

Something like Vlad's benchmark for SD would be pretty handy and doable:

SD WebUI Benchmark Data

SD WebUI Benchmark Data; Author: Vladimir Mandic <https://github.com/vladmandic>

vladmandic.github.io

Maybe use llama-bench as the baseline and build a DB on top of it. Don't even need to get fancy with it, any nosql DB or just your regular rdbms (if it's already in place) will work.

I did not know that program exists. If that could be extended to support output to a URL I could setup a database and provide a restful endpoint to publish the benchmark results.

 

[Wirko]

All content on TPU is static (i.e. no animation, no dynamic HTML). How would you make a table with that many columns useful/readable?

TPUCPUGPUSSD databases are already close to what would be needed and usable. Their search pages are dynamic HTML, even if the lists are filtered on the server, not the browser.

 

[igormp]

I did not know that program exists. If that could be extended to support output to a URL I could setup a database and provide a restful endpoint to publish the benchmark results.

That should be totally doable. Any way that you can think of to validate results with a TPU account or something like that?

 

[Easy Rhino]

That should be totally doable. Any way that you can think of to validate results with a TPU account or something like that?

Yes, if the project is forked then a custom binary could be created specifically for TPU. Unfortunately I do not know CPP.

 

[igormp]

Yes, if the project is forked then a custom binary could be created specifically for TPU. Unfortunately I do not know CPP.

I can lend a hand if you want.
Any language you or the TPU team would be comfortable with? The bench itself could also be just wrapped (in the sense of invoking it and storing the results) within another language to make it easier for others to work with.