• Welcome to TechPowerUp Forums, Guest! Please check out our forum guidelines for info related to our community.

We Tested NVIDIA's new ChatRTX: Your Own GPU-accelerated AI Assistant with Photo Recognition, Speech Input, Updated Models

btarunr

Editor & Senior Moderator
Staff member
Joined
Oct 9, 2007
Messages
47,164 (7.57/day)
Location
Hyderabad, India
System Name RBMK-1000
Processor AMD Ryzen 7 5700G
Motherboard ASUS ROG Strix B450-E Gaming
Cooling DeepCool Gammax L240 V2
Memory 2x 8GB G.Skill Sniper X
Video Card(s) Palit GeForce RTX 2080 SUPER GameRock
Storage Western Digital Black NVMe 512GB
Display(s) BenQ 1440p 60 Hz 27-inch
Case Corsair Carbide 100R
Audio Device(s) ASUS SupremeFX S1220A
Power Supply Cooler Master MWE Gold 650W
Mouse ASUS ROG Strix Impact
Keyboard Gamdias Hermes E2
Software Windows 11 Pro
NVIDIA today unveiled ChatRTX, the AI assistant that runs locally on your machine, and which is accelerated by your GeForce RTX GPU. NVIDIA had originally launched this as "Chat with RTX" back in February 2024, back then this was regarded more as a public tech demo. We reviewed the application in our feature article. The ChatRTX rebranding is probably aimed at making the name sound more like ChatGPT, which is what the application aims to be—except it runs completely on your machine, and is exhaustively customizable. The most obvious advantage of a locally-run AI assistant is privacy—you are interacting with an assistant that processes your prompt locally, and accelerated by your GPU; the second is that you're not held back by performance bottlenecks by cloud-based assistants.

ChatRTX is a major update over the Chat with RTX tech-demo from February. To begin with, the application has several stability refinements from Chat with RTX, which felt a little rough on the edges. NVIDIA has significantly updated the LLMs included with the application, including Mistral 7B INT4, and Llama 2 7B INT4. Support is also added for additional LLMs, including Gemma, a local LLM trained by Google, based on the same technology used to make Google's flagship Gemini model. ChatRTX now also supports ChatGLM3, for both English and Chinese prompts. Perhaps the biggest upgrade ChatRTX is its ability to recognize images on your machine, as it incorporates CLIP (contrastive language-image pre-training) from OpenAI. CLIP is an LLM that recognizes what it's seeing in image collections. Using this feature, you can interact with your image library without the need for metadata. ChatRTX doesn't just take text input—you can speak to it. It now accepts natural voice input, as it integrates the Whisper speech-to-text NLI model.



DOWNLOAD: NVIDIA ChatRTX

As with the original Chat with RTX tech-demo, the new ChatRTX application's biggest feature is its ability to let users switch between AI models, or to let it build and train your own dataset based on text and images on your local machine. You can point it to a folder with documents such as plaintext, Word (doc) and PDFs, as well as images; and it will train itself on answering queries related to the dataset.


There remain some major limitations with ChatRTX which we had hoped would be fixed since its February release, and that is context—the ability to ask follow-up questions. Apparently, follow-ups are harder to implement than they seem, as the model has to connect the new question to the previous ones and its responses to them. It's also inaccurate in attributing its responses to the right text tiles. The browser-based frontend only supports Chrome and Edge, it's buggy with Firefox.

View at TechPowerUp Main Site
 

dgianstefani

TPU Proofreader
Staff member
Joined
Dec 29, 2017
Messages
4,990 (2.00/day)
Location
Swansea, Wales
System Name Silent
Processor Ryzen 7800X3D @ 5.15ghz BCLK OC, TG AM5 High Performance Heatspreader
Motherboard ASUS ROG Strix X670E-I, chipset fans replaced with Noctua A14x25 G2
Cooling Optimus Block, HWLabs Copper 240/40 + 240/30, D5/Res, 4x Noctua A12x25, 1x A14G2, Mayhems Ultra Pure
Memory 32 GB Dominator Platinum 6150 MT 26-36-36-48, 56.6ns AIDA, 2050 FCLK, 160 ns tRFC, active cooled
Video Card(s) RTX 3080 Ti Founders Edition, Conductonaut Extreme, 18 W/mK MinusPad Extreme, Corsair XG7 Waterblock
Storage Intel Optane DC P1600X 118 GB, Samsung 990 Pro 2 TB
Display(s) 32" 240 Hz 1440p Samsung G7, 31.5" 165 Hz 1440p LG NanoIPS Ultragear, MX900 dual gas VESA mount
Case Sliger SM570 CNC Aluminium 13-Litre, 3D printed feet, custom front, LINKUP Ultra PCIe 4.0 x16 white
Audio Device(s) Audeze Maxwell Ultraviolet w/upgrade pads & LCD headband, Galaxy Buds 3 Pro, Razer Nommo Pro
Power Supply SF750 Plat, full transparent custom cables, Sentinel Pro 1500 Online Double Conversion UPS w/Noctua
Mouse Razer Viper Pro V2 8 KHz Mercury White w/Tiger Ice Skates & Pulsar Supergrip tape
Keyboard Wooting 60HE+ module, TOFU-R CNC Alu/Brass, SS Prismcaps W+Jellykey, LekkerV2 mod, TLabs Leath/Suede
Software Windows 11 IoT Enterprise LTSC 24H2
Benchmark Scores Legendary
V cool, gonna try it out later today
 
Joined
Oct 6, 2021
Messages
1,605 (1.43/day)
The level of trickery only increases.

All that's left is for Nvidia to create an assistant with the appearance of an Anime character to further captivate lonely nerds and keep them on a leash. If anyone wants, feel free to write down the idea for a film.
 
Joined
Sep 24, 2008
Messages
2,686 (0.46/day)
System Name Dire Wolf IV
Processor Intel Core i9 14900K
Motherboard Asus ROG STRIX Z790-I GAMING WIFI
Cooling Arctic Liquid Freezer II 280 w/Thermalright Contact Frame
Memory 2x24GB Corsair DDR5 6667
Video Card(s) NVIDIA RTX4080 FE
Storage AORUS Gen4 7300 1TB + Western Digital SN750 500GB
Display(s) Alienware AW3423DWF (QD-OLED, 3440x1440, 165hz)
Case Corsair Airflow 2000D
Power Supply Corsair SF1000L
Mouse Razer Deathadder Essential
Keyboard Chuangquan CQ84
Software Windows 11 Professional
The level of trickery only increases.

All that's left is for Nvidia to create an assistant with the appearance of an Anime character to further captivate lonely nerds and keep them on a leash. If anyone wants, feel free to write down the idea for a film.
Her (film) - Wikipedia
 
Joined
Oct 18, 2013
Messages
6,133 (1.52/day)
Location
Over here, right where you least expect me to be !
System Name The Little One
Processor i5-11320H @4.4GHZ
Motherboard AZW SEI
Cooling Fan w/heat pipes + side & rear vents
Memory 64GB Crucial DDR4-3200 (2x 32GB)
Video Card(s) Iris XE
Storage WD Black SN850X 4TB m.2, Seagate 2TB SSD + SN850 4TB x2 in an external enclosure
Display(s) 2x Samsung 43" & 2x 32"
Case Practically identical to a mac mini, just purrtier in slate blue, & with 3x usb ports on the front !
Audio Device(s) Yamaha ATS-1060 Bluetooth Soundbar & Subwoofer
Power Supply 65w brick
Mouse Logitech MX Master 2
Keyboard Logitech G613 mechanical wireless
Software Windows 10 pro 64 bit, with all the unnecessary background shitzu turned OFF !
Benchmark Scores PDQ
And so it begins....

Have rope, will strangle....

First they captured the AI cloud-base, now they are wiggling their way into your pc's insides, which will spawn the command & control of all pc's everywhere, all the time, all at once...

And then, all that will remain is "Hello SkyNet, how can I help you infiltrate & destroy humanity today ?

No thanks, cause resistance is NOT futile !
 
Joined
May 1, 2024
Messages
5 (0.03/day)
Location
Dallas, TX
Processor Ryzen 7 5800X3D
Motherboard Asus Prime Pro X370
Cooling Corsair H115i RGB Pro XT
Memory G Skill Ripjaws V DDR4 3200 CL14
Video Card(s) MSI RTX 4080 Super Ventus 3X OC
Storage 4TB Crucial PCIe 3.0 NVME
Display(s) LG UltraGear 27" OLED 240HZ
Case Corsair Air 540
Audio Device(s) Fiio K7BT w Audeze LCD-X
Power Supply EVGA SuperNova G2 1000
Mouse Corsair IronClaw
Keyboard Corsair K70 RGB (1st gen)
Nice, I will have to check this out. Wonder how much VRAM matters for this. 16GB seems to be the bare minimum for any sort of AI work from my limited knowledge.
 
Joined
Sep 6, 2013
Messages
3,306 (0.81/day)
Location
Athens, Greece
System Name 3 desktop systems: Gaming / Internet / HTPC
Processor Ryzen 5 5500 / Ryzen 5 4600G / FX 6300 (12 years latter got to see how bad Bulldozer is)
Motherboard MSI X470 Gaming Plus Max (1) / MSI X470 Gaming Plus Max (2) / Gigabyte GA-990XA-UD3
Cooling Νoctua U12S / Segotep T4 / Snowman M-T6
Memory 32GB - 16GB G.Skill RIPJAWS 3600+16GB G.Skill Aegis 3200 / 16GB JUHOR / 16GB Kingston 2400MHz (DDR3)
Video Card(s) ASRock RX 6600 + GT 710 (PhysX)/ Vega 7 integrated / Radeon RX 580
Storage NVMes, ONLY NVMes/ NVMes, SATA Storage / NVMe boot(Clover), SATA storage
Display(s) Philips 43PUS8857/12 UHD TV (120Hz, HDR, FreeSync Premium) ---- 19'' HP monitor + BlitzWolf BW-V5
Case Sharkoon Rebel 12 / CoolerMaster Elite 361 / Xigmatek Midguard
Audio Device(s) onboard
Power Supply Chieftec 850W / Silver Power 400W / Sharkoon 650W
Mouse CoolerMaster Devastator III Plus / CoolerMaster Devastator / Logitech
Keyboard CoolerMaster Devastator III Plus / CoolerMaster Devastator / Logitech
Software Windows 10 / Windows 10&Windows 11 / Windows 10
The level of trickery only increases.

All that's left is for Nvidia to create an assistant with the appearance of an Anime character to further captivate lonely nerds and keep them on a leash. If anyone wants, feel free to write down the idea for a film.
In my opinion the reason why Cortana failed in Windows. Imagine having the HALO hologram on screen when wanting some assistance instead of some text or plain audio.
 
Joined
Jan 10, 2011
Messages
1,430 (0.28/day)
Location
[Formerly] Khartoum, Sudan.
System Name 192.168.1.1~192.168.1.100
Processor AMD Ryzen5 5600G.
Motherboard Gigabyte B550m DS3H.
Cooling AMD Wraith Stealth.
Memory 16GB Crucial DDR4.
Video Card(s) Gigabyte GTX 1080 OC (Underclocked, underpowered).
Storage Samsung 980 NVME 500GB && Assortment of SSDs.
Display(s) ViewSonic VA2406-MH 75Hz
Case Bitfenix Nova Midi
Audio Device(s) On-Board.
Power Supply SeaSonic CORE GM-650.
Mouse Logitech G300s
Keyboard Kingston HyperX Alloy FPS.
VR HMD A pair of OP spectacles.
Software Ubuntu 24.04 LTS.
Benchmark Scores Me no know English. What bench mean? Bench like one sit on?
I find it amusing that a company whose entire business is built on top of graphics, can't be bothered to develop a decent, native GUI instead of this webserver-based crap.
Oh well... At least now people can get their wrong and nonsensical answers without risking [much of] their privacy.

Imagine having the HALO hologram on screen when wanting some assistance instead of some text or plain audio.
We had better.
Clippy-Featured.jpg


now they are wiggling their way into your pc's insides,
I have some really bad news for you, mate...
 
Last edited:
Joined
Feb 18, 2005
Messages
5,752 (0.80/day)
Location
Ikenai borderline!
System Name Firelance.
Processor Threadripper 3960X
Motherboard ROG Strix TRX40-E Gaming
Cooling IceGem 360 + 6x Arctic Cooling P12
Memory 8x 16GB Patriot Viper DDR4-3200 CL16
Video Card(s) MSI GeForce RTX 4060 Ti Ventus 2X OC
Storage 2TB WD SN850X (boot), 4TB Crucial P3 (data)
Display(s) 3x AOC Q32E2N (32" 2560x1440 75Hz)
Case Enthoo Pro II Server Edition (Closed Panel) + 6 fans
Power Supply Fractal Design Ion+ 2 Platinum 760W
Mouse Logitech G602
Keyboard Razer Pro Type Ultra
Software Windows 10 Professional x64
This makes me crave the sweet release of death.
 
Joined
Jul 13, 2016
Messages
3,243 (1.07/day)
Processor Ryzen 7800X3D
Motherboard ASRock X670E Taichi
Cooling Noctua NH-D15 Chromax
Memory 32GB DDR5 6000 CL30
Video Card(s) MSI RTX 4090 Trio
Storage Too much
Display(s) Acer Predator XB3 27" 240 Hz
Case Thermaltake Core X9
Audio Device(s) Topping DX5, DCA Aeon II
Power Supply Seasonic Prime Titanium 850w
Mouse G305
Keyboard Wooting HE60
VR HMD Valve Index
Software Win 10
Meh, just get GPT4All. Better and not locked down to Nvidia only.
 
Joined
Feb 18, 2005
Messages
5,752 (0.80/day)
Location
Ikenai borderline!
System Name Firelance.
Processor Threadripper 3960X
Motherboard ROG Strix TRX40-E Gaming
Cooling IceGem 360 + 6x Arctic Cooling P12
Memory 8x 16GB Patriot Viper DDR4-3200 CL16
Video Card(s) MSI GeForce RTX 4060 Ti Ventus 2X OC
Storage 2TB WD SN850X (boot), 4TB Crucial P3 (data)
Display(s) 3x AOC Q32E2N (32" 2560x1440 75Hz)
Case Enthoo Pro II Server Edition (Closed Panel) + 6 fans
Power Supply Fractal Design Ion+ 2 Platinum 760W
Mouse Logitech G602
Keyboard Razer Pro Type Ultra
Software Windows 10 Professional x64
It would be great to be able to buy next year a gaming PC which is not an AI PC.
Good luck with that. They're stuffing "AI" into anything they can, I'm honestly amazed there aren't AI dildos yet.
 

dgianstefani

TPU Proofreader
Staff member
Joined
Dec 29, 2017
Messages
4,990 (2.00/day)
Location
Swansea, Wales
System Name Silent
Processor Ryzen 7800X3D @ 5.15ghz BCLK OC, TG AM5 High Performance Heatspreader
Motherboard ASUS ROG Strix X670E-I, chipset fans replaced with Noctua A14x25 G2
Cooling Optimus Block, HWLabs Copper 240/40 + 240/30, D5/Res, 4x Noctua A12x25, 1x A14G2, Mayhems Ultra Pure
Memory 32 GB Dominator Platinum 6150 MT 26-36-36-48, 56.6ns AIDA, 2050 FCLK, 160 ns tRFC, active cooled
Video Card(s) RTX 3080 Ti Founders Edition, Conductonaut Extreme, 18 W/mK MinusPad Extreme, Corsair XG7 Waterblock
Storage Intel Optane DC P1600X 118 GB, Samsung 990 Pro 2 TB
Display(s) 32" 240 Hz 1440p Samsung G7, 31.5" 165 Hz 1440p LG NanoIPS Ultragear, MX900 dual gas VESA mount
Case Sliger SM570 CNC Aluminium 13-Litre, 3D printed feet, custom front, LINKUP Ultra PCIe 4.0 x16 white
Audio Device(s) Audeze Maxwell Ultraviolet w/upgrade pads & LCD headband, Galaxy Buds 3 Pro, Razer Nommo Pro
Power Supply SF750 Plat, full transparent custom cables, Sentinel Pro 1500 Online Double Conversion UPS w/Noctua
Mouse Razer Viper Pro V2 8 KHz Mercury White w/Tiger Ice Skates & Pulsar Supergrip tape
Keyboard Wooting 60HE+ module, TOFU-R CNC Alu/Brass, SS Prismcaps W+Jellykey, LekkerV2 mod, TLabs Leath/Suede
Software Windows 11 IoT Enterprise LTSC 24H2
Benchmark Scores Legendary
Good luck with that. They're stuffing "AI" into anything they can, I'm honestly amazed there aren't AI dildos yet.
They exist according to the Kagi search I just did.

Glad I didn't use Google or I might start getting targeted adverts...
 
Joined
Nov 30, 2019
Messages
22 (0.01/day)
System Name X64
Processor i9-10920X
Motherboard ASUS Pro WS X299 SAGE II
Cooling Air
Memory Corsair Dominator Platinum RGB 64GB (4x16GB) DDR4 3600 CL16
Video Card(s) MSI GTX 1070 Gaming
Storage Samsung 850 Pro SSD
Display(s) Dell U2417H
Case CM C700M
Power Supply Seasonic Prime TX 1300W ATX 3.0
AI this AI that...and here i am just wanting a reasonably good and affordable GPU without a need to sell a kidney.
 

bug

Joined
May 22, 2015
Messages
13,712 (3.97/day)
Processor Intel i5-12600k
Motherboard Asus H670 TUF
Cooling Arctic Freezer 34
Memory 2x16GB DDR4 3600 G.Skill Ripjaws V
Video Card(s) EVGA GTX 1060 SC
Storage 500GB Samsung 970 EVO, 500GB Samsung 850 EVO, 1TB Crucial MX300 and 2TB Crucial MX500
Display(s) Dell U3219Q + HP ZR24w
Case Raijintek Thetis
Audio Device(s) Audioquest Dragonfly Red :D
Power Supply Seasonic 620W M12
Mouse Logitech G502 Proteus Core
Keyboard G.Skill KM780R
Software Arch Linux + Win10
Windows-only. Shove it, Nvidia.
 
Joined
Oct 18, 2013
Messages
6,133 (1.52/day)
Location
Over here, right where you least expect me to be !
System Name The Little One
Processor i5-11320H @4.4GHZ
Motherboard AZW SEI
Cooling Fan w/heat pipes + side & rear vents
Memory 64GB Crucial DDR4-3200 (2x 32GB)
Video Card(s) Iris XE
Storage WD Black SN850X 4TB m.2, Seagate 2TB SSD + SN850 4TB x2 in an external enclosure
Display(s) 2x Samsung 43" & 2x 32"
Case Practically identical to a mac mini, just purrtier in slate blue, & with 3x usb ports on the front !
Audio Device(s) Yamaha ATS-1060 Bluetooth Soundbar & Subwoofer
Power Supply 65w brick
Mouse Logitech MX Master 2
Keyboard Logitech G613 mechanical wireless
Software Windows 10 pro 64 bit, with all the unnecessary background shitzu turned OFF !
Benchmark Scores PDQ
I'm honestly amazed there aren't AI dildos yet
"I am fully functional, and programmed in multiple techniques" - Lt. Cmdr Data :D

'nuff said !
 
  • Love
Reactions: bug
Joined
Oct 6, 2021
Messages
1,605 (1.43/day)
Nah, I was pondering something more profound and ominous, envisioning the CEO in leather jacket entering a sinister pact with pharmaceutical companies. Their aim? To employ any means necessary to tether people to their screens, fostering obesity to peddle weight loss pills as miraculous solution. You can take it, Netflix, :p
 
Joined
Sep 17, 2014
Messages
22,278 (6.02/day)
Location
The Washing Machine
Processor 7800X3D
Motherboard MSI MAG Mortar b650m wifi
Cooling Thermalright Peerless Assassin
Memory 32GB Corsair Vengeance 30CL6000
Video Card(s) ASRock RX7900XT Phantom Gaming
Storage Lexar NM790 4TB + Samsung 850 EVO 1TB + Samsung 980 1TB + Crucial BX100 250GB
Display(s) Gigabyte G34QWC (3440x1440)
Case Lian Li A3 mATX White
Audio Device(s) Harman Kardon AVR137 + 2.1
Power Supply EVGA Supernova G2 750W
Mouse Steelseries Aerox 5
Keyboard Lenovo Thinkpad Trackpoint II
Software W11 IoT Enterprise LTSC
Benchmark Scores Over 9000
Holy cow... 11.6GB...????
Fits in an x70's VRAM!

Until the next update

Nah, I was pondering something more profound and ominous, envisioning the CEO in leather jacket entering a sinister pact with pharmaceutical companies. Their aim? To employ any means necessary to tether people to their screens, fostering obesity to peddle weight loss pills as miraculous solution. You can take it, Netflix, :p
All that's left for you to do is find a link that says Huang tried Ozempic and you're up for internet hero status. The ingredients are all there, live and in effect already... :D Who needs Netflix?
 
Joined
Oct 1, 2014
Messages
1,964 (0.53/day)
Location
Calabash, NC
System Name The Captain (2.0)
Processor Ryzen 7 7700X
Motherboard Asus ROG Strix X670E-A
Cooling 280mm Arctic Liquid Freezer II, 4x Be Quiet! 140mm Silent Wings 4 (1x exhaust 3x intake)
Memory 32GB (2x16) Kingston Fury Beast CL30 6000MT/s
Video Card(s) MSI GeForce RTX 3070 SUPRIM X
Storage 1x Crucial MX500 500GB SSD; 1x Crucial MX500 500GB M.2 SSD; 1x WD Blue HDD, 1x Crucial P5 Plus
Display(s) Aorus CV27F 27" 1080p 165Hz
Case Phanteks Evolv X (Anthracite Gray)
Power Supply Corsair RMx (2021) 1000W 80-Plus Gold
Mouse Varies based on mood/task; is currently Razer Basilisk V3 Pro or Razer Cobra Pro
Keyboard Varies based on mood; currently Razer Blackwidow V4 75% and Hyper X Alloy 65
Good luck with that. They're stuffing "AI" into anything they can, I'm honestly amazed there aren't AI dildos yet.

They exist according to the Kagi search I just did.

Glad I didn't use Google or I might start getting targeted adverts...

"AI" in dildos now? Is NOTHING sacred? :cry: It's a good thing I'm more of a "hands on" kind of gal... Go to hell, Skynet Dildo!
 
Joined
Oct 6, 2021
Messages
1,605 (1.43/day)
Fits in an x70's VRAM!

Until the next update


All that's left for you to do is find a link that says Huang tried Ozempic and you're up for internet hero status. The ingredients are all there, live and in effect already... :D Who needs Netflix?
I'll have my GPT henchman do it;
Placing creation against the creator is a cliché but it works. :toast:
 
Joined
Apr 19, 2018
Messages
1,227 (0.51/day)
Processor AMD Ryzen 9 5950X
Motherboard Asus ROG Crosshair VIII Hero WiFi
Cooling Arctic Liquid Freezer II 420
Memory 32Gb G-Skill Trident Z Neo @3806MHz C14
Video Card(s) MSI GeForce RTX2070
Storage Seagate FireCuda 530 1TB
Display(s) Samsung G9 49" Curved Ultrawide
Case Cooler Master Cosmos
Audio Device(s) O2 USB Headphone AMP
Power Supply Corsair HX850i
Mouse Logitech G502
Keyboard Cherry MX
Software Windows 11
Fits in an x70's VRAM!

Until the next update
12.9GB after decompression, so no, you need an upgrade! nGreedia will kindly help you out though. ;)
 
Joined
Jan 18, 2020
Messages
800 (0.46/day)
But won't someone think about Clippy ?

Can copilot be run locally or is that coming?
 

bug

Joined
May 22, 2015
Messages
13,712 (3.97/day)
Processor Intel i5-12600k
Motherboard Asus H670 TUF
Cooling Arctic Freezer 34
Memory 2x16GB DDR4 3600 G.Skill Ripjaws V
Video Card(s) EVGA GTX 1060 SC
Storage 500GB Samsung 970 EVO, 500GB Samsung 850 EVO, 1TB Crucial MX300 and 2TB Crucial MX500
Display(s) Dell U3219Q + HP ZR24w
Case Raijintek Thetis
Audio Device(s) Audioquest Dragonfly Red :D
Power Supply Seasonic 620W M12
Mouse Logitech G502 Proteus Core
Keyboard G.Skill KM780R
Software Arch Linux + Win10
But won't someone think about Clippy ?

Can copilot be run locally or is that coming?
I think they can all run locally, but they eat too much RAM.
Remember the debacle around Gemini on Pixels, when Google wanted to leave out the 8 because of "hardware limitations", despite it being exactly the same as the Pro, only with 8GB RAM instead of 12?

I believe what is going on here is companies figuring out ways to shrink their models while still offering useful functionality or otherwise compressing the models better, before declaring them ready to run on local machines.
 
Top