How I built a cheap AI and Deep Learning Workstation quickly

9 min read1 day ago

With the growing popularity of generative AI I decided that I need my own AI/Deep Learning workstation with a dedicated GPU. My goal was to learn more about the types of GPUs available. Whether you are trying to build your own workstation or planning to rent one or more GPUs in the cloud, it is important to understand what is available and how much it costs. Taking the current prices of GPUs and other components you need to take the word “cheap” with the grain of salt. By cheap I mean picking parts that offer the best opportunity at a lower price than the high-end hardware.

Building a Deep Learning workstation can be a costly and time consuming endeavor. The top of the line GPUs cost several thousand dollars. In addition, the more powerful GPU you chose, the more powerful other components such as a CPU should be. A low end CPU can become a bottleneck for a powerful GPU because a GPU is a co-processor, that is the CPU has to move data from the system RAM to GPU video RAM and back. Moreover, the more video memory a GPU has, the more system RAM is required to use the GPU to full capacity. At the time of writing, the cost of RAM is not high, so that should not be a problem.

Why you should not build a Deep Learning workstation

Before we further discuss my journey of building an AI workstation I would like to share my thoughts on why it may not be a good idea to build your own AI PC.

Firstly, as was mentioned before, the prices of the components to build a powerful Deep Learning workstation such as a CPU, motherboard, and the video card are quite high.
Secondly, some knowledge and experience of building PCs are required to pick the components that will work with each other and do the assembly.
Thirdly, there are some free options such as Google Colab and Kaggle offering GPUs for free.
Fourthly, Small Language Models can be run using Ollama on a CPU and don’t require a dedicated GPU.
Fifthly, you own an Apple laptop with a neural engine and can run experiments with neural networks using Apple silicon.
Finally, a home computer with a GPU won’t allow training the state-of-the art models. That process requires thousands of expensive GPUs and even running inference requires multiple high end GPUs.

So the conclusion is that if you are just beginning with AI and Deep Learning and do not have a lot of time it is better to start with free options such as Google Colab or rent GPUs online which can cost a few dollars per hour. For experiments with Large Language Models using third-party APIs is a viable option.

When it’s a good idea to build a Deep Learning Workstation

I believe the following may be good reasons to build an AI workstation yourself:

If you are interested in hardware powering AI and would like to learn how to build your own PC, building an AI workstation is for you.
If you prefer to develop and debug some training code locally that you later plan to run in the cloud, possibly on several GPUs, building your own AI PC may be an option for you.
If you are working on some research projects that require a GPU around the clock but your budget is tight, it may be more cost efficient to build a powerful AI workstation yourself.
An additional reason why you may need a workstation with a GPU is you are going to use it for other purposes such as content reaction or gaming.

Main considerations when building a Deep Learning workstation

First, you need to understand what AI or Deep Learning experiments you are going to run. It could be image generation, text to speech or natural language processing and generation. The models that you are going to use have memory requirements. As was mentioned before, the state-of-the art LLMs require several powerful GPUs to serve and using a third-party APIs can be an only solution here.

Consumer GPUs such as the NVIDIA GeForce RTX 4090 do not offer more than 24 GB of video RAM which limit the types of AI models that you can train locally. Having several high-end GPUs is expensive and if you need more memory or performance, using cloud GPUs can solve the issue. In a nutshell, the first decision is on the amount of memory that you need and what you can afford.

Second, you need to select the right CPU depending on the GPU you’ve picked on step one. The more powerful and expensive GPU you’ve selected the more powerful and expensive GPU you need. You can think of a GPU as an engine of a car and of CPU as a gas pump. A powerful engine requires a pump that can provide it with enough gas.

Third, after you’ve selected the CPU you need to pick a motherboard that supports the CPU. Once again, an expensive and powerful GPU will require a more expensive motherboard. While a CPU can support a couple of generations of memory, such as DDR4, DDR5 or DDR6 that is coming soon, a motherboard supports only one. You need to get the type of memory that fits into your motherboard and the amount that is greater than the size of video memory of the GPU. This is because models and data are loaded first into the computer’s RAM and then to the RAM of the video card.

Fourth, graphic cards or GPUs are pretty big, they can occupy two or three slots and they are pretty long. If you’ve selected a datacenter GPU with passive cooling, you would need to attach a separate fan to it unless you have a server case that has additional fans to cool the GPU. Long story short, you need to check the size of the GPU before selecting a case and usually powerful GPUs require a big one.

Fifth, most GPUs are power-hungry and drawing 250 W or more is usual for a high-end GPU. That requires a Power Supply Unit or PSU that can provide enough power to the GPU and other parts of your workstation. Moreover, devices like video cards can pull only up to 75 W from the motherboard, the rest comes from a special connector from the PSU. Powerful PSUs usually have a cable with the connector required.

Finally, other peripherals such as storage. A deep learning workstation requires fast storage such as an SSD. The capacity depends on the amount of models and data you are going to keep. In addition you may need a cooler for the CPU and additional fans. I do not count a monitor, mouse, and keyboard.

Why I decided to build my own AI/Deep Learning workstation

I tried Google Colab for some experiments but usually the training code used in Jupyter-like notebooks is not production-ready. I’d like to write and debug proper code in an IDE, that later can be run on a GPU or multiple GPUs in the cloud. Also, I wanted to learn more about GPUs, their capabilities and what can be done with a certain amount of memory. In addition, it was interesting for me to brush up my computer-building skills. Furthermore, I would like to edit videos and remove background noise from audio files using the capabilities of my GPU.

What my approach was

It’s been a while since I’ve built a PC myself and I had to do a lot of research on modern components and how to select ones that fit each other. My first consideration was that I would like to learn more about generative AI and not become an expert in building the most powerful PCs possible. The second consideration is the amount of time that I’d like to spend on building the workstation and how much time I can spend on my experiments with AI which is not a lot as I have a full-time job. All that put a limit on the amount of money I wanted to spend on the project.

For my first iteration of the workstation I decided to focus on a budget GPU with as much memory as possible. My choice was NVIDIA GeForce RTX 3060 with 12 GB of video RAM. This GPU has third-generation Tensor Cores and supports some smaller precision data types that accelerate model inference. If your goal is to learn some TensorFlow or PyTorch a GPU with a smaller amount of memory will work. It is interesting to me what I can do with 12 GB of video RAM and I will share my findings in additional posts.

Later I may try a more powerful GPU. Another idea is to try an older GPU with more memory. Squeezing as much performance as possible out of hardware was not my goal. What was important to me is to better understand GPUs and get up-to-speed with modern hardware to build something more powerful later.

Furthermore, research and picking the right components for the PC was taking too much time and led to analysis paralysis so I decided to buy a refurbished PC with a minitower case that can fit my 3060 GPU. The single thing that I needed to replace was a PSU as the one that came with the PC didn’t provide enough power and didn’t have a special power connector for the GPU.

I settled on Dell Optiplex 9020 minitower Desktop PC with Intel Core i7–4770–3.4 GHz CPU, 32 GB of RAM and 512 GB SSD. As a CPU can be a bottleneck for a GPU I needed something powerful such as Intel Core i7 and to learn more about bottlenecking I plan to run some experiments with my setup. The maximum amount of RAM supported is 32 GB. The native power source unit provides 295 W.

As a PSU I picked Thermaltake Smart 500W. The power consumption of GeForce RTX 3060 graphics card is 170W. The required system power is 550 W from the card’s specification. As 295 W was enough to power the PC and the card requires only 170 W which in total gives 465 W, I decided that 500 W would be enough. There are variants of the same PSU with 600 W and 700 W though.

Please note that I picked a refurbished PC with a pretty big, minitower case. If you pick something smaller, you may need to buy a new case as your GPU may not fit. Moreover, this case may not be big enough for a more powerful GPU or a datacenter GPU that requires a blow-through fan.

A possible multi-GPU setup

One GPU may not provide enough memory and computational power for your tasks. The solution is to fit more than one GPUs into your rig. There are several potential problems with that approach.

Firstly, GPUs are wide, occupy several slots so you need a big case and a motherboard with enough PCI slots to fit all the GPUs. A possible solution for any motherboard is to use a PCI extension cable but that may look messy.

Secondly, as GPUs are power-hungry, you need a very powerful PSU and possibly end up with a big electricity bill. Noise from the fans may be an additional problem.

Finally, consumer GPUs have fans on their side, so you can’t pack several of these close to each other, otherwise they won’t be able to cool themselves. A possible solution would be to use refurbished datacenter GPUs with blow through fans. Another approach is to buy a refurbished server but all of that may drastically increase the cost of the project and the electricity bill. Please share your thoughts in the comments.

I may try a two-GPU setup in the next iterations of building my workstation and will share my experience in a separate article. For now, if I decide to run experiments with multiple GPUs, I will rely on cloud providers.

Main takeaways

Building a Deep learning workstation can be a lot of fun and help save on cloud costs. Also it’s a good way to learn more about the hardware powering AI. In addition, having your own AI PC gives more privacy as you do not have to share any data with model providers. On the flip side, it may be a costly experience. The best approach looks to be to start small and iterate.

Please share in the comments your experience of building an AI or Deep Learning workstation.

Listen to the podcast version part 1 and part 2 of this article generated by NotebookLM.