The Hypervisor
One of the first major decisions when creating a home server environment is the choice of hypervisor. A hypervisor allows a single physical server to run multiple virtual machines (VMs) in a process called virtualization, which has a number of benefits. We won't dive into all the benefits here, but briefly: virtualization can increase the efficiency, reliability, and economic feasibility of your homelab.
There are many good options when it comes to hypervisors: Hyper-V, VMware ESXi, Xen, Bhyve, and Linux's KVM are all popular in the homelab community. I've personally used Hyper-V and ESXi in both my home server environment and in the industry. Both are highly functional pieces of software, but can be cost prohibitive depending on your home server setup. While there are technically (feature-limited) versions of both ESXi and Hyper-V available free for individual use, I've moved away from these more traditional options in favor of another solution: Proxmox Virtual Environment.
Proxmox is an open source virtualization management solution built on Debian Linux. It uses KVM (Kernel-based Virutal Machines) and LXC (Linux Containers) to offer both heavyweight and lightweight virtualization options depending on your needs. It also supports ZFS, an incredibly robust filesystem which is widely regarded as one of the most reliable in the industry. Proxmox is controlled largely from its Web GUI, but you can also interface with it directly from the command line if you're comfortable doing so.
Installation
Below is an installation tutorial from Craft Computing, who has lots of good Proxmox and home-server content on his YouTube channel. Keep reading though, as there are some tweaks to Proxmox that he doesn't cover here!
Setup
After installing Proxmox, we need to correctly configure the package manager sources for personal use.
Proxmox offers a subscription, and you won't be able to update or install packages by default unless you pay the subscription fee or enable the community repository.
Comment out the pve-enterprise repository and add the pve-no-subscription repository as detailed on this wiki page.
In versions prior to 7.4, I also like to install PVEDiscordDark, which gives the Proxmox Web GUI a nice dark theme. The default theme in these older versions is bright white, and I've fallen in love with all things Dark Mode™. Save your eyes, and enjoy some late night homelabbing.
As I mentioned previously, one of the great things about Proxmox is its built in support for the ZFS filesystem. Whenever I set up a new Proxmox installation, I almost always create a ZFS storage pool to keep my VMs, containers, and data safe. See this video from Techno Tim for help setting up ZFS and other useful Proxmox features. ZFS is one of the most advanced and robust filesystems in existence, but remember: ZFS or traditional RAID are no substitute for a true backup! ZFS can help prevent against bitrot, faulty drives, and more - but it can't save your data in the event of a true disaster. You should always keep your truly valuable data stored in at least three different places. One these locations should be offsite. Consider investing in cloud storage, or backing up data at a friend or family member's home. Read up on Proxmox Backup and Restore.
Finally, consider installing some useful packages such as tmux and bpytop. These will be helpful when using Proxmox from the command line.
Conclusion
Now that Proxmox is up and running, you can create VMs and containers to your heart's content! Proxmox is very powerful, and we've only just scratched the surface. Check out PCIe passthrough, or further explore the difference between VMs and containers. Most importantly, have fun!