• Welcome to TechPowerUp Forums, Guest! Please check out our forum guidelines for info related to our community.
  • The forums have been upgraded with support for dark mode. By default it will follow the setting on your system/browser. You may override it by scrolling to the end of the page and clicking the gears icon.

Talk: File system of choice / with encryption / with compression / deduplication

Joined
Jul 31, 2024
Messages
1,497 (5.20/day)
This is not real a support topic. More a talk topic

I'm curious on the file system the userbase here uses for their desktop boxes / ...
If possible please provide how you set it up if you want to share.

I use:
Main box:
  • GPT -> LVM2 -> LUKS -> BTRFS + ZSTD compression factor3
  • tmpfs for lots of temporary junk files / lint files
  • swap = none, but 64GiB DRAM
  • ext4 - old unreliable 1.5TB SATAII-HDD downloads = source code / junk (package manager checks checksum anyway for those downloads)
  • vfat - boot: efistub-kernel
Backup:
  • GPT -> LVM2 -> LUKS -> EXT4

Does anyone uses compression, encryption or deduplication?

edit: reformat - forgot about my hdd.
edit2: Deduplication
 
Last edited:
For all of my systems I use GPT layout disks in the following layout:
[GPT]
-> [BOOT] - FAT32
-> [ROOT] - XFS
-> [SWAP] 1:1 RAM

(The only exception is my media server which is the same setup on MBR, since it's an old box).

No compression or encryption. I used to use EXT4 but found it unreliable compared to XFS. I have used BTRFS in the past but it's a bit overkill for my needs so I swapped back to XFS. I have multiple onsite and offsite backups, onsite drives are also formatted as XFS.
 
I can understand compression for archives as long as it doesn't make files become lossy.
 
XFS root. I use SEDs and encrypted containers for my encryption needs. Compression I keep to dedicated archived, no in FS stuff.
 
Windows user here so ...
  • NTFS all day every day. No compression on any volumes. Typically use the 4K default cluster size.
  • Boot/OS volume 500-ish GB drive C. Whatever is leftover on the device as Data volume drive D.
  • NVMe SSD's: Typically I leave some extra free space for overprovisioning.
  • Optanes: Pairs striped in Windows Disk Management (become dynamic volumes)
  • RAM Drive for temporary files, downloads, etc...
  • Even with 64GB to 128GB RAM ... OS managed SWAP file ( sure why not )
Backup: Various levels of NAS, USB, DVD
 
Ext4 because well proven and fast. Will likely switch to Btrfs in the future. Might aswell consider encryption then. Since I don‘t have large amounts of compressible data, I don’t see much point in FS level compression.
 
For backup, just use restic + rclone to S3 (or any other of its gazillion supported storages) and rclone to duplicate the storage bucket to another continent at any other provider, maybe a third copy on something local. Encrypts data by default before it leaves your machine, incremental, deduplication, compressed, you can mount and/or restore any backup partially/individually

On Windows use Duplicati to S3
 
For backup, just use restic + rclone to S3 (or any other of its gazillion supported storages) and rclone to duplicate the storage bucket to another continent at any other provider, maybe a third copy on something local. Encrypts data by default before it leaves your machine, incremental, deduplication, compressed, you can mount and/or restore any backup partially/individually

On Windows use Duplicati to S3
For BSD/Linux I highly recommend Tarsnap as well: https://www.tarsnap.com/

Relatively cheap, easy to set up and automate and very secure. The guy behind it is also extremely helpful, I've emailed him a couple of times for account changes and he's done it without delay. I've been using this for...6 years now? No issues, even when I had my very jank setup for it under WSL2 to back up my Windows box.
 
Relatively cheap
I had to lol .. $0.25 / GB-month

I'm backing up 10 TB, just paid my storage provider $0.0017 per GB per month for it
 
Windows here mostly, so that means NTFS. No compressed files or encryption.

I've had my Linux time but then I usually settled for whatever was default (usually ext4), without any peculiar partition arrangements (so, everything on /, no LVM, etc.). I'm mildly interested in atomic COW, deduplication and such but I find it annoying that most of it is just command line only, with one single GUI tool existing as far I saw back in the day, and already dead development-wise when I heard of it.
 
I had to lol .. $0.25 / GB-month

I'm backing up 10 TB, just paid my storage provider $0.0017 per GB per month for it
To my knowledge the initial upload cost will be expensive but then the maintenance after that will be quite cheap. Still, yeah, you're definitely getting the FAR cheaper option there. It seems my knowledge of online storage costs is not that comprehensive! :)
 
To my knowledge the initial upload cost will be expensive but then the maintenance after that will be quite cheap. Still, yeah, you're definitely getting the FAR cheaper option there. It seems my knowledge of online storage costs is not that comprehensive! :)
The software looks like a solid solution, and if you want to backup only a few gigs, then it's definitely an option. otoh there's free cloud storage options for this size, and with client-side encryption you don't have to worry about them indexing your personal data
 
what's s3's business model? they charge for in-/egress? egress only? so you're only paying them when you actually have to access your backup?
 
what's s3's business model? they charge for in-/egress? egress only? so you're only paying them when you actually have to access your backup?
Depends on the plan you pick. With "S3" I meant S3-compatible storage, of which there's dozens of offerings
 
oh, ic
you're not using amazon itself tho? last i've heard their business model can be quite succinctly summarised as scam but i've not delved too deep into it, there might be plans that are priced acceptably depending on the use-case
(i know you mentioned oracle cloud having some decent-priced plans for some of tpus use-cases)
 
oh, ic
you're not using amazon itself tho? last i've heard their business model can be quite succinctly summarised as scam but i've not delved too deep into it, there might be plans that are priced acceptably depending on the use-case
(i know you mentioned oracle cloud having some decent-priced plans for some of tpus use-cases)
S3 is expensive, yeah, there are storage, ingress and egress costs, but it's one of the best options for large, performant, distributed object storage platforms out there.

For a regular individual its pricing and complexity is indeed non-sense. You can have a cheaper experience with the likes of Backblaze or Wasabi, which also offer S3-compatible storage.
 
ZFS with zstd level 1 and AES-GCM encryption on everything here. I set it on the root FS and everything inherits it.

Desktop = striped mirror of 4 NVMe SSDs in a pool, NAS = mirror of SATA SSDs and a bunch of spinning rust, backups ... ZFS, laptop ... you guessed it.

Bad hardware/firmware/cables/drivers has lost or corrupted data eventually on every other filesystem I've used, even with ECC memory.

What really sold me on ZFS was I had a machine once with Samsung SATA SSDs with a firmware bug that would random corrupt data in the ZFS partitions (basically all over the LBA range) when TRIM was enabled for a different partition - NCQ TRIM commands while other writes in the ATA queue triggered the firmware bug. ZFS kept on finding the corruption and repairing it for months.
 
Back
Top