Do you think we will see a return to SSD/NVMe cached HDD in the LE* world?

AnthonySmith

Looking at some interesting charts today https://uk.pcpartpicker.com/trends/price/internal-hard-drive/#storage.ssdm2nvme.1000

showing around a 60 - 125% (roughly) increase in NVMe prices in just 3 months, and with a few big names like WD recently announcing that they have already sold all stock for 2026 to AI/Top 10 customers, 2027 being close to the same position and its expected all manufacturors will be in the same position within weeks or months and that will extend to 2028 in some cases.... things are going to get MUCH worse some casual bro science reading suggests 1000% increase by 2027.

The thing with NVMe is the 2nd hard market is terrible, you literally cant trust them and NVMe in general have a much shorter life than spinning rust.

I am wondering what this really means, if you think LE hosts are going to be paying $5000 per 1TB NVMe (Not even considering RAM and Power Prices) and selling at $7 you are delusional.

I wonder if we will start to see more NVMe cached options come up, HDD's are still expensive, not NVMe expensive and have a much longer life span.

I know many of you will turn your nose up at this and want low price, fast and reliable, but with the recent OVH and Hetzner corrective steps are you going to be willing to pay double or triple for a VPS or would you rather stick to the cheaper options and reduce your requirements (optimise)

Maybe the $7 is about to self-correct anyway, after all the recent talk about it.

An interesting few years ahead.

yoursunny

$7/year will return to $7/month as it used to be.
Spec will return to 1 core, 256MB RAM, 5GB storage as it used to be.

Chievo

It highly depends. I mean people wanna go with nvme probably paying more. How much ? It is another story . Big players would offer it for "cheaper". I guess they have plenty of hardware at old price . I guess their team have gone wild before the increase and lot of money was invested in hardware at " normal " pricing. I would not expect 1000% increase. Could happen but it would be weird . But true nobody knows what s going to happen tomorrow. For obvious reasons it is going to hurt. Many websites literally would be closed if the hosting is X 10 pricing (VPS or Dedi ) . Sad but true. Let s speak in a year or so about it to see what has happened.

AnthonySmith

Well, to put it another way, no hard drive manufacturers will release any disks to the consumer market or to smaller customers (even people like Digital Ocean OVH and Hetzner are small in this scale) outside their top 10 (actually more like top 6 who are already happy to pay 3x in advance) for all of 2027 and 2028.. NVMe will be worth more than gold in weight.

How many hosts that we know outside the big cloud players AWS/Azure etc do you think are in the top 10 customers for the factories? 0

I guess it can go 1 of 3 ways, people own/have what they have, we have stock or we don't, or a 2GB VPS with 25GB NVMe goes to $70 p/month, or something else.

Chievo

@AnthonySmith said:
Well, to put it another way, no hard drive manufacturers will release any disks to the consumer market or to smaller customers (even people like Digital Ocean OVH and Hetzner are small in this scale) outside their top 10 (actually more like top 6 who are already happy to pay 3x in advance) for all of 2027 and 2028.. NVMe will be worth more than gold in weight.

How many hosts that we know outside the big cloud players AWS/Azure etc do you think are in the top 10 customers for the factories? 0

I guess it can go 1 of 3 ways, people own/have what they have, we have stock or we don't, or a 2GB VPS with 25GB NVMe goes to $70 p/month, or something else.

Speculation bubble . Bubbles are done artificially for short time period . If bubble is maintained in the time usually the side effects would be really really hard . So meh . I would not guess we would see a vps 25 NVMe for 70 dollars because of that. Obviously I could be completely wrong . I agree that could be the case. At the end of the day , it does not depend from us.

someTom

in the total cost for purchasing, installing and running a beefy VPS server for 5 years, how much does the cost of the NVMe drive matter? does it even make a difference?

AnthonySmith

@someTom said:
in the total cost for purchasing, installing and running a beefy VPS server for 5 years, how much does the cost of the NVMe drive matter? does it even make a difference?

yes.

Few years ago, 1TB NVME x 4 = $110 ish each so lets just say $450

Now 2 scenarios.

1. I simply cannot even buy them, or there is a 2-year wait for stock, I have no stock, so I can't even sell you a VPS, only a few places have stock; they put the price up, simple supply/demand.
2. I can get NVMe, but predictions are correct and I now pay $3000 for the same drive that cost $110 and I need 4, per node my costs just went up by $11,500, which is more than the entire server, including ram used to cost.

And let's not even consider the DDR5 300% increase that has already happened and will continue to go the same way.

If you dont think what it costs us within a 1000% margin will have an impact on what it will cost you, I dont even know why we bother with schools any more haha.

host_c

@Chievo is right, we need 6-12 moths to say something, for now, market is as stable as BTC, LTC, XMR or whatever alse cryptocurrency - in other words, too damn volatile. why? because suppliers also smell blood and wish to make a buck on this, good for them, and I would do the same, yet this will not last forever, so just wait a bit more.

@someTom

That really depends on the Type of NVME you use, but either consumer or enterprise or even used enterprise will suck in price today.

Fast math, basic stuff, nothing fancy config wise per node.

Before November 2025:

1. • 1U server, something DDR4 ( ryzen or whatever Xeon Gen 2 ), 2x PSU, 1x CPU, caddies, no drives no nvme, just the barbone unit - 800 USD ( lets assume it is something good and not grandpas E5-VX )
2. • 128 DDR4 - 250 USD
3. • NVME - here I will assume Consumer ones, as generally very few spend 4x the $$ on Enterprise NVME, so... 2x4TB - 600 USD ( 300 USD / unit )
4. • Drives, 10 TB SATA NEW - 250 USD each, so that is 1000 USD total

A good node with new parts ( nvme and HDD ) = 2850 USD ( give or take 10% )

Today February 2026:

1. • 1U server, something DDR4 ( ryzen or whatever Xeon Gen 2 ), 2x PSU, 1x CPU, caddies, no drives no nvme, just the barbone unit - 800 USD - here we have almost no change yet.....
2. • 128 DDR4 - 700 USD to 1000 USD - it really depends if 4x32, 2x64 but let's assume 900 USD ( ~7 USD / GB )
3. • 2x4 TB NVME, Samsung or whatever "established" brand - 1300 USD
4. • 4x 10 TB SATA, NEW - 1250

Same node but with today pricess = 4250 USD

So that is almost x2, and only 2 SKU of that node hiked in value, RAM and NVME. ( drives price hike will hit storage providers, not VPS providers )

If we are talking about DDR5 Nodes, that will be a x4 by today. - OMFG
If we are talking about large storage nodes ( 12-24 bay units ) that will also be x1.5 - x2 today as the price / drives times 24 will make a lot of difference.

And all that in ~4 moths.

now, deploy 3 nodes today ( whatever config ) versus 3 nodes in june 2025.

Also, the gap between enterprise used parts ( NVME, RAM, SAS or SATA drives ) is shrinking rapidly. For some SKU it is not worth anymore to buy used parts as you have no warranty. ( 90 days is not warranty, that is shipping assurance )

So new services don't look good at same price / config as in 2025.

Volatile market, weekly changes in price and availability cannot be quantified as they have no logic.
No logic will lead to a simple answer/fix, make the services 50-100% more, and see what happens, you really do not have any options when you have 2 or 3 variables out of a mix of 4 that are dynamic.

Your acquisitions budget and cash-flow was not designed for what is happening today, so as an operator, you have very few options, regardless the size of your industry.

/2c

HOST-C

AnthonySmith

I think the answer is simple, people are just going to have to earn more money so they can pay more that always works as a solution..... hahahahahaha.

host_c

@AnthonySmith said:
I think the answer is simple, people are just going to have to earn more money so they can pay more that always works as a solution..... hahahahahaha.

Sadly yes....

I did not even mention that for those that have employees, all those fellas, will want a raise in 2026 as....... Boss shit is expensive, I need more. - and the fella is right, so that increase will add to monthly expenses of the operator and later be reflected in the price of the services provided.

I am not saying it is the end of the world, we are far from it and we have seen worse, I am saying that it is not pretty, not pretty at all right now. And sincerely, I have the feeling none saw it to be at this scale, I don't mean small worms like myself, I mean the bigger players.

When stuff grows in price gradually -> people adjust, when they take off.... that is a different story. We just need the time to adjust.

Neoon

Fuck no, I keep everything on SSD and NVMe.
I got 1TB new SSD's in the 2 rigs, they should last through 2027.

But its the same with no IPv4, it needs to be cheaper.
If you can really offer such a better price with HDD cached, its on you.

AnthonySmith

@Neoon said:
Fuck no, I keep everything on SSD and NVMe.
I got 1TB new SSD's in the 2 rigs, they should last through 2027.

But its the same with no IPv4, it needs to be cheaper.
If you can really offer such a better price with HDD cached, its on you.

So, for clarity, even if the VPS price goes up 5x you will pay it to keep it on NVMe ?

I know you personally probably have a ton of prepaid resources, but i mean in general, you need a new VPS in 1 year from now, you see a VPS with 25GB NVMe is now $40 /month at the bottom end or $7 with nvme cached hdd, you still pick the NVMe ?

kako1talk

The AI market eating up the global NAND/NVMe supply is a harsh reality.

To answer your question: I'd take the NVMe-cached HDD route in a heartbeat. The whole spirit of the 'LE' world is price first, specs second. If I have to pay double or triple just to keep pure NVMe, I might as well move to mainstream cloud providers.

Spinning rust with a solid caching layer is still incredibly resilient. Most of us don't need sustained sequential writes anyway. Let the AI bros fight over the NVMe stock; we just need affordable storage.

Neoon

@AnthonySmith said:

@Neoon said:
Fuck no, I keep everything on SSD and NVMe.
I got 1TB new SSD's in the 2 rigs, they should last through 2027.

But its the same with no IPv4, it needs to be cheaper.
If you can really offer such a better price with HDD cached, its on you.

So, for clarity, even if the VPS price goes up 5x you will pay it to keep it on NVMe ?

I know you personally probably have a ton of prepaid resources, but i mean in general, you need a new VPS in 1 year from now, you see a VPS with 25GB NVMe is now $40 /month at the bottom end or $7 with nvme cached hdd, you still pick the NVMe ?

Okay, if the VPS price goes up, the question is, do I keep it, yes or no?
I recently dropped 7$/y boxes because they didn't meet my quality req.

Depends on location, network specs if I keep it and how big the price increase right.
I have no problem dropping them.

I recently got a HDD dedi, in SG though, it was 16€/m for me, steal.
Its 2x2TB HDD only, cached with ECC memory and ZFS, that was a trade I was willing to accept.

But otherwise, I am not willing to accept shitty I/O anymore, except its a backup vps.

AnthonySmith

@Neoon said: But otherwise, I am not willing to accept shitty I/O anymore, except its a backup vps.

This... same for me really, I have a local spinning rust raid 10 for backups, and zfs raid for backup storage on Tierhive, but apart from that NVMe or die tbh.

Neoon

@AnthonySmith said:

@Neoon said: But otherwise, I am not willing to accept shitty I/O anymore, except its a backup vps.

This... same for me really, I have a local spinning rust raid 10 for backups, and zfs raid for backup storage on Tierhive, but apart from that NVMe or die tbh.

On nodes, that run a bunch of VM's you want NVMe, you don't even want SSD.
SSD is fine for personal PC's or servers you use for yourself.
Even HDD for certain use cases.

Falzo

aren't WD and Seagate sold out for their 2026 HDD production already? probably not worth it as you need quite a proper set up array, if it really should be performant. small array with little caching will not hold up to that promise. people already complain if something is limited to 10k iops on 4k because they don't get the difference. yet in high density setups you will want proper limits for balancing.

so no cached storage will not come back at scale. (doesn't mean that some smolish summer host will try...)

host_c

Spinning Rust cannot keep up with Flash based storage @ 4K , not even in higher numbers. ( below Raid10 12 bay node )

Processor  : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
CPU cores  : 80 @ 2500.000 MHz
AES-NI     : ✔ Enabled
VM-x/AMD-V : ✔ Enabled
RAM        : 503.5 GiB
Distro     : Debian GNU/Linux 13 (trixie)
Kernel     : 6.17.2-1-pve
VM Type    : NONE
IPv4/IPv6  : ✔ Online / ✔ Online

fio Disk Speed Tests (Mixed R/W 50/50) 
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 176.16 MB/s  (44.0k) | 1.68 GB/s    (26.3k)
Write      | 176.62 MB/s  (44.1k) | 1.69 GB/s    (26.4k)
Total      | 352.78 MB/s  (88.1k) | 3.37 GB/s    (52.7k)
           |                      |
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 3.87 GB/s     (7.5k) | 3.84 GB/s     (3.7k)
Write      | 4.08 GB/s     (7.9k) | 4.09 GB/s     (4.0k)
Total      | 7.95 GB/s    (15.5k) | 7.94 GB/s     (7.7k)

Same node, NVME

fio Disk Speed Tests (Mixed R/W 50/50)
---------------------------------
Block Size | 4k            (IOPS) | 64k           (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 466.87 MB/s (116.7k) | 975.12 MB/s  (15.2k)
Write      | 468.10 MB/s (117.0k) | 980.25 MB/s  (15.3k)
Total      | 934.97 MB/s (233.7k) | 1.95 GB/s    (30.5k)
           |                      |
Block Size | 512k          (IOPS) | 1m            (IOPS)
  ------   | ---            ----  | ----           ----
Read       | 1.05 GB/s     (2.0k) | 1.08 GB/s     (1.0k)
Write      | 1.10 GB/s     (2.1k) | 1.15 GB/s     (1.1k)
Total      | 2.15 GB/s     (4.2k) | 2.24 GB/s     (2.1k)

someTom

@AnthonySmith said: If you dont think what it costs us within a 1000% margin will have an impact on what it will cost you, I dont even know why we bother with schools any more haha.

i wrote "the total cost for purchasing, installing and running a beefy VPS server for 5 years"

you just compare the hardware component cost and thats it.

i thought there was some more cost involved, but maybe i bothered with school too much.

AnthonySmith

I probably misunderstood sorry, I am not sure if I do now.

All good 👍

someTom

@AnthonySmith said: I probably misunderstood sorry, I am not sure if I do now.

i am guessing your calculation of price per vps offer is _total _cost of running the vps divided by x customers + margin.

which means on top of hardware i guess you pay for housing, power, connectivity, and maybe even wages, for 5-10 years of use.

question is, if 1000 bucks more or less for memory make that big of a difference in TCO that it would influence the enduser price more than 5%.

admittedly my school was a business school ;-)

AnthonySmith

@someTom said:

@AnthonySmith said: I probably misunderstood sorry, I am not sure if I do now.

i am guessing your calculation of price per vps offer is _total _cost of running the vps divided by x customers + margin.

which means on top of hardware i guess you pay for housing, power, connectivity, and maybe even wages, for 5-10 years of use.

question is, if 1000 bucks more or less for memory make that big of a difference in TCO that it would influence the enduser price more than 5%.

admittedly my school was a business school ;-)

Ah ok, sorry, I did missunderstand.

Well, 1000 bucks over 5 years, probably not a killer, but an extra 12,000 bucks over 5 years per node, probably is a killer yes, it would make the difference.

For a few reasons, before it's even paid for, a rack of servers just costs me an extra 250k, cashflow, loans, ROI, yeah its a nightmare.

Obviously this is based on worst case predictions, but if hosts costs go up by 500%, while energy, spares, and everything goes up with it, then its going to have more than a 5% increase on your costs, I think Hetzner and OVH have already shown that, hell even 1password just did a 33% price hike because of their own upstream price hikes.

Everything is increasing at a much higher rate than salaries, the middle classes are driving around in older and older cars, some are now sending kids to public schools (the horror) etc etc.

I feel like we are just seeing the start of a very civilised collapse/apocalypse.

Or... they could all recover next month, in which case, I will be here... back in my box

havoc

I doubt anyone is going into hybrid drives big if both SSD and HDD are short. Hybrids have always been in a rather awkward spot where nobody likes/wants them

In process of grabbing some 4TB enterprise SSDs off ebay. So should be set for 2026 and 2027 for personal use

Less sure as to what I should be doing with online storage. Do storage box prices get auto-adjusted or are they locked in?

WSS

Butts

Lee

@AnthonySmith said: Maybe the $7 is about to self-correct anyway, after all the recent talk about it.

I have enough hoarded RAM/NMVe to hopefully see me through this and keep all my current hardware in Telehouse up and running. With no impact to pricing. Foresight, I seen this coming so started buying while prices were still reasonable.

But, larger providers are already starting to struggle. I have heard that Hetzner for example has constrained capacity due to hardware reserves coming under pressure sooner than later. So it's going to have a real impact on budget providers for sure.

It will rebalance in time, just going to take quite some time, but don't expect prices to return to what they were.

Advin

The main problem is that an NVMe-cached HDD is still insanely slow compared to a pure SATA SSD or an NVMe SSD. It is only marginally better than having a standard HDD array but still incomparable to actual SSD storage.

Even if NVMe pricing went 5x, we would never substitute it with NVME-cached HDD. There just isn't enough IOPS or throughput to host general purpose VMs, especially on very dense hypervisors.

AnthonySmith

@Advin said:
The main problem is that an NVMe-cached HDD is still insanely slow compared to a pure SATA SSD or an NVMe SSD. It is only marginally better than having a standard HDD array but still incomparable to actual SSD storage.

Even if NVMe pricing went 5x, we would never substitute it with NVME-cached HDD. There just isn't enough IOPS or throughput to host general purpose VMs, especially on very dense hypervisors.

What if it went 50x ?

host_c

@AnthonySmith said:

@Advin said:
The main problem is that an NVMe-cached HDD is still insanely slow compared to a pure SATA SSD or an NVMe SSD. It is only marginally better than having a standard HDD array but still incomparable to actual SSD storage.

Even if NVMe pricing went 5x, we would never substitute it with NVME-cached HDD. There just isn't enough IOPS or throughput to host general purpose VMs, especially on very dense hypervisors.

What if it went 50x ?

Even if it went 100X

Yet, for the sake of this discussion, I will provision a 24 bay large HW raid 10 array backed by large controller cache in ~2 weeks and run some tests. It might behave like a SSD platform, but never as a NVME.

Hikes in prices for NAND storage will reflect itself in product price rather then in alternatives, as for today modern needs, you don't have options for +150.000 IOPS / storage media, an you kinda need at least 5-15K on a boot drive so the VPS feels snappy.

Yet, @AnthonySmith you also have a point, for normal work loads, some websites, small databases, casual VPS that does proxy or vpn, you are perfectly fine with a fast spinning rust raid array. Unfortunately there is a "trend" "show off" performance in yabs/or other regarding IO speeds, so those offers are most of the time considered "slow", yet there should be a definition and understanding of "slow" versus use case.

Not all users are Power Users, some don't wish or care to understand the difference per use case on the above, as a result operators are "forced" to provide NVME just to look good, even if sometimes that is not necessary for the work-load.

This is especially valid for the small 5 to 30 GB NVME VPS. You are short in space here to do complicated work that needs high amount of IOPS, you would be much better off with a 200 GB fast RAID10 array on spinning rust, at least you would gain a larger usable capacity for the same amount of $$ payed on a service.

Cheers!

somik

Running RAM cached HDD server for personal use. Works fine.

Would not use it to host 10+ VMs though. If it's a website, then caching is fine.

Advin

@AnthonySmith said:

@Advin said:
The main problem is that an NVMe-cached HDD is still insanely slow compared to a pure SATA SSD or an NVMe SSD. It is only marginally better than having a standard HDD array but still incomparable to actual SSD storage.

Even if NVMe pricing went 5x, we would never substitute it with NVME-cached HDD. There just isn't enough IOPS or throughput to host general purpose VMs, especially on very dense hypervisors.

What if it went 50x ?

I would sell the company if it actually went that high. I would have a couple of million dollars and could retire. /s

I mean, yes, if it did somehow 50x in price, then HDDs are the only viable option. But I do not think it is a realistic scenario. Perhaps at that point, I would still consider purchasing NVMe, but only offer small quantities alongside bulk HDD storage, as many storage VPS providers do these days.

havoc

As a side note I reckon there is a good gap for a provider that can figure out how to do hybrid at filesystem level rather than hardware. Think zfs with mix of ssd & hdd. Very easy to do on lxc, less confident about vm because most (all?) virt solutions seem to do block writes

I've been toying with HDD + those really crappy 16gb optanes for metadata and while the benchmarks disappoint (minimal gain) the lived experience has actually been a noticable improvement. Cause what I used to notice on HDDs is open a folder and it takes 2 seconds to spin up the drive to show contents. So even if throughput doesn't move an inch that 2 seconds of staring at a folder waiting is what registers psychologically

AnthonySmith

Well I remember paying over $1000 for a 1TB NVMe not THAT long ago, market pressure is set to go way higher than it was when 1TB was a big NVMe so.... Interesting times ahead.

host_c

@havoc

I have played with 2x 24 SSD ( 128 to 256 GB ) in DELL MD1220 + Optane or NVME as chache, 256 GB RAM, high freq CPU and whatever you can imagine you can tweak in zfs. Used Debian, Ubuntu, Freenas, TrueNAS Core/Scale.

same for 12 or 24HDD in DELL MD1200.

Sincerely, total waste of time.

The second you hit it with random HIGH IO from VM's over 2x10GBPS or 2x25GBPS it will utterly suck.
Storage exported via ISCSI was much faster then NFS, but I did not fall of my chair performance wise.

Other the the ability to extend a pool on the fly by replacing drives with larger ones and the strong data integrity ZFS gives you, I personally see no real value in it for our use case.

/2c after burning far too many hours in tests.

ZizzyDizzyMC

@host_c said:
@havoc

I have played with 2x 24 SSD ( 128 to 256 GB ) in DELL MD1220 + Optane or NVME as chache, 256 GB RAM, high freq CPU and whatever you can imagine you can tweak in zfs. Used Debian, Ubuntu, Freenas, TrueNAS Core/Scale.

same for 12 or 24HDD in DELL MD1200.

Sincerely, total waste of time.

The second you hit it with random HIGH IO from VM's over 2x10GBPS or 2x25GBPS it will utterly suck.
Storage exported via ISCSI was much faster then NFS, but I did not fall of my chair performance wise.

Other the the ability to extend a pool on the fly by replacing drives with larger ones and the strong data integrity ZFS gives you, I personally see no real value in it for our use case.

/2c after burning far too many hours in tests.

I spent about 4 weeks tuning and exporting storage over 40g from truenas using iscsi and the performance in proxmox was always abysmal, switched to nfs export and got 7gbps throughput (up from 1.3) and a 5x increase in available iops.

Any idea what I did wrong?

havoc

@host_c said:

Sincerely, total waste of time.

You do seem better equipped than me on hardware and I don't doubt that you're in a better position to test this, but I think you may have missed the plot here

Point of my post was that throughput doesn't measure real experience. So if you hit me with "The second you hit it with random HIGH IO" then I think you didn't read my post

host_c

@ZizzyDizzyMC said: I spent about 4 weeks tuning and exporting storage over 40g from truenas using iscsi and the performance in proxmox was always abysmal, switched to nfs export and got 7gbps throughput (up from 1.3) and a 5x increase in available iops.

Any idea what I did wrong?

ISCSI writes by default are synchronous/SYNC, you most probably mounted NFS ASYNC, hence the 3-7X difference, that slow performance over ISCSI is the real performance of your raid array sincerely ( I presume 4 disks? ) , ASYNC is RAM CACHED, SYNC is flushed to the disk in real time + ACK that it got written back to the node - much much slower yet much much safer integrity wise in case a power outage / kernel stalls on both ends.

iSCSI (sync) = disk performance - the hard core truth
NFS (async) = RAM performance - the nice multi GBPS you wish to see, but it is a lye

NFS can also be SYNC, in that case it will behave lice ISCSI SYNC, yet most use it ASYNC.

Also, Promxox ISCSI implementation is a bit ...... not the best in my view compared to vMware or Microsoft ( no offense folks, it is the truth )

Atention!!!!!!!!!!!!
Proxmox does NOT have a clustered filesystem like VMFS5/6.
iSCSI is block access, like physically having those disks in the server you mounted them on. ( well not disks, exported LUN )
It is NOT file-system-aware!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
If you mount the same LUN on 2 servers, bingo, they will write data on each-other's blocks. There are work-arrounds like GFS2 yet, it hardly worth the effort.

I would personally avoid NFS for VM storage as well!!!!!

Why:
Proxmox mounts NFS persistent, if the NFS share becomes unavailable (NAS reboot, switch reload, VLAN issue, usual stuff that can happen), the mount can enter D state. - you are fuked, and this is a node reboot fix only as the kernel will wait indefinitely.

Over time, the node may start misbehaving (hung tasks, blocked I/O, GUI lag, usual shit-show )

Yes, that persistence is part of NFS consistency model and that is a very good thing, but operationally it can be painful, and once you start hitting it a few times and your cluster crashes because the ISO repository is not accessible anymore, you will wipe it out completely from your config, we did.

ISCSI was designed for shared storage to phisical server, that is why you even have ISCSI built in the lan card on servers so you can boot the OS from ISCSI while having 0 drives in the server

Also, I will be an ass, but........
Depending on the config:
NFS outperforms iSCSI
iSCSI outperforms NFS
Both perform terribly in my view as are a bit tricky to maintain.
If you want simplicity -> local RAID still wins in latency.

Small note:

Fibre Channel was purpose-built for SAN delivered the most stable and ow latency shared storage - and I loved it as it truly was a set up and forget setup.

@havoc

Sorry for that, I did miss by a mile, and that was not my point. I thought you encountered the same issues as we did, expecting wow, while getting mehhhhh.

ZizzyDizzyMC

@host_c said:

@ZizzyDizzyMC said: I spent about 4 weeks tuning and exporting storage over 40g from truenas using iscsi and the performance in proxmox was always abysmal, switched to nfs export and got 7gbps throughput (up from 1.3) and a 5x increase in available iops.

Any idea what I did wrong?

ISCSI writes by default are synchronous/SYNC, you most probably mounted NFS ASYNC, hence the 3-7X difference, that slow performance over ISCSI is the real performance of your raid array sincerely ( I presume 4 disks? ) , ASYNC is RAM CACHED, SYNC is flushed to the disk in real time + ACK that it got written back to the node - much much slower yet much much safer integrity wise in case a power outage / kernel stalls on both ends.

iSCSI (sync) = disk performance - the hard core truth
NFS (async) = RAM performance - the nice multi GBPS you wish to see, but it is a lye

NFS can also be SYNC, in that case it will behave lice ISCSI SYNC, yet most use it ASYNC.

Also, Promxox ISCSI implementation is a bit ...... not the best in my view compared to vMware or Microsoft ( no offense folks, it is the truth )

Atention!!!!!!!!!!!!
Proxmox does NOT have a clustered filesystem like VMFS5/6.
iSCSI is block access, like physically having those disks in the server you mounted them on. ( well not disks, exported LUN )
It is NOT file-system-aware!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
If you mount the same LUN on 2 servers, bingo, they will write data on each-other's blocks. There are work-arrounds like GFS2 yet, it hardly worth the effort.

I would personally avoid NFS for VM storage as well!!!!!

Why:
Proxmox mounts NFS persistent, if the NFS share becomes unavailable (NAS reboot, switch reload, VLAN issue, usual stuff that can happen), the mount can enter D state. - you are fuked, and this is a node reboot fix only as the kernel will wait indefinitely.

Over time, the node may start misbehaving (hung tasks, blocked I/O, GUI lag, usual shit-show )

Yes, that persistence is part of NFS consistency model and that is a very good thing, but operationally it can be painful, and once you start hitting it a few times and your cluster crashes because the ISO repository is not accessible anymore, you will wipe it out completely from your config, we did.

ISCSI was designed for shared storage to phisical server, that is why you even have ISCSI built in the lan card on servers so you can boot the OS from ISCSI while having 0 drives in the server

Also, I will be an ass, but........
Depending on the config:
NFS outperforms iSCSI
iSCSI outperforms NFS
Both perform terribly in my view as are a bit tricky to maintain.
If you want simplicity -> local RAID still wins in latency.

Small note:

Fibre Channel was purpose-built for SAN delivered the most stable and ow latency shared storage - and I loved it as it truly was a set up and forget setup.

@havoc

Sorry for that, I did miss by a mile, and that was not my point. I thought you encountered the same issues as we did, expecting wow, while getting mehhhhh.

Okay this might actually make sense, my storage cluster is backed by 1TB of ram, and so far no client has managed to saturate it.
Also NFS mount has held stable for several hundred days.
Only reason the server doesn't have local storage is because I purposely split storage and compute, as it was cheaper that way to maintain quality service. Still is even with price increases in hardware - but most people aren't at PB scale like me,

imok

I use storage too 😀

host_c

@ZizzyDizzyMC said: Only reason the server doesn't have local storage is because I purposely split storage and compute

We did start out the same in this case. And yes, It s much more "smart" that way, you only deal with 1 or 2 large storage units. - that is a big plus.

I also agree that it is stable, till one day it is not. Furthermore, if you do not use MLAG or MPIO, you have a hard time when you need to do FW updates on the switch or have an actual network error, and error I mean human error in the DC staff that unplugged the wrong SFP during an install or an upgrade - that happens more then some think. Or a DAC/AOC/SFP module just dyes, or a NIC fails.

Also, a thing I learned while working in Corporate, there is a reason Real Storage appliances like EMC, NetAPP or other have 2x controllers, the expansion shelves have 2x controllers and in this use case use SAS drives that re DP ( dual port ). In case a controller goes down, the other one takes over, and you do not loose the storage that is shared via whatever protocol you choose. ( sharing part involves redundant network setup, but you also need redundant access for the storage itself )

The main issue if you do centralized storage and not use enterprise purpose built systems, the failure of any 1 part in the mix ( switch, storage, shelf etc..... ) will lead to silent data corruption on all the vm's that use that share on the nodes.

We tested this out with Proxmox = Not pretty, all dyed, vMware is much more "smart" on this as it will put the VM in instant standby until the storage comes back online ( proxmox did not, yet we did not test this on 9.1 ).

Since we switched to Proxmox, for the above reasons we moved away from centralized storage to local one. If one SRV fails, then you have to deal with that one server + customers, you basically limit your blast radius.

I am not saying your setup is wrong, please don't take it like that, I am just saying that the risk in my view, does not worth it. Too many moving parts in the mix, and Proxmox is not for this. If we would have remained on vMware, then yes, we would have stayed on centralized storage also.

Cheers!

EDIT:

I also learn to presume that eventually 1 part will always fail, as an example, if a memory module dyes in the storage unit I built , I have to take offline all the VM's to change that module. Not pretty. Hence I personally prefer an as modular setup as possible. This is a personal decision, not a best practice or a recommendation.

beagle

I saw some chatter on the TrueNAS forums that NVME-oF may bring some performance to HDD arrays.

Being a more modern protocol it seems to be more efficient and a natural substitute to the ageing iSCSI.

chengzi

I am running bcachefs with two enterprise NVMe (for btree + write back cache) + HDDs, works fine for me

Glad I bought the two enterprise 4TB NVMe second hand for $350 last year