So I have a bit of a tricky one that been puzzling me for some time now.
If using (samba) shared folders created on the NAS I get great speeds, sustained read around 950mbps and write up to 850mbps with no unusual spikes/dips.
Now if I throw iSCSI into the mix, connecting the LUN on a Server 2012 R2 system, I still get similar read speeds, but writes fluctuate constantly from a few kbps to a peak of 35mbps, sometimes even appearing like its doing nothing (network throughput stats in task manager,) this is irrespective of the data transfer being small or large files.
I've spent ages trying to explore some of the options in iSCSI Initiator but can't find anything that seems to be affecting the performance, I'm not using encryption as I'm aware that would reduce performance further.
Enabling Windows write caching on the mounted LUN does give me the "illusion" of a speed boost up to 800mbps in the file copy dialog for a short while but then tapers off to a halt, all the while network stats still show the same 35mbps peak and then dropping to zero and fluctuating.
The performance of shared folders on the NAS itself indicates to me there's no bottleneck on the HDD's/NAS/LAN, so not sure whats going on here.
What are you using on the NAS side? what format is it? zfs? what OS are you using?
does it have a lot of ram?
The Windows iSCSI driver is notorious for that kind of thing. Try to find a different driver from a third party.
That's the best in the business.
Good news: we make one indeed and it's at least faster compared to MSFT one.
Bad news: Microsoft banned our design (monolithic SCSI port, undocumented one they use for own storage drivers but for third-parties they promote completely different port-miniport design) from certification so we can't logo our initiator... One big reason we moved it to "Legacy" tools and don't really support anymore. So OP is encouraged to give it a try but we can't help if something goes wrong.
Are you connecting to the NAS iSCSI via a dedicated network? My experience has been that sharing the regular network with iSCSI give horrible results. It doesn't need to be anything fancy or expensive - I have an iSCSI network running over a $60 unmannaged Netgear GS108, and it's been going strong for over 6 years!
I would say that a NAS is not the best for iSCSI. I've tried a few of them, and they never give great performance. If you really need to use iSCSI for performance reasons, a SAN is going to give you the speed. You'll have to pay for it, too, so you'll have to decide if the price is worth the performance.
You have to chop it with VLANs and configure QoS: heavy load spikes increase latency and result dropped SCSI commands with a long hops.
@OP: I'm curious how the performance would be if you direct connected the NAS to the server. The initiators in WS2012 R2 and 2016 are much better than prior iterations, though some 3rd party solutions can still perform better. I feel like there's a piece of information about the way the NAS is being used that we're missing.
I didn't notice any changes... Same old code even not being aware there are multiple CPU cores and I/O should be NUMA bound.
Based on some terribly designed testing we've done, I've seen iSCSI perform acceptably with Synology units with 2 switches in between and on a flat network. But it was a single thick-provisioned LUN on top of the whole array. Not a lot to break.
Get 40-100 GigE NIC, 20+ core CPU, open a Task Manager and enjoy how "nice" will cores jump up and down when I/O thread from iSCSI initiator is re-scheduled to run on another core (and NUMA node as well).
Thanks guys for all the feedback, and sorry I've taken so long to reply.
What are you using on the NAS side? what format is it? zfs? what OS are you using? does it have a lot of ram?
No discernible difference in performance though.
I feel like there's a piece of information about the way the NAS is being used that we're missing.
Ok let me just clear the air here, this whole setup isn't a production environment, rather my personal test lab at home. That said it should be noted that everything here is the very definition of budget, hehe.
yeah... sounds like its ether a consumer end NAS or a really old computer you converted to a NAS? ether way, It's like you said, It is probably just because it's not the greatest machine in the world. Reads are easy for anything to process but if there's not very much RAM and the cpu is slow, your gonna have a bad time. I would recommend looking into a new machine and just swap over the hard drives. Could have something to do with using EXT4 also, As this is the most basic way to do a NAS. You will find that switching to something like ZFS will improve performance, but you would need lots more RAM.
EDIT: also since iSCSI is a SAN protocol, it is a little weird to be having that working with a NAS. I am guessing it is primarily a NAS and as such might be using some kind of odd trickery on the back-end to get a SAN protocol working and that is eating up a lot system resources. Just a guess though.
yeah... sounds like its ether a consumer end NAS or a really old computer you converted to a NAS?
aye, look what I found. Good read.
looks like it could be definitely related to the low specs of the NAS as you already ruled out the switch.
Maybe you could look into getting a refurb Server on ebay? I just got a crazy good deal on a dual CPU with 96gb of RAM for $300 I bet you could fine one that would work for you for like $100 or less.