@agente posted a new topic, subject "Benchmark nodes". It read:

   Hi, I have two nodes with two different VPN (I must use it on CGNAT network). One of this node seems very slow during downloading. Speedtest seems ok but I think is a delay or dns problem to slow down everything.  
    My question is: Can I benchmark my nodes? Can I simulate storj gateway test on my nodes?

After which, @SGC added

    personally i might try to switch the vpn's around on the nodes…
    
    that would indicate if one vpn is performing worse than another.  
    if that is not the case then it has to be the one node, that could be on a SMR HDD or something.
    
    latency would usually be an indicator of a bad connection or stressed cpu, hdd or such things…  
    vpn's use encryption and not all cpu's can hardware offload such tasks.
    
    if you are on linux i would recommend netdata it is very good for finding issues.
    
    learn.netdata.cloud
    
    ### Get started with Netdata | Learn Netdata
    
    Download and install the open-source Netdata monitoring agent on physical/virtual servers, Linux (Ubuntu/Debian/CentOS/etc), Docker, Kubernetes, and many others, often with one command.
    
    its an amazing tool and have helped me many times, it can be a bit resource heavy to run on smaller / older systems… so might want to turn it off again when you are done.
    
    i'm sure there are similar good tools for windows, but nothing comes to mind right now.

After which, @agente added

    I dont think I can emulate storj clients with netdata but simply my connection. I need to simulate clients requests understanding if I'm serving files with a decent “reactivity” or not

After which, @Toyoo added

    You could try capturing some incoming packets with wireshark, then replaying them with hping. Your node will obviously fail the request, as crypto will not match, but at least you can benchmark the establishment of the TCP connection/routing UDP and the first reply.

So @Vadim replied

    I think this will be very good featere from storj, if it will be posible to Benchmark nodes from different satelites. then node operators can think what they can do better, or if there is some bottleneck. This will be very healthy for the whole storj system. As node operators will start to understand why their node perform good or bad.

Then @Alexey said

    Data is transferred directly from the customers, not satellites. Measurement from the satellites' location will be useless.

After which, @SGC added

    agente:
    
    > I dont think I can emulate storj clients with netdata but simply my connection. I need to simulate clients requests understanding if I'm serving files with a decent “reactivity” or not
    
    yes netdata is monitoring, but it can usually tell you if there is something misbehaving with the hardware or other odd behavior.
    
    alternatively you could try to do read write and io tests on the HDD's and see how they perform.  
    in most cases of nodes performing worse than one another, its the HDD's that play the major factor even tho other things can impact ingress and downloads.
    
    using storj as a benchmark, tho it can be done using the storj network private version, i don't think anyone ever really tried that… and one would basically have to build a mini storj network to do that… so its just not really viable.
    
    besides, tho it would be cool with an official benchmark tool from storj, thus far we haven't really had much practical use for one, as most stuff can be benchmarked just fine using 3rd party software solutions.
    
    but maybe you are right, its about time we got an official storj software suit recommendation of which 3rd party softwares to use for benchmarking node performance.  
    maybe a bit of a how to manual for the individual tests.

Then @Vadim said

    Then it cold be possibilities to send data to specific nodes for the test prepose. and read them.

So @agente replied

    It will help owners to improve nodes. It's difficult to understand if you are going to “serve” your files fast or not during the whole process from client back to client.  
    It's time to improve our performance to compete with s3

To which @SGC replied

    with programming it could ofc be possible to have like temporary benchmark pieces, and then a node could request a benchmark which other nodes performed on it.
    
    ofc this sort of feature would need some limitations and possible protections against exploits.
    
    and not sure such a feature would be high on storjlabs to do list…  
    also there would be the question of difficulty, as it might involve rather deep changes to code / options the node software provides.

@Alexey replied

    No, it could not. Again, data is transferred from the customers, not satellites. If we run own uplinks for that purpose, you can only measure something for that one location. This benchmark will be useless, It would be like testing your internet speed with Speedtest - many ISP optimizes their routes to get fastest result with Speedtest, this would be the same. You would optimize your nodes for this one location, but will have worse results with the real customers from other locations.

So @IsThisOn replied

    A benchmark would be nice, it would probably show that there is no CPU or HDD bottleneck. Both CPU and HDD utilization is extremely low. I hate SMR drives but to be fair, even this things do not get utilized by storj.
    
    Much more likely would be what @Alexey tries to explain. Data is coming from peers. Ask yourself, how does data get from person A to person B? Assuming person A uses ISP A and person B uses ISP B.
    
    Maybe ISP A and B have a direct connection. That could be a single 10Gbit link that connects ALL customers from ISP A to ISP B. Or they don't even have direct peering and they are only connected by a central exchange like DE-CIX. Which brings you to the next question, what connection does ISP A have to DE-CIX? And again, remember you share that connection with ALL other customers.
    
    All that stuff is called **peering** and I would argue is more important than speed/bandwith.
    
    Lets use a real world example: Downloading Cyberpunk ukeith@example.com. A friend of mine has a 10Gbit Fiber. Another friend has 1Gbit from a different ISP. Guess who was faster? Jup, the 1Gbit one. Why? Bad peering. Your ISP also has internet connections to other networks. CDNs, content providers and so on. So it could be that when you download a game from GOG, you share one single 1Gbit connection from your ISP to the GOG servers with EVERY other customer of your ISP! To keep it short, we ignore cache servers and xgs-pon.  
    That is why in the end my 1Gbit friend got between 800-900mbit download speed, while the poor 10Gbit guy got 3mbit. No, this is not a typo! He really got 3mbit on a 10Gbit fiber connection!  
    **So while his connection in theory is 10 times faster, in real life it was 300 times slower!**
    
    How can you test or benchmark **peering**?  
    Well, that is pretty hard. You can try at PeeringDB but that will not get you far. More realistic is to switch to real ISP that has good peering. How can you spot a good ISP? Not that easy, but for one, they offer real dual stack and not CGNAT. What definitely will not perform well is a VPN. A VPN will either cost way more than you earn with Storj or will be overbooked and have bad peering.
    
    What does **speedtest.net** by ookla really benchmark?  
    If you go this webpage and look at the testserver, you can see that it is probably a server from your ISP. So what you are testing is a iperf speedtest between your device and a server in your “local ISP network”. But more interesting would be, how fast ist the speed between your device and mine? That would be a interesting benchmark. I for example am in Europe and have a 1Gbit fiber connection. My speedtest to Wyyerd fiber in California is 25mbit down and 400mbit up. That gives me a good estimation on how fast a Wyyerd customer can up and download data from me.
    
    To come back to @agente, why do you need a VPN for CGNAT? I assume you have a public IPv6? Does Storj fully support IPv6?

Then @agente said

    Ipv6 works but then you will serve only ipv6 clients, if I understand well.
    
    My thoughts about performance testing is to start from simple and then trying to testing peering conditions. We can just start to test a file request from a specific location (maybe one of the gateways). This can help to understand if you have bottleneck somewhere in your configuration (vpn, smr, usb channel busy from other devices that slowdown everything… etc).  
    I think many nodes operators are not system administrators. We need simply tools for a better storj speed tuning. If speed matters…

To which @IsThisOn replied

    The most important metric to test is peering. You can't test peering because peering is different for every customer. All other metrics are not important because they are not a bottleneck. Your idea to test peering between our node and a gateway would not get you any insights, because you get data from multiple users and not a single gateway.
    
    > If speed matters
    
    That is the thing, it does not   
    It could matter if utilization would be WAY higher.  
    By getting the data from multiple (underutilized) nodes, there is already enough speed.
    
    > Ipv6 works but then you will serve only ipv6 clients, if I understand well
    
    That is correct. Hopefully one day utilization is so high, that you don't need IPv4 users to fill your disks

Then @Toyoo said

    I think there was an idea posted on the forum some time ago to build a benchmark into storage node code. As in, each node would periodically test other nodes and share results, either to a satellite, or directly to the peer node. As there is now ~15k nodes all over the world, each node operator would get a lot of data.
    
    It would be quite a lot of coding though, as well as adding some sort of social contract (how much do you trust other nodes to give true results?).

After which, @serger001 added

    Toyoo:
    
    > would periodically test other nodes
    
    up/dl?  
    who will pay for this?

@IsThisOn replied

    This could implemented with a few lines of code, but why?
    
    So we can waste bandwidth to test how fast my connection to other random nodes is, while these aren't even the guys that will down or upload data from me?
    
    There is wordplay in German for that:  
    Wer misst, misst Mist.  
    Translation: Who measures, measures rubbish

To which @Toyoo replied

    serger001:
    
    > up/dl?  
    >  who will pay for this?
    
    New social contract.
    
    IsThisOn:
    
    > So we can waste bandwidth to test how fast my connection to other random nodes is, while these aren't even the guys that will down or upload data from me?
    
    Yeah. It would still more useful than a synthetic benchmark from a single location.
    
    There's a saying, you never have the right data, so you need to leverage data you have (paraphrased).

Then @serger001 said

    Toyoo:
    
    > New social contract.
    
    And… Who need it? If you have 1 node with 2-5tb, you dont need another benchmark. You can benchmark network and disk with standard utils with google. If you have 10+ nodes and 50+tb, you dont need benchmark, becouse you can (and know how) benchmark your hardware by standard utils.
    
    Network benchmark-torrents.  
    Hdd benchmark - torrents with low seeders and high leechers. And little script for create, read, delete thousands small files

Then @Toyoo said

    I don't know who _needs_ it. I know I'd gladly trade off some of my bandwidth in exchange for some other nodes provide me additional insight into how well my nodes function.

After which, @agente added

    How? I have a node in my home. I buy a VPS and I want to simulate e file request to my home node and check speed of the entire operation.

So @IsThisOn replied

    Easy
    
      * Go to speedtest.net
      * click on “change server”
      * select a random server by searching for a city. As a European I searched for California and clicked on Wyyerd Fiber
      * Get random benchmark numbers that don't mean a lot  
    ???
      * profit 
    
    
    
    Is it a arbitrary number, that does not say a lot? Yes, but the same would be true if you would speedtest against my node.

Then @agente said

    You're going to test from that city to your pc. Yes, it's easy. I've asked something different.  
    I need to test storj node too.

@IsThisOn replied

    City to your PC is what you are asking for, minus CPU and HDD.
    
    CPU is easy, look at the usage. It is simply not there.  
    HDD is easy, look at the usage. It is simply not there. If you really wanna go further, use a benchmark tool like CrystalDiskMark. I asked before on the forum what would be realistic benchmarks settings but got no answers. So I choose a worst case setting and still got 30mbit on my shitty SMR drive which again, exceeds realistic write speeds by far.
    
    Other option would be to use the success rate script, to see if your node fails down or uploads. If that is not the case, you can stop looking for bottlenecks/errors.
    
    I know this sounds very patronizing, but I know it because I went down the same road as you do now. We are unhappy with the numbers and are looking for bottlenecks. But there are no bottlenecks. A toaster could run a node. Only real bottleneck we have is underutilization of Storj. And we discuss a Disneyland phantasyworld where we have to fine tune our ZFS arrays with SLOG and ARC and fusion pools, because demand is so high

After which, @agente added

    If you have nodes installed in VM even changing vm software can improve your latency, or just change some settings. There are multiple layers of latencies in some environment.

Then @IsThisOn said

    And latency is important how? Serious question, if have no idea.
    
    I would assume, if latency is too high, you would loose some upload races and thous not reach 100% in the success script.


{'source': 'discourse', 'url': 'https://forum.storj.io/t/benchmark-nodes/19891', 'title': 'Benchmark nodes', 'views': 249, 'has_accepted_answer': False, 'word_count': 2369, 'like_count': 8, 'reply_count': 18, 'participant_count': 7, 'posts': 26, 'tags': 'multiple-nodes hardware vpn cgnat', 'author': '@agente', 'participants': '@agente, @SGC, @Toyoo, @Vadim, @Alexey, @IsThisOn, @serger001', 'image_url': [None], 'created_at': '2022-10-09T15:42:15.897Z', 'last_posted_at': '2022-10-12T15:35:00.831Z'}