Do you like benchmarking stuff? Do you want someone else to pay for the servers? enquire within

InceptionHosting

Looking for someone or a handful of someones who genuinely enjoy benchmarking stuff, beyond the standard dd test that is.

I am ideally looking for people who already know the latest cool benchmark scripts, are willing to try new things and do or at least learn how to benchmark networks and latency in a meaningful way (methods can be discussed/provided)

If you have some experience benchmarking, get a kick out of doing it and would like to do it without having to pay for the servers yourself then drop me a PM or register your interest in the comments below.

The plan/idea is to scrape the depths of the internet for lesser-known hosts and odd locations to see what is available and bring some unique finds to the community.

If there is no interest in this I will do it any way but I thought it would be a good to have some community involvement in this.

Cheers.

Ant.

ehab

@Neoon is your guy but beware he doesn't take s--

InceptionHosting

@ehab said: @Neoon is your guy but beware he doesn't take s--

yeah but if something is not quite right, or god forbid, blinks!, he is like:

PHP_Backend

@AnthonySmith said: are willing to try new things and do or at least learn how to benchmark networks and latency in a meaningful way

While I don't have any experience on this bench staff, but I do want to learn

Neoon

@ehab said:
@Neoon is your guy but beware he doesn't take s--

I am to lazy, however, put that login data into a input form, that outputs it to json and a bot will be doing the benchmarks for you.

Iroshan464

Would love to do this.
Unfortunately, I am occupied with some online courses these days. ?
Trying to do LFS201 certification.

InceptionHosting

@Neoon said: I am to lazy, however, put that login data into a input form, that outputs it to json and a bot will be doing the benchmarks for you.

InceptionHosting

@Iroshan464 said: LFS201 certification.

nice one. hope it goes well.

sureiam

Not sure I have the free time currently for this but might in about a month or two. However I love the idea as a lot of reviews for me lack proper network testing. There are many applications and solutions that depend on solid routing to keep things running well. Such as voip for example. In fact many VPS have enough performance to get by but without a solid network and peering are borderline useless. Frankly what's the point of having a datacenter virtual system if I could just connect a decent low power single board Celeron with a raid 1 enclosure into a UPS at home and exceed what I would get elsewhere?

More network testing please!

_MS_

@AnthonySmith said:
I am ideally looking for people who already know the latest cool benchmark scripts, are willing to try new things and do or at least learn how to benchmark networks and latency in a meaningful way (methods can be discussed/provided)

So, signing into the server, running Bench.sh, YABS, Nench.sh, etc., and posting the benchmarks? Yes?

seriesn

Let me tag some of the bench junkies:

@vyas
@cybertech
@Mason
@sonic

vyas

@Neoon said:

@ehab said:
@Neoon is your guy but beware he doesn't take s--

I am to lazy, however, put that login data into a input form, that outputs it to json and a bot will be doing the benchmarks for you.

@Neoon
Will that be micro bots, or meta bots, or nanobots?

@seriesn
Thanks for the mention.

@AnthonySmith
Sounds like a cool idea, maybe puts a method to (my) madness. Count me in, a review or two here and there should be doable

Neoon

@vyas said:

@Neoon said:

@ehab said:
@Neoon is your guy but beware he doesn't take s--

I am to lazy, however, put that login data into a input form, that outputs it to json and a bot will be doing the benchmarks for you.

@Neoon
Will that be micro bots, or meta bots, or nanobots?

@seriesn
Thanks for the mention.

@AnthonySmith
Sounds like a cool idea, maybe puts a method to (my) madness. Count me in, a review or two here and there should be doable

AntonBenchBot.
BAaS.

Neoon

@AnthonySmith said:

@ehab said: @Neoon is your guy but beware he doesn't take s--

yeah but if something is not quite right, or god forbid, blinks!, he is like:

Lies.
There is a reason why the LXD wrapper I wrote for microLXC is called LSD and not LXD.

Because you need drugs, to get use to that shit API.

seriesn

@Neoon said:

@AnthonySmith said:

@ehab said: @Neoon is your guy but beware he doesn't take s--

yeah but if something is not quite right, or god forbid, blinks!, he is like:

Lies.
There is a reason why the LXD wrapper I wrote for microLXC is called LSD and not LXD.

Because you need drugs, to get use to that shit API.

Could have been LCD too

Neoon

@seriesn said:

@Neoon said:

@AnthonySmith said:

@ehab said: @Neoon is your guy but beware he doesn't take s--

yeah but if something is not quite right, or god forbid, blinks!, he is like:

Lies.
There is a reason why the LXD wrapper I wrote for microLXC is called LSD and not LXD.

Because you need drugs, to get use to that shit API.

Could have been LCD too

seriesn

@Neoon said:

@seriesn said:

@Neoon said:

@AnthonySmith said:

@ehab said: @Neoon is your guy but beware he doesn't take s--

yeah but if something is not quite right, or god forbid, blinks!, he is like:

Lies.
There is a reason why the LXD wrapper I wrote for microLXC is called LSD and not LXD.

Because you need drugs, to get use to that shit API.

Could have been LCD too

sanvit

Would love to learn as well.

Neoon

@seriesn said:

@Neoon said:

@seriesn said:

@Neoon said:

@AnthonySmith said:

@ehab said: @Neoon is your guy but beware he doesn't take s--

yeah but if something is not quite right, or god forbid, blinks!, he is like:

Lies.
There is a reason why the LXD wrapper I wrote for microLXC is called LSD and not LXD.

Because you need drugs, to get use to that shit API.

Could have been LCD too

bikegremlin

Very interested to learn, but very tight with time.
Can't even put my finger on when it will be possible (hoping for another corona-virus lockdown ).

serv_ee

Just prime95 them.

Jokes aside. Use PTS, it isn't meant for servers exactly but it can bench them as well.

seriesn

@AnthonySmith here is my 7 cents, how about anonymous provider review?

Chosen and trusted members of the community will signup for providers at a perticular time frame, use service for a month or 2 and share review.

Maybe a thread where providers can optin, so if our review (good or bad) gets posted, we agree to refund the service amount? Or a community funded sponsorship, where the reviewer is reimbursed for the amount from a fund where we can all chip in?

Till the refund time kicks in, it wouldn't be too easy to track users down with who is what and where.

Would be nice to see some true and real reviews and not just numbers, chairs and tables?

Should be good for seo and good KB for the community.

InceptionHosting

yeah maybe, sounds like a pita to keep organised though

seriesn

@AnthonySmith said:
yeah maybe, sounds like a pita to keep organised though

Spreadsheet? I don't mind helping out if needed . Pretty sure I have an hour or 2 to spare every day:)

Mr_Tom

@seriesn said: Chosen and trusted members of the community will signup for providers at a perticular time frame, use service for a month or 2 and share review

Is that to prevent a provider putting the "known" tester on an empty node?

I'm sure a similar idea was discussed previously somewhere?

vyas

@seriesn said:

@AnthonySmith said:
yeah maybe, sounds like a pita to keep organised though

Spreadsheet? I don't mind helping out if needed . Pretty sure I have an hour or 2 to spare every day:)

Good suggestion

james50a

Id be down to help. While benchmarks are fun some of what id be interested in testing is how dif providers handle sustained loads instead of just a single burst.

InceptionHosting

@james50a said: id be interested in testing is how dif providers handle sustained loads instead of just a single burst.

Any sensible one would suspend you and offer you some dedicated server options

james50a

@AnthonySmith said:

@james50a said: id be interested in testing is how dif providers handle sustained loads instead of just a single burst.

Any sensible one would suspend you and offer you some dedicated server options

ofc but its just how far can you push it at what %. whole point of it is for testing anyways and its just another datapoint that people may find usefull.

seriesn

@Mr_Tom said:

@seriesn said: Chosen and trusted members of the community will signup for providers at a perticular time frame, use service for a month or 2 and share review

Is that to prevent a provider putting the "known" tester on an empty node?

I'm sure a similar idea was discussed previously somewhere?

Yeah more or less. Prevent gaming?

seriesn

@james50a said:

@AnthonySmith said:

@james50a said: id be interested in testing is how dif providers handle sustained loads instead of just a single burst.

Any sensible one would suspend you and offer you some dedicated server options

ofc but its just how far can you push it at what %. whole point of it is for testing anyways and its just another datapoint that people may find usefull.

The limit usually exists on their TOS/AUP

havoc

@AnthonySmith said: would be a good to have some community involvement in this.

Sounds like a plan. Willing to kick in a bit of cash too if a suitably safe method can be found. (via inception preferably)

I'd personally be super interested in having some of the bigger players listed as reference too. I know LES intuitively is better bang per buck, but can't really visualize how it compares to say vultr and say GCP.

Also think it would make sense to work out how to aggregate this meaningfully before everyone goes missioning off.

corbpie

I have a parser for a few popular benchmarks that cleans up the results into a Json file for easier viewing/inserting into DB

InceptionHosting

https://lowendspirit.com/pq-hosting-moldova-netherlands-russia-latvia-ukraine-and-hong-kong-unmetered-bw

This contains the sort of benchmarks I was referring to, I tried to get a real world view of what performance I could actually expect rather than a point in time snapshot of performance.

cybertech

@AnthonySmith said:
https://lowendspirit.com/pq-hosting-moldova-netherlands-russia-latvia-ukraine-and-hong-kong-unmetered-bw

This contains the sort of benchmarks I was referring to, I tried to get a real world view of what performance I could actually expect rather than a point in time snapshot of performance.

What a good find. "Hong King" for EUR 3,00

vyas

@AnthonySmith said:
https://lowendspirit.com/pq-hosting-moldova-netherlands-russia-latvia-ukraine-and-hong-kong-unmetered-bw

This contains the sort of benchmarks I was referring to, I tried to get a real world view of what performance I could actually expect rather than a point in time snapshot of performance.

As @SmallWeb says...
Noice!

A bit of spellchecking (comments in writers corner) and we should be off to the races.

Mason

@AnthonySmith said:
https://lowendspirit.com/pq-hosting-moldova-netherlands-russia-latvia-ukraine-and-hong-kong-unmetered-bw

This contains the sort of benchmarks I was referring to, I tried to get a real world view of what performance I could actually expect rather than a point in time snapshot of performance.

Good write up, Ant!

Just a quick thought I had -- for your extended fio tests, you're limiting the block size to just 64k, when in reality the block size will likely be higher as the system tries to use what is most optimal. The yabs test shows the disk performs very well for the higher block sizes (512k or 1m), which is also using fio for non-sequential, random mixed read and writes. So I'm not entirely sure the bs=64k fio tests are really representative of system performance.

InceptionHosting

I felt 64k would be more representative of real world workload, but now I type that, I have no rational behind it so I will do some further reading.

I am not looking for maximum performance, I am looking for real work load performance or as close as I can get when benchmarking.

InceptionHosting

I would like to say hong king is a joke based on British ownership, but it was just a typo

vyas

hong king

Hong Kong

King Kong

-FTFY

InceptionHosting

@Mason ok so upon doing some reading I should probably pick 4k or 512k going forward, I will probably go with 4k unless anyone thinks different?

PHP_Backend

@cybertech said:
What a good find. "Hong King" for EUR 3,00

Even better, There is one for 1€ (stocked out).
Someone should make a scrapper for this one

Mason

@AnthonySmith said:
@Mason ok so upon doing some reading I should probably pick 4k or 512k going forward, I will probably go with 4k unless anyone thinks different?

If given the option, I'd opt for 512k.

If you look through the YABS results thread, nearly every single test does better (in bandwidth terms) with bs=512k over bs=4k. The low block sizes are really just to stress test how many IOPS you can get out of the machine. From the light reading I did when researching this problem many months ago, the disk controller and caching is going to auto-optimize the block size anyway and adjust as necessary to get the best throughput, with the direct=1 flag, fio is bypassing these mechanisms so setting the block size that low (4k) seems not too realistic (at least to me. open for debate).

And that's essentially the reasoning behind why I put multiple fio tests in the yabs script with increasing block sizes, so that the trade off between raw bandwidth and IOPS can be seen.

InceptionHosting

@Mason said: nearly every single test does better (in bandwidth terms) with bs=512k over bs=4k.

I am thinking in terms of real-world uncached random file access and database reads, which is going to account for a huge majority of real-world use cases, one of the studies I read over was by EMC (SAN makers) and the report concluded that the difference between 4k and 512k was essentially just a margin of error and then I made the assumption that 4k is more likely to be aligned for simulating actual use in most cases.

It's an interesting one, I honestly don't feel qualified to say for sure 1 way or the other but I will keep researching, I wonder if I can somehow get stats from a number of VPS nodes to see what is most aligned 4,64,512k or 1m

Mason

@AnthonySmith said: I am thinking in terms of real-world uncached random file access and database reads, which is going to account for a huge majority of real-world use cases, one of the studies I read over was by EMC (SAN makers) and the report concluded that the difference between 4k and 512k was essentially just a margin of error and then I made the assumption that 4k is more likely to be aligned for simulating actual use in most cases.

It's an interesting one, I honestly don't feel qualified to say for sure 1 way or the other but I will keep researching, I wonder if I can somehow get stats from a number of VPS nodes to see what is most aligned 4,64,512k or 1m

Could be the case! It probably all boils down to what is actually being represented in the test. Are we copying a large video file or making many database updates to many distinct tables, rows? I'd imagine the fio parameters would vary widely in trying to represent those two cases. (just some food for thought )

Edit:

And... just so I can pick on you and play devil's advocate...

report concluded that the difference between 4k and 512k was essentially just a margin of error

If that was the case, I'm shocked the results vary so widely. You're getting way different throughput depending on which block size you go with.

more likely to be aligned for simulating actual use in most cases

If we're simulating real use case, you'd probably want to use a mixed R/W test rather than a pure read or pure write test. yabs uses a 50/50 split, but real-world is probably more like 60(R)/40(W) or 65(R)/35(W). Again, just more food for thought

InceptionHosting

@Mason said: If we're simulating real use case, you'd probably want to use a mixed R/W test rather than a pure read or pure write test. yabs uses a 50/50 split, but real-world is probably more like 60(R)/40(W) or 65(R)/35(W). Again, just more food for thought

makes total sense, I think given that, perhaps the best thing to do is rather than a separate read and write test I do a 50/50 split as you suggested and do 1 at 4k and 1 at 512k and graph them both.

Mason

@AnthonySmith said:

@Mason said: If we're simulating real use case, you'd probably want to use a mixed R/W test rather than a pure read or pure write test. yabs uses a 50/50 split, but real-world is probably more like 60(R)/40(W) or 65(R)/35(W). Again, just more food for thought

makes total sense, I think given that, perhaps the best thing to do is rather than a separate read and write test I do a 50/50 split as you suggested and do 1 at 4k and 1 at 512k and graph them both.

Sounds like a good idea to me! Would make some interesting graphs if you plot the read, write, and cumulative speed results independent of each other on the same graph.

InceptionHosting

@Mason said:

@AnthonySmith said:

@Mason said: If we're simulating real use case, you'd probably want to use a mixed R/W test rather than a pure read or pure write test. yabs uses a 50/50 split, but real-world is probably more like 60(R)/40(W) or 65(R)/35(W). Again, just more food for thought

makes total sense, I think given that, perhaps the best thing to do is rather than a separate read and write test I do a 50/50 split as you suggested and do 1 at 4k and 1 at 512k and graph them both.

Sounds like a good idea to me! Would make some interesting graphs if you plot the read, write, and cumulative speed results independent of each other on the same graph.

In the end I went with 4,64,512 and 1m will be an interesting graph I have just set off a 24-hour net and disk test on 2 servers at 11:00 am (UK/London) I will stop it just after 11:00 am tomorrow and graph the results for a new post.