GCP Cloud Performance: Time-Based Score Variations

In May 2022, one of our customers asked us to tune Elasticsearch with Esrally for cloud providers. We started with trying multiple combinations of manual runs on all cloud providers. We were collecting scaling runs with 2/4/8/16 cores. In the above data collection, we could not see the proportionate scores. Hence, we decided to experiment with running the Elasticsearch ESRally benchmark throughout the day.

As Esrally doesn’t run for a particular duration, we carried out the runs 50 times so that it will span a whole day. And here is what we saw!

Used configurations are:

Altra  - t2a-standard-16
Intel Icelake - n2d-standard-16
Milan - c2d-standard-16

Elasticsearch 8.4.1 
Esrally 2.6.0

Server	Altra.	Intel Icelake.	Milan.
Client	Altra	Altra	Altra

Variation is observed according to time of day.

AMD is the best where SD is lowest.

But Intel and Altra show large standard deviations.

NGINX-wrk benchmark also shows such behaviour on GCP.

NGINX- wrk runs are carried out 1440 times keeping each run ‘s duration 60 seconds.

Variation in p95 latency is observed through the time of day.

Both Intel and Altra show 10% standard deviation in p95 latency numbers.

Do consider Time-based Score Variations before running network applications:

1. Time of day does affect latency since neighboring VMs might be busy or idle depending on the time of day. 

2. Run-to-run variation is a function of time of day. Eventually, we were able to help the customer figure out where the performance difference was coming from. To ensure a specific output through the day the scaling of VM has been suggested.