== sanity test 398b: DIO and buffer IO race ============== 07:53:16 (1730811196) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.22703 s, 41.0 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11074: Tue Nov 5 07:53:32 2024 read: IOPS=109, BW=436KiB/s (447kB/s)(5992KiB/13734msec) clat (usec): min=54, max=190868, avg=5080.03, stdev=5507.46 lat (usec): min=55, max=190869, avg=5080.72, stdev=5507.47 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 115], 10.00th=[ 2409], 20.00th=[ 2900], | 30.00th=[ 3294], 40.00th=[ 3785], 50.00th=[ 4490], 60.00th=[ 5276], | 70.00th=[ 6063], 80.00th=[ 6980], 90.00th=[ 8356], 95.00th=[ 9765], | 99.00th=[ 13698], 99.50th=[ 15664], 99.90th=[ 22152], 99.95th=[191890], | 99.99th=[191890] bw ( KiB/s): min= 216, max= 501, per=24.53%, avg=428.78, stdev=60.94, samples=27 iops : min= 54, max= 125, avg=107.07, stdev=15.19, samples=27 write: IOPS=114, BW=458KiB/s (469kB/s)(6296KiB/13734msec) clat (usec): min=431, max=23213, avg=3839.89, stdev=3394.25 lat (usec): min=432, max=23214, avg=3840.97, stdev=3394.07 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 545], 10.00th=[ 603], 20.00th=[ 766], | 30.00th=[ 947], 40.00th=[ 2147], 50.00th=[ 3097], 60.00th=[ 3851], | 70.00th=[ 4948], 80.00th=[ 6521], 90.00th=[ 8717], 95.00th=[10945], | 99.00th=[14222], 99.50th=[14615], 99.90th=[18220], 99.95th=[23200], | 99.99th=[23200] bw ( KiB/s): min= 256, max= 600, per=24.70%, avg=451.93, stdev=67.29, samples=27 iops : min= 64, max= 150, avg=112.85, stdev=16.90, samples=27 lat (usec) : 100=2.02%, 250=0.59%, 500=0.33%, 750=9.37%, 1000=6.22% lat (msec) : 2=4.33%, 4=30.31%, 10=41.24%, 20=5.47%, 50=0.10% lat (msec) : 250=0.03% cpu : usr=0.25%, sys=23.56%, ctx=5824, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11075: Tue Nov 5 07:53:32 2024 read: IOPS=109, BW=437KiB/s (447kB/s)(5940KiB/13595msec) clat (usec): min=47, max=164836, avg=4935.00, stdev=4974.07 lat (usec): min=48, max=164838, avg=4935.67, stdev=4974.10 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 108], 10.00th=[ 2376], 20.00th=[ 2900], | 30.00th=[ 3294], 40.00th=[ 3785], 50.00th=[ 4293], 60.00th=[ 5014], | 70.00th=[ 5932], 80.00th=[ 6849], 90.00th=[ 8225], 95.00th=[ 9372], | 99.00th=[ 12387], 99.50th=[ 15795], 99.90th=[ 28181], 99.95th=[164627], | 99.99th=[164627] bw ( KiB/s): min= 152, max= 592, per=24.94%, avg=435.89, stdev=79.61, samples=27 iops : min= 38, max= 148, avg=108.85, stdev=19.88, samples=27 write: IOPS=116, BW=467KiB/s (478kB/s)(6348KiB/13595msec) clat (usec): min=408, max=50481, avg=3889.18, stdev=3615.81 lat (usec): min=408, max=50482, avg=3890.84, stdev=3615.26 clat percentiles (usec): | 1.00th=[ 502], 5.00th=[ 553], 10.00th=[ 594], 20.00th=[ 766], | 30.00th=[ 979], 40.00th=[ 2024], 50.00th=[ 3163], 60.00th=[ 3916], | 70.00th=[ 5014], 80.00th=[ 6652], 90.00th=[ 8848], 95.00th=[10552], | 99.00th=[14877], 99.50th=[16450], 99.90th=[19792], 99.95th=[50594], | 99.99th=[50594] bw ( KiB/s): min= 216, max= 584, per=25.36%, avg=464.04, stdev=71.66, samples=27 iops : min= 54, max= 146, avg=115.89, stdev=17.95, samples=27 lat (usec) : 50=0.10%, 100=2.25%, 250=0.65%, 500=0.49%, 750=9.60% lat (usec) : 1000=5.79% lat (msec) : 2=5.01%, 4=29.30%, 10=41.86%, 20=4.79%, 50=0.10% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.24%, sys=23.03%, ctx=5942, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11076: Tue Nov 5 07:53:32 2024 read: IOPS=109, BW=439KiB/s (450kB/s)(6028KiB/13722msec) clat (usec): min=49, max=27511, avg=4899.98, stdev=2881.06 lat (usec): min=49, max=27512, avg=4900.72, stdev=2881.07 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 99], 10.00th=[ 2245], 20.00th=[ 2835], | 30.00th=[ 3261], 40.00th=[ 3687], 50.00th=[ 4424], 60.00th=[ 5211], | 70.00th=[ 6063], 80.00th=[ 6915], 90.00th=[ 8356], 95.00th=[ 9503], | 99.00th=[13829], 99.50th=[17957], 99.90th=[24249], 99.95th=[27395], | 99.99th=[27395] bw ( KiB/s): min= 200, max= 536, per=24.88%, avg=434.85, stdev=72.64, samples=27 iops : min= 50, max= 134, avg=108.67, stdev=18.14, samples=27 write: IOPS=114, BW=456KiB/s (467kB/s)(6260KiB/13722msec) clat (usec): min=420, max=173228, avg=3996.72, stdev=5543.00 lat (usec): min=420, max=173230, avg=3997.99, stdev=5543.15 clat percentiles (usec): | 1.00th=[ 515], 5.00th=[ 553], 10.00th=[ 611], 20.00th=[ 775], | 30.00th=[ 947], 40.00th=[ 2040], 50.00th=[ 3130], 60.00th=[ 4047], | 70.00th=[ 5080], 80.00th=[ 6849], 90.00th=[ 8717], 95.00th=[ 10290], | 99.00th=[ 14091], 99.50th=[ 17171], 99.90th=[ 40109], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 280, max= 568, per=24.60%, avg=450.26, stdev=79.84, samples=27 iops : min= 70, max= 142, avg=112.52, stdev=19.99, samples=27 lat (usec) : 50=0.03%, 100=2.47%, 250=0.75%, 500=0.26%, 750=9.41% lat (usec) : 1000=6.58% lat (msec) : 2=4.52%, 4=28.61%, 10=42.77%, 20=4.33%, 50=0.23% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=23.10%, ctx=5822, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11077: Tue Nov 5 07:53:32 2024 read: IOPS=112, BW=450KiB/s (461kB/s)(6052KiB/13451msec) clat (usec): min=48, max=18866, avg=4932.48, stdev=2730.45 lat (usec): min=48, max=18866, avg=4933.14, stdev=2730.45 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 116], 10.00th=[ 2376], 20.00th=[ 2868], | 30.00th=[ 3294], 40.00th=[ 3752], 50.00th=[ 4490], 60.00th=[ 5211], | 70.00th=[ 5997], 80.00th=[ 6915], 90.00th=[ 8291], 95.00th=[ 9765], | 99.00th=[13698], 99.50th=[16057], 99.90th=[18482], 99.95th=[18744], | 99.99th=[18744] bw ( KiB/s): min= 232, max= 560, per=25.52%, avg=446.04, stdev=69.60, samples=26 iops : min= 58, max= 140, avg=111.46, stdev=17.44, samples=26 write: IOPS=115, BW=464KiB/s (475kB/s)(6236KiB/13451msec) clat (usec): min=478, max=160734, avg=3777.91, stdev=5258.66 lat (usec): min=478, max=160736, avg=3779.69, stdev=5258.38 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 553], 10.00th=[ 611], 20.00th=[ 734], | 30.00th=[ 898], 40.00th=[ 1713], 50.00th=[ 3032], 60.00th=[ 3720], | 70.00th=[ 4686], 80.00th=[ 5997], 90.00th=[ 8291], 95.00th=[ 10290], | 99.00th=[ 14484], 99.50th=[ 16188], 99.90th=[ 47973], 99.95th=[160433], | 99.99th=[160433] bw ( KiB/s): min= 272, max= 624, per=25.16%, avg=460.50, stdev=72.85, samples=26 iops : min= 68, max= 156, avg=115.08, stdev=18.21, samples=26 lat (usec) : 50=0.07%, 100=2.21%, 250=0.46%, 500=0.16%, 750=10.29% lat (usec) : 1000=6.18% lat (msec) : 2=4.82%, 4=29.30%, 10=41.63%, 20=4.82%, 50=0.03% lat (msec) : 250=0.03% cpu : usr=0.20%, sys=23.35%, ctx=5981, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1748KiB/s (1790kB/s), 436KiB/s-450KiB/s (447kB/s-461kB/s), io=23.4MiB (24.6MB), run=13451-13734msec WRITE: bw=1830KiB/s (1874kB/s), 456KiB/s-467KiB/s (467kB/s-478kB/s), io=24.6MiB (25.7MB), run=13451-13734msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11070: Tue Nov 5 07:53:36 2024 read: IOPS=84, BW=337KiB/s (345kB/s)(5992KiB/17774msec) clat (usec): min=1562, max=174358, avg=4840.64, stdev=5593.68 lat (usec): min=1563, max=174360, avg=4841.37, stdev=5593.72 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1909], 10.00th=[ 2040], 20.00th=[ 2245], | 30.00th=[ 2507], 40.00th=[ 2933], 50.00th=[ 3425], 60.00th=[ 4015], | 70.00th=[ 4817], 80.00th=[ 6521], 90.00th=[ 10028], 95.00th=[ 12518], | 99.00th=[ 16712], 99.50th=[ 18220], 99.90th=[ 25822], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 112, max= 760, per=24.74%, avg=333.23, stdev=156.38, samples=35 iops : min= 28, max= 190, avg=83.14, stdev=39.18, samples=35 write: IOPS=88, BW=354KiB/s (363kB/s)(6296KiB/17774msec) clat (usec): min=2239, max=30792, avg=6634.59, stdev=4033.91 lat (usec): min=2239, max=30793, avg=6635.44, stdev=4033.92 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 2966], 10.00th=[ 3195], 20.00th=[ 3523], | 30.00th=[ 3916], 40.00th=[ 4359], 50.00th=[ 5080], 60.00th=[ 5932], | 70.00th=[ 7504], 80.00th=[ 9503], 90.00th=[12387], 95.00th=[14484], | 99.00th=[20317], 99.50th=[22676], 99.90th=[27657], 99.95th=[30802], | 99.99th=[30802] bw ( KiB/s): min= 136, max= 712, per=24.80%, avg=349.91, stdev=155.70, samples=35 iops : min= 34, max= 178, avg=87.31, stdev=38.97, samples=35 lat (msec) : 2=4.00%, 4=41.80%, 10=39.94%, 20=13.57%, 50=0.65% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=10.99%, ctx=3663, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11071: Tue Nov 5 07:53:36 2024 read: IOPS=83, BW=334KiB/s (342kB/s)(5940KiB/17774msec) clat (usec): min=1580, max=174198, avg=4761.73, stdev=5635.23 lat (usec): min=1581, max=174199, avg=4762.42, stdev=5635.25 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1926], 10.00th=[ 2040], 20.00th=[ 2245], | 30.00th=[ 2573], 40.00th=[ 2933], 50.00th=[ 3425], 60.00th=[ 3982], | 70.00th=[ 4621], 80.00th=[ 6259], 90.00th=[ 9503], 95.00th=[ 12256], | 99.00th=[ 18220], 99.50th=[ 21103], 99.90th=[ 30278], 99.95th=[175113], | 99.99th=[175113] bw ( KiB/s): min= 112, max= 720, per=24.68%, avg=332.46, stdev=165.11, samples=35 iops : min= 28, max= 180, avg=83.03, stdev=41.32, samples=35 write: IOPS=89, BW=357KiB/s (366kB/s)(6348KiB/17774msec) clat (usec): min=2115, max=49237, avg=6693.65, stdev=4144.36 lat (usec): min=2116, max=49239, avg=6694.51, stdev=4144.37 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 2933], 10.00th=[ 3130], 20.00th=[ 3523], | 30.00th=[ 3818], 40.00th=[ 4424], 50.00th=[ 5145], 60.00th=[ 6128], | 70.00th=[ 7898], 80.00th=[ 9896], 90.00th=[12387], 95.00th=[14484], | 99.00th=[20317], 99.50th=[24249], 99.90th=[29754], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min= 168, max= 736, per=24.92%, avg=351.66, stdev=149.89, samples=35 iops : min= 42, max= 184, avg=87.83, stdev=37.52, samples=35 lat (msec) : 2=3.91%, 4=42.68%, 10=38.80%, 20=13.80%, 50=0.78% lat (msec) : 250=0.03% cpu : usr=0.17%, sys=10.84%, ctx=3639, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11072: Tue Nov 5 07:53:36 2024 read: IOPS=84, BW=339KiB/s (347kB/s)(6028KiB/17804msec) clat (usec): min=1485, max=30207, avg=4777.93, stdev=3768.64 lat (usec): min=1486, max=30207, avg=4778.69, stdev=3768.63 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1893], 10.00th=[ 2008], 20.00th=[ 2245], | 30.00th=[ 2540], 40.00th=[ 2933], 50.00th=[ 3458], 60.00th=[ 4015], | 70.00th=[ 4621], 80.00th=[ 6521], 90.00th=[ 9765], 95.00th=[13304], | 99.00th=[18482], 99.50th=[22152], 99.90th=[28443], 99.95th=[30278], | 99.99th=[30278] bw ( KiB/s): min= 120, max= 744, per=24.86%, avg=334.83, stdev=151.18, samples=35 iops : min= 30, max= 186, avg=83.54, stdev=37.87, samples=35 write: IOPS=87, BW=352KiB/s (360kB/s)(6260KiB/17804msec) clat (msec): min=2, max=176, avg= 6.72, stdev= 5.85 lat (msec): min=2, max=176, avg= 6.73, stdev= 5.85 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 13], 95.00th=[ 15], | 99.00th=[ 21], 99.50th=[ 23], 99.90th=[ 41], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 152, max= 768, per=24.57%, avg=346.71, stdev=160.00, samples=35 iops : min= 38, max= 192, avg=86.51, stdev=40.04, samples=35 lat (msec) : 2=4.69%, 4=41.18%, 10=40.04%, 20=13.22%, 50=0.85% lat (msec) : 250=0.03% cpu : usr=0.21%, sys=11.06%, ctx=3659, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11073: Tue Nov 5 07:53:36 2024 read: IOPS=84, BW=340KiB/s (348kB/s)(6052KiB/17814msec) clat (usec): min=1437, max=30464, avg=4819.78, stdev=3765.37 lat (usec): min=1437, max=30465, avg=4820.46, stdev=3765.40 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 1926], 10.00th=[ 2040], 20.00th=[ 2278], | 30.00th=[ 2540], 40.00th=[ 2933], 50.00th=[ 3425], 60.00th=[ 3982], | 70.00th=[ 4817], 80.00th=[ 6652], 90.00th=[10290], 95.00th=[13042], | 99.00th=[17957], 99.50th=[23200], 99.90th=[29492], 99.95th=[30540], | 99.99th=[30540] bw ( KiB/s): min= 120, max= 744, per=24.91%, avg=335.51, stdev=153.87, samples=35 iops : min= 30, max= 186, avg=83.71, stdev=38.51, samples=35 write: IOPS=87, BW=350KiB/s (358kB/s)(6236KiB/17814msec) clat (msec): min=2, max=176, avg= 6.70, stdev= 5.87 lat (msec): min=2, max=176, avg= 6.70, stdev= 5.87 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 4], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 5], 50.00th=[ 6], 60.00th=[ 7], | 70.00th=[ 8], 80.00th=[ 10], 90.00th=[ 12], 95.00th=[ 15], | 99.00th=[ 21], 99.50th=[ 23], 99.90th=[ 47], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 160, max= 728, per=24.49%, avg=345.57, stdev=153.58, samples=35 iops : min= 40, max= 182, avg=86.23, stdev=38.46, samples=35 lat (msec) : 2=4.00%, 4=42.38%, 10=40.01%, 20=12.70%, 50=0.88% lat (msec) : 250=0.03% cpu : usr=0.22%, sys=10.82%, ctx=3661, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1348KiB/s (1380kB/s), 334KiB/s-340KiB/s (342kB/s-348kB/s), io=23.4MiB (24.6MB), run=17774-17814msec WRITE: bw=1411KiB/s (1445kB/s), 350KiB/s-357KiB/s (358kB/s-366kB/s), io=24.6MiB (25.7MB), run=17774-17814msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11110: Tue Nov 5 07:53:43 2024 read: IOPS=60, BW=960KiB/s (983kB/s)(5920KiB/6165msec) clat (usec): min=63, max=28959, avg=11071.31, stdev=3699.49 lat (usec): min=63, max=28959, avg=11071.94, stdev=3699.49 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 7177], 10.00th=[ 7963], 20.00th=[ 8717], | 30.00th=[ 9634], 40.00th=[10290], 50.00th=[10945], 60.00th=[11469], | 70.00th=[11863], 80.00th=[12911], 90.00th=[15270], 95.00th=[16581], | 99.00th=[24773], 99.50th=[28705], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 736, max= 1184, per=24.50%, avg=951.75, stdev=158.44, samples=12 iops : min= 46, max= 74, avg=59.42, stdev= 9.86, samples=12 write: IOPS=64, BW=1033KiB/s (1058kB/s)(6368KiB/6165msec) clat (usec): min=758, max=63632, avg=5115.38, stdev=5155.34 lat (usec): min=759, max=63634, avg=5116.64, stdev=5155.59 clat percentiles (usec): | 1.00th=[ 758], 5.00th=[ 832], 10.00th=[ 898], 20.00th=[ 1418], | 30.00th=[ 2311], 40.00th=[ 2802], 50.00th=[ 3589], 60.00th=[ 4621], | 70.00th=[ 5800], 80.00th=[ 8029], 90.00th=[11469], 95.00th=[14222], | 99.00th=[21365], 99.50th=[23200], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 800, max= 1280, per=25.31%, avg=1034.42, stdev=166.15, samples=12 iops : min= 50, max= 80, avg=64.58, stdev=10.38, samples=12 lat (usec) : 100=0.78%, 250=0.65%, 500=0.13%, 1000=6.90% lat (msec) : 2=5.34%, 4=16.15%, 10=31.51%, 20=36.85%, 50=1.56% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=19.21%, ctx=2793, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11111: Tue Nov 5 07:53:43 2024 read: IOPS=58, BW=940KiB/s (962kB/s)(5744KiB/6112msec) clat (usec): min=60, max=73153, avg=11589.97, stdev=4700.85 lat (usec): min=60, max=73153, avg=11590.60, stdev=4700.86 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 7439], 10.00th=[ 8356], 20.00th=[ 9372], | 30.00th=[ 9765], 40.00th=[10552], 50.00th=[10945], 60.00th=[11731], | 70.00th=[12649], 80.00th=[13566], 90.00th=[15401], 95.00th=[17695], | 99.00th=[22414], 99.50th=[30016], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 704, max= 1120, per=24.02%, avg=933.33, stdev=117.25, samples=12 iops : min= 44, max= 70, avg=58.33, stdev= 7.33, samples=12 write: IOPS=66, BW=1071KiB/s (1096kB/s)(6544KiB/6112msec) clat (usec): min=688, max=47939, avg=4738.96, stdev=4538.59 lat (usec): min=689, max=47940, avg=4740.64, stdev=4538.07 clat percentiles (usec): | 1.00th=[ 775], 5.00th=[ 832], 10.00th=[ 930], 20.00th=[ 1713], | 30.00th=[ 2507], 40.00th=[ 2933], 50.00th=[ 3425], 60.00th=[ 4047], | 70.00th=[ 5080], 80.00th=[ 6456], 90.00th=[ 9896], 95.00th=[14091], | 99.00th=[19792], 99.50th=[22414], 99.90th=[47973], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min= 832, max= 1504, per=25.84%, avg=1056.00, stdev=236.34, samples=12 iops : min= 52, max= 94, avg=66.00, stdev=14.77, samples=12 lat (usec) : 100=0.78%, 250=0.13%, 750=0.39%, 1000=5.86% lat (msec) : 2=5.34%, 4=20.31%, 10=30.21%, 20=35.55%, 50=1.30% lat (msec) : 100=0.13% cpu : usr=0.11%, sys=19.98%, ctx=2786, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11112: Tue Nov 5 07:53:43 2024 read: IOPS=63, BW=1016KiB/s (1040kB/s)(6256KiB/6159msec) clat (usec): min=60, max=64392, avg=11199.48, stdev=4432.02 lat (usec): min=61, max=64393, avg=11200.13, stdev=4432.00 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 7373], 10.00th=[ 8094], 20.00th=[ 8979], | 30.00th=[ 9634], 40.00th=[10290], 50.00th=[10814], 60.00th=[11469], | 70.00th=[11994], 80.00th=[13042], 90.00th=[14746], 95.00th=[16909], | 99.00th=[23200], 99.50th=[34341], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min= 736, max= 1344, per=25.95%, avg=1008.00, stdev=171.24, samples=12 iops : min= 46, max= 84, avg=63.00, stdev=10.70, samples=12 write: IOPS=61, BW=979KiB/s (1003kB/s)(6032KiB/6159msec) clat (usec): min=684, max=51774, avg=4661.97, stdev=5163.98 lat (usec): min=685, max=51775, avg=4663.55, stdev=5163.80 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 807], 10.00th=[ 922], 20.00th=[ 1205], | 30.00th=[ 2278], 40.00th=[ 2737], 50.00th=[ 3163], 60.00th=[ 3654], | 70.00th=[ 5145], 80.00th=[ 6456], 90.00th=[11076], 95.00th=[13960], | 99.00th=[22414], 99.50th=[43254], 99.90th=[51643], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min= 544, max= 1280, per=23.16%, avg=946.67, stdev=208.04, samples=12 iops : min= 34, max= 80, avg=59.17, stdev=13.00, samples=12 lat (usec) : 100=1.04%, 250=0.39%, 750=0.65%, 1000=6.51% lat (msec) : 2=6.12%, 4=18.23%, 10=29.43%, 20=36.59%, 50=0.78% lat (msec) : 100=0.26% cpu : usr=0.06%, sys=18.82%, ctx=2732, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11113: Tue Nov 5 07:53:43 2024 read: IOPS=62, BW=994KiB/s (1018kB/s)(6032KiB/6068msec) clat (usec): min=60, max=31341, avg=11250.83, stdev=3293.11 lat (usec): min=60, max=31342, avg=11251.44, stdev=3293.12 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 7177], 10.00th=[ 8160], 20.00th=[ 9110], | 30.00th=[ 9765], 40.00th=[10421], 50.00th=[11207], 60.00th=[11994], | 70.00th=[12649], 80.00th=[13435], 90.00th=[14877], 95.00th=[16057], | 99.00th=[20055], 99.50th=[21627], 99.90th=[31327], 99.95th=[31327], | 99.99th=[31327] bw ( KiB/s): min= 800, max= 1248, per=25.45%, avg=988.92, stdev=136.71, samples=12 iops : min= 50, max= 78, avg=61.67, stdev= 8.57, samples=12 write: IOPS=64, BW=1031KiB/s (1056kB/s)(6256KiB/6068msec) clat (usec): min=761, max=55781, avg=4603.30, stdev=5249.70 lat (usec): min=762, max=55782, avg=4604.39, stdev=5249.79 clat percentiles (usec): | 1.00th=[ 783], 5.00th=[ 824], 10.00th=[ 873], 20.00th=[ 1106], | 30.00th=[ 2114], 40.00th=[ 2573], 50.00th=[ 2999], 60.00th=[ 3589], | 70.00th=[ 4686], 80.00th=[ 6259], 90.00th=[10421], 95.00th=[13698], | 99.00th=[26870], 99.50th=[32900], 99.90th=[55837], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 832, max= 1216, per=25.24%, avg=1031.67, stdev=126.19, samples=12 iops : min= 52, max= 76, avg=64.33, stdev= 8.03, samples=12 lat (usec) : 100=0.91%, 250=0.26%, 1000=8.46% lat (msec) : 2=5.86%, 4=18.36%, 10=27.73%, 20=37.11%, 50=1.17% lat (msec) : 100=0.13% cpu : usr=0.03%, sys=19.28%, ctx=2720, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3885KiB/s (3978kB/s), 940KiB/s-1016KiB/s (962kB/s-1040kB/s), io=23.4MiB (24.5MB), run=6068-6165msec WRITE: bw=4088KiB/s (4186kB/s), 979KiB/s-1071KiB/s (1003kB/s-1096kB/s), io=24.6MiB (25.8MB), run=6068-6165msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11114: Tue Nov 5 07:53:43 2024 read: IOPS=60, BW=962KiB/s (985kB/s)(5920KiB/6156msec) clat (usec): min=1814, max=70740, avg=6266.81, stdev=5716.60 lat (usec): min=1814, max=70742, avg=6267.46, stdev=5716.64 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2114], 10.00th=[ 2278], 20.00th=[ 2573], | 30.00th=[ 2966], 40.00th=[ 3326], 50.00th=[ 4047], 60.00th=[ 5407], | 70.00th=[ 7504], 80.00th=[ 9765], 90.00th=[11731], 95.00th=[14877], | 99.00th=[24249], 99.50th=[29754], 99.90th=[70779], 99.95th=[70779], | 99.99th=[70779] bw ( KiB/s): min= 736, max= 1153, per=24.73%, avg=957.42, stdev=155.36, samples=12 iops : min= 46, max= 72, avg=59.83, stdev= 9.70, samples=12 write: IOPS=64, BW=1034KiB/s (1059kB/s)(6368KiB/6156msec) clat (usec): min=2701, max=42577, avg=9526.94, stdev=5114.36 lat (usec): min=2702, max=42578, avg=9528.30, stdev=5114.33 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3687], 10.00th=[ 3949], 20.00th=[ 4948], | 30.00th=[ 6587], 40.00th=[ 7308], 50.00th=[ 8455], 60.00th=[ 9634], | 70.00th=[10945], 80.00th=[12911], 90.00th=[16909], 95.00th=[19530], | 99.00th=[23987], 99.50th=[28967], 99.90th=[42730], 99.95th=[42730], | 99.99th=[42730] bw ( KiB/s): min= 800, max= 1248, per=25.29%, avg=1030.00, stdev=165.76, samples=12 iops : min= 50, max= 78, avg=64.33, stdev=10.33, samples=12 lat (msec) : 2=0.91%, 4=27.99%, 10=42.32%, 20=25.65%, 50=2.99% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=10.25%, ctx=1198, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11115: Tue Nov 5 07:53:43 2024 read: IOPS=58, BW=936KiB/s (958kB/s)(5744KiB/6139msec) clat (usec): min=1911, max=82101, avg=6773.90, stdev=6629.41 lat (usec): min=1911, max=82102, avg=6774.67, stdev=6629.38 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 2245], 10.00th=[ 2376], 20.00th=[ 2638], | 30.00th=[ 3064], 40.00th=[ 3621], 50.00th=[ 4621], 60.00th=[ 5997], | 70.00th=[ 7767], 80.00th=[ 9765], 90.00th=[13173], 95.00th=[16909], | 99.00th=[28181], 99.50th=[39060], 99.90th=[82314], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 704, max= 1152, per=23.99%, avg=928.67, stdev=129.11, samples=12 iops : min= 44, max= 72, avg=58.00, stdev= 8.05, samples=12 write: IOPS=66, BW=1066KiB/s (1092kB/s)(6544KiB/6139msec) clat (usec): min=2653, max=37973, avg=9016.66, stdev=4994.83 lat (usec): min=2653, max=37974, avg=9017.82, stdev=4994.83 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3294], 10.00th=[ 3851], 20.00th=[ 4752], | 30.00th=[ 5735], 40.00th=[ 6849], 50.00th=[ 7898], 60.00th=[ 9241], | 70.00th=[10421], 80.00th=[12387], 90.00th=[16057], 95.00th=[18744], | 99.00th=[24773], 99.50th=[25560], 99.90th=[38011], 99.95th=[38011], | 99.99th=[38011] bw ( KiB/s): min= 768, max= 1504, per=25.98%, avg=1058.25, stdev=223.47, samples=12 iops : min= 48, max= 94, avg=66.08, stdev=13.90, samples=12 lat (msec) : 2=0.39%, 4=27.21%, 10=45.44%, 20=23.44%, 50=3.39% lat (msec) : 100=0.13% cpu : usr=0.08%, sys=9.91%, ctx=1149, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11116: Tue Nov 5 07:53:43 2024 read: IOPS=63, BW=1011KiB/s (1035kB/s)(6256KiB/6187msec) clat (usec): min=1819, max=28938, avg=6528.15, stdev=4308.83 lat (usec): min=1820, max=28938, avg=6528.83, stdev=4308.85 clat percentiles (usec): | 1.00th=[ 2024], 5.00th=[ 2245], 10.00th=[ 2409], 20.00th=[ 2835], | 30.00th=[ 3294], 40.00th=[ 3949], 50.00th=[ 5080], 60.00th=[ 6652], | 70.00th=[ 8455], 80.00th=[10028], 90.00th=[11994], 95.00th=[14746], | 99.00th=[20055], 99.50th=[25297], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 704, max= 1312, per=25.88%, avg=1001.92, stdev=174.47, samples=12 iops : min= 44, max= 82, avg=62.58, stdev=10.86, samples=12 write: IOPS=60, BW=975KiB/s (998kB/s)(6032KiB/6187msec) clat (usec): min=2383, max=54534, avg=9507.12, stdev=5940.86 lat (usec): min=2384, max=54537, avg=9508.34, stdev=5940.91 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3392], 10.00th=[ 3720], 20.00th=[ 4752], | 30.00th=[ 6259], 40.00th=[ 7177], 50.00th=[ 7963], 60.00th=[ 9372], | 70.00th=[10683], 80.00th=[12780], 90.00th=[17171], 95.00th=[20317], | 99.00th=[30278], 99.50th=[36439], 99.90th=[54789], 99.95th=[54789], | 99.99th=[54789] bw ( KiB/s): min= 672, max= 1344, per=23.51%, avg=957.50, stdev=185.76, samples=12 iops : min= 42, max= 84, avg=59.83, stdev=11.61, samples=12 lat (msec) : 2=0.26%, 4=26.69%, 10=45.44%, 20=24.22%, 50=3.26% lat (msec) : 100=0.13% cpu : usr=0.06%, sys=10.05%, ctx=1170, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11117: Tue Nov 5 07:53:43 2024 read: IOPS=67, BW=1079KiB/s (1105kB/s)(6032KiB/5590msec) clat (usec): min=1880, max=29635, avg=5609.11, stdev=4187.17 lat (usec): min=1880, max=29635, avg=5609.80, stdev=4187.14 clat percentiles (usec): | 1.00th=[ 2114], 5.00th=[ 2245], 10.00th=[ 2376], 20.00th=[ 2638], | 30.00th=[ 2933], 40.00th=[ 3261], 50.00th=[ 3752], 60.00th=[ 4490], | 70.00th=[ 6194], 80.00th=[ 8848], 90.00th=[11207], 95.00th=[13698], | 99.00th=[21890], 99.50th=[22414], 99.90th=[29754], 99.95th=[29754], | 99.99th=[29754] bw ( KiB/s): min= 864, max= 1280, per=27.79%, avg=1075.91, stdev=136.12, samples=11 iops : min= 54, max= 80, avg=67.09, stdev= 8.58, samples=11 write: IOPS=69, BW=1119KiB/s (1146kB/s)(6256KiB/5590msec) clat (usec): min=2804, max=52657, avg=8731.75, stdev=6097.66 lat (usec): min=2806, max=52658, avg=8733.01, stdev=6097.63 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 3425], 10.00th=[ 3752], 20.00th=[ 4228], | 30.00th=[ 4686], 40.00th=[ 5538], 50.00th=[ 6587], 60.00th=[ 7963], | 70.00th=[ 9896], 80.00th=[13042], 90.00th=[16450], 95.00th=[19792], | 99.00th=[33817], 99.50th=[40633], 99.90th=[52691], 99.95th=[52691], | 99.99th=[52691] bw ( KiB/s): min= 928, max= 1312, per=27.56%, avg=1122.36, stdev=141.75, samples=11 iops : min= 58, max= 82, avg=70.00, stdev= 8.80, samples=11 lat (msec) : 2=0.26%, 4=33.98%, 10=43.62%, 20=19.27%, 50=2.73% lat (msec) : 100=0.13% cpu : usr=0.23%, sys=9.27%, ctx=978, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3871KiB/s (3964kB/s), 936KiB/s-1079KiB/s (958kB/s-1105kB/s), io=23.4MiB (24.5MB), run=5590-6187msec WRITE: bw=4073KiB/s (4171kB/s), 975KiB/s-1119KiB/s (998kB/s-1146kB/s), io=24.6MiB (25.8MB), run=5590-6187msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11128: Tue Nov 5 07:53:47 2024 read: IOPS=2, BW=2337KiB/s (2393kB/s)(7168KiB/3067msec) clat (usec): min=329, max=860066, avg=382785.93, stdev=348419.36 lat (usec): min=330, max=860067, avg=382787.20, stdev=348419.37 clat percentiles (usec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 66847], | 30.00th=[183501], 40.00th=[183501], 50.00th=[325059], 60.00th=[404751], | 70.00th=[404751], 80.00th=[843056], 90.00th=[859833], 95.00th=[859833], | 99.00th=[859833], 99.50th=[859833], 99.90th=[859833], 99.95th=[859833], | 99.99th=[859833] bw ( KiB/s): min= 1875, max= 5626, per=42.14%, avg=2813.50, stdev=1875.00, samples=4 iops : min= 1, max= 5, avg= 2.00, stdev= 2.00, samples=4 write: IOPS=1, BW=1669KiB/s (1709kB/s)(5120KiB/3067msec) clat (msec): min=25, max=132, avg=70.68, stdev=52.65 lat (msec): min=25, max=132, avg=70.71, stdev=52.64 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 26], | 30.00th=[ 26], 40.00th=[ 26], 50.00th=[ 48], 60.00th=[ 48], | 70.00th=[ 123], 80.00th=[ 123], 90.00th=[ 133], 95.00th=[ 133], | 99.00th=[ 133], 99.50th=[ 133], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 1875, max= 5626, per=33.44%, avg=3126.33, stdev=2164.78, samples=3 iops : min= 1, max= 5, avg= 2.33, stdev= 2.31, samples=3 lat (usec) : 500=8.33% lat (msec) : 50=25.00%, 100=8.33%, 250=25.00%, 500=16.67%, 1000=16.67% cpu : usr=0.03%, sys=28.57%, ctx=957, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11130: Tue Nov 5 07:53:47 2024 read: IOPS=1, BW=1153KiB/s (1181kB/s)(2048KiB/1776msec) clat (msec): min=114, max=858, avg=486.54, stdev=525.49 lat (msec): min=114, max=858, avg=486.55, stdev=525.49 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 115], 10.00th=[ 115], 20.00th=[ 115], | 30.00th=[ 115], 40.00th=[ 115], 50.00th=[ 115], 60.00th=[ 860], | 70.00th=[ 860], 80.00th=[ 860], 90.00th=[ 860], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 1855, max= 2048, per=29.23%, avg=1951.50, stdev=136.47, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 write: IOPS=5, BW=5766KiB/s (5904kB/s)(10.0MiB/1776msec) clat (msec): min=26, max=187, avg=76.25, stdev=59.32 lat (msec): min=26, max=187, avg=76.31, stdev=59.32 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 27], 10.00th=[ 27], 20.00th=[ 29], | 30.00th=[ 32], 40.00th=[ 34], 50.00th=[ 43], 60.00th=[ 54], | 70.00th=[ 113], 80.00th=[ 113], 90.00th=[ 165], 95.00th=[ 188], | 99.00th=[ 188], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 3710, max= 6144, per=52.71%, avg=4927.00, stdev=1721.10, samples=2 iops : min= 3, max= 6, avg= 4.50, stdev= 2.12, samples=2 lat (msec) : 50=41.67%, 100=16.67%, 250=33.33%, 1000=8.33% cpu : usr=0.00%, sys=23.21%, ctx=355, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11131: Tue Nov 5 07:53:47 2024 read: IOPS=4, BW=4261KiB/s (4363kB/s)(6144KiB/1442msec) clat (usec): min=379, max=847359, avg=182250.44, stdev=328999.00 lat (usec): min=380, max=847360, avg=182251.31, stdev=328999.21 clat percentiles (usec): | 1.00th=[ 379], 5.00th=[ 379], 10.00th=[ 379], 20.00th=[ 424], | 30.00th=[ 424], 40.00th=[ 42206], 50.00th=[ 42206], 60.00th=[ 95945], | 70.00th=[107480], 80.00th=[107480], 90.00th=[851444], 95.00th=[851444], | 99.00th=[851444], 99.50th=[851444], 99.90th=[851444], 99.95th=[851444], | 99.99th=[851444] bw ( KiB/s): min= 3744, max= 4087, per=58.64%, avg=3915.50, stdev=242.54, samples=2 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=2 write: IOPS=4, BW=4261KiB/s (4363kB/s)(6144KiB/1442msec) clat (usec): min=29139, max=86161, avg=52093.60, stdev=20812.82 lat (usec): min=29156, max=86229, avg=52156.94, stdev=20824.78 clat percentiles (usec): | 1.00th=[29230], 5.00th=[29230], 10.00th=[29230], 20.00th=[34341], | 30.00th=[34341], 40.00th=[46924], 50.00th=[46924], 60.00th=[52167], | 70.00th=[63701], 80.00th=[63701], 90.00th=[86508], 95.00th=[86508], | 99.00th=[86508], 99.50th=[86508], 99.90th=[86508], 99.95th=[86508], | 99.99th=[86508] bw ( KiB/s): min= 1872, max= 8175, per=53.74%, avg=5023.50, stdev=4456.89, samples=2 iops : min= 1, max= 7, avg= 4.00, stdev= 4.24, samples=2 lat (usec) : 500=16.67% lat (msec) : 50=33.33%, 100=33.33%, 250=8.33%, 1000=8.33% cpu : usr=0.00%, sys=25.82%, ctx=392, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11134: Tue Nov 5 07:53:47 2024 read: IOPS=9, BW=9961KiB/s (10.2MB/s)(5120KiB/514msec) clat (usec): min=385, max=73673, avg=24679.30, stdev=33543.22 lat (usec): min=386, max=73674, avg=24680.21, stdev=33542.98 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 388], 10.00th=[ 388], 20.00th=[ 388], | 30.00th=[ 457], 40.00th=[ 457], 50.00th=[ 2966], 60.00th=[ 2966], | 70.00th=[45876], 80.00th=[45876], 90.00th=[73925], 95.00th=[73925], | 99.00th=[73925], 99.50th=[73925], 99.90th=[73925], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min= 8175, max= 8175, per=100.00%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.6MiB/s (14.3MB/s)(7168KiB/514msec) clat (usec): min=33526, max=82969, avg=53918.63, stdev=18792.95 lat (usec): min=33608, max=82986, avg=54025.09, stdev=18766.29 clat percentiles (usec): | 1.00th=[33424], 5.00th=[33424], 10.00th=[33424], 20.00th=[37487], | 30.00th=[44303], 40.00th=[44303], 50.00th=[50594], 60.00th=[53216], | 70.00th=[53216], 80.00th=[76022], 90.00th=[83362], 95.00th=[83362], | 99.00th=[83362], 99.50th=[83362], 99.90th=[83362], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min=12263, max=12263, per=100.00%, avg=12263.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (usec) : 500=16.67% lat (msec) : 4=8.33%, 50=33.33%, 100=41.67% cpu : usr=0.00%, sys=45.61%, ctx=94, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6678KiB/s (6838kB/s), 1153KiB/s-9961KiB/s (1181kB/s-10.2MB/s), io=20.0MiB (20.0MB), run=514-3067msec WRITE: bw=9349KiB/s (9573kB/s), 1669KiB/s-13.6MiB/s (1709kB/s-14.3MB/s), io=28.0MiB (29.4MB), run=514-3067msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11129: Tue Nov 5 07:53:47 2024 read: IOPS=2, BW=2273KiB/s (2327kB/s)(7168KiB/3154msec) clat (msec): min=15, max=834, avg=231.95, stdev=300.79 lat (msec): min=15, max=834, avg=231.95, stdev=300.79 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 21], | 30.00th=[ 47], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 305], | 70.00th=[ 305], 80.00th=[ 355], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 1896, max= 4096, per=45.56%, avg=2958.00, stdev=1146.96, samples=4 iops : min= 1, max= 4, avg= 2.50, stdev= 1.29, samples=4 write: IOPS=1, BW=1623KiB/s (1662kB/s)(5120KiB/3154msec) clat (msec): min=48, max=805, avg=292.70, stdev=301.70 lat (msec): min=48, max=805, avg=292.76, stdev=301.70 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 50], 10.00th=[ 50], 20.00th=[ 50], | 30.00th=[ 144], 40.00th=[ 144], 50.00th=[ 155], 60.00th=[ 155], | 70.00th=[ 313], 80.00th=[ 313], 90.00th=[ 810], 95.00th=[ 810], | 99.00th=[ 810], 99.50th=[ 810], 99.90th=[ 810], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 1896, max= 3792, per=28.37%, avg=2578.67, stdev=1053.52, samples=3 iops : min= 1, max= 3, avg= 2.00, stdev= 1.00, samples=3 lat (msec) : 20=8.33%, 50=33.33%, 250=16.67%, 500=25.00%, 1000=16.67% cpu : usr=0.06%, sys=4.69%, ctx=63, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11132: Tue Nov 5 07:53:47 2024 read: IOPS=1, BW=1104KiB/s (1131kB/s)(2048KiB/1855msec) clat (msec): min=132, max=858, avg=495.40, stdev=513.10 lat (msec): min=132, max=858, avg=495.40, stdev=513.10 clat percentiles (msec): | 1.00th=[ 133], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 133], | 30.00th=[ 133], 40.00th=[ 133], 50.00th=[ 133], 60.00th=[ 860], | 70.00th=[ 860], 80.00th=[ 860], 90.00th=[ 860], 95.00th=[ 860], | 99.00th=[ 860], 99.50th=[ 860], 99.90th=[ 860], 99.95th=[ 860], | 99.99th=[ 860] bw ( KiB/s): min= 2043, max= 2043, per=31.46%, avg=2043.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=5, BW=5520KiB/s (5653kB/s)(10.0MiB/1855msec) clat (msec): min=39, max=179, avg=83.08, stdev=46.60 lat (msec): min=40, max=179, avg=83.13, stdev=46.60 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 48], 40.00th=[ 51], 50.00th=[ 57], 60.00th=[ 80], | 70.00th=[ 117], 80.00th=[ 117], 90.00th=[ 134], 95.00th=[ 180], | 99.00th=[ 180], 99.50th=[ 180], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 4096, max= 6131, per=56.25%, avg=5113.50, stdev=1438.96, samples=2 iops : min= 4, max= 5, avg= 4.50, stdev= 0.71, samples=2 lat (msec) : 50=25.00%, 100=33.33%, 250=33.33%, 1000=8.33% cpu : usr=0.00%, sys=9.17%, ctx=242, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11133: Tue Nov 5 07:53:47 2024 read: IOPS=3, BW=3717KiB/s (3806kB/s)(6144KiB/1653msec) clat (msec): min=12, max=818, avg=191.53, stdev=314.65 lat (msec): min=12, max=818, avg=191.53, stdev=314.65 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 111], | 70.00th=[ 180], 80.00th=[ 180], 90.00th=[ 818], 95.00th=[ 818], | 99.00th=[ 818], 99.50th=[ 818], 99.90th=[ 818], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 4096, max= 4096, per=63.08%, avg=4096.00, stdev= 0.00, samples=2 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=2 write: IOPS=3, BW=3717KiB/s (3806kB/s)(6144KiB/1653msec) clat (msec): min=41, max=122, avg=73.31, stdev=30.30 lat (msec): min=41, max=122, avg=73.37, stdev=30.29 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 45], | 30.00th=[ 45], 40.00th=[ 64], 50.00th=[ 64], 60.00th=[ 84], | 70.00th=[ 85], 80.00th=[ 85], 90.00th=[ 124], 95.00th=[ 124], | 99.00th=[ 124], 99.50th=[ 124], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 4096, max= 6144, per=56.33%, avg=5120.00, stdev=1448.15, samples=2 iops : min= 4, max= 6, avg= 5.00, stdev= 1.41, samples=2 lat (msec) : 20=25.00%, 50=16.67%, 100=25.00%, 250=25.00%, 1000=8.33% cpu : usr=0.06%, sys=8.66%, ctx=97, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11135: Tue Nov 5 07:53:47 2024 read: IOPS=3, BW=3305KiB/s (3385kB/s)(5120KiB/1549msec) clat (usec): min=11620, max=22797, avg=16971.80, stdev=4424.84 lat (usec): min=11621, max=22798, avg=16973.05, stdev=4424.75 clat percentiles (usec): | 1.00th=[11600], 5.00th=[11600], 10.00th=[11600], 20.00th=[11600], | 30.00th=[14091], 40.00th=[14091], 50.00th=[16712], 60.00th=[16712], | 70.00th=[19792], 80.00th=[19792], 90.00th=[22676], 95.00th=[22676], | 99.00th=[22676], 99.50th=[22676], 99.90th=[22676], 99.95th=[22676], | 99.99th=[22676] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 write: IOPS=4, BW=4628KiB/s (4739kB/s)(7168KiB/1549msec) clat (msec): min=33, max=975, avg=199.93, stdev=343.11 lat (msec): min=33, max=975, avg=200.00, stdev=343.11 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 51], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 82], 60.00th=[ 89], | 70.00th=[ 89], 80.00th=[ 114], 90.00th=[ 978], 95.00th=[ 978], | 99.00th=[ 978], 99.50th=[ 978], 99.90th=[ 978], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 2043, max=10240, per=67.56%, avg=6141.50, stdev=5796.15, samples=2 iops : min= 1, max= 10, avg= 5.50, stdev= 6.36, samples=2 lat (msec) : 20=33.33%, 50=16.67%, 100=33.33%, 250=8.33%, 1000=8.33% cpu : usr=0.13%, sys=8.79%, ctx=183, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=6493KiB/s (6649kB/s), 1104KiB/s-3717KiB/s (1131kB/s-3806kB/s), io=20.0MiB (20.0MB), run=1549-3154msec WRITE: bw=9091KiB/s (9309kB/s), 1623KiB/s-5520KiB/s (1662kB/s-5653kB/s), io=28.0MiB (29.4MB), run=1549-3154msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=11143: Tue Nov 5 07:53:53 2024 read: IOPS=0, BW=2328KiB/s (2384kB/s)(12.0MiB/5278msec) clat (msec): min=191, max=3614, avg=1751.87, stdev=1731.65 lat (msec): min=191, max=3614, avg=1751.87, stdev=1731.65 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 192], 10.00th=[ 192], 20.00th=[ 192], | 30.00th=[ 192], 40.00th=[ 1452], 50.00th=[ 1452], 60.00th=[ 1452], | 70.00th=[ 3608], 80.00th=[ 3608], 90.00th=[ 3608], 95.00th=[ 3608], | 99.00th=[ 3608], 99.50th=[ 3608], 99.90th=[ 3608], 99.95th=[ 3608], | 99.99th=[ 3608] bw ( KiB/s): min= 7816, max= 8192, per=100.00%, avg=8004.00, stdev=265.87, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 lat (msec) : 250=33.33% cpu : usr=0.02%, sys=1.65%, ctx=40, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11144: Tue Nov 5 07:53:53 2024 read: IOPS=0, BW=1087KiB/s (1113kB/s)(4096KiB/3767msec) clat (nsec): min=63457k, max=63457k, avg=63456614.00, stdev= 0.00 lat (nsec): min=63459k, max=63459k, avg=63458970.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[63701], 5.00th=[63701], 10.00th=[63701], 20.00th=[63701], | 30.00th=[63701], 40.00th=[63701], 50.00th=[63701], 60.00th=[63701], | 70.00th=[63701], 80.00th=[63701], 90.00th=[63701], 95.00th=[63701], | 99.00th=[63701], 99.50th=[63701], 99.90th=[63701], 99.95th=[63701], | 99.99th=[63701] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=2175KiB/s (2227kB/s)(8192KiB/3767msec) clat (msec): min=130, max=3566, avg=1848.41, stdev=2429.76 lat (msec): min=130, max=3566, avg=1848.74, stdev=2429.80 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 131], 10.00th=[ 131], 20.00th=[ 131], | 30.00th=[ 131], 40.00th=[ 131], 50.00th=[ 131], 60.00th=[ 3574], | 70.00th=[ 3574], 80.00th=[ 3574], 90.00th=[ 3574], 95.00th=[ 3574], | 99.00th=[ 3574], 99.50th=[ 3574], 99.90th=[ 3574], 99.95th=[ 3574], | 99.99th=[ 3574] bw ( KiB/s): min= 8192, max= 8192, per=94.18%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=2.97%, ctx=47, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11145: Tue Nov 5 07:53:53 2024 write: IOPS=2, BW=10.5MiB/s (11.0MB/s)(12.0MiB/1140msec) clat (msec): min=136, max=836, avg=375.27, stdev=399.36 lat (msec): min=137, max=836, avg=375.64, stdev=399.32 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 138], 10.00th=[ 138], 20.00th=[ 138], | 30.00th=[ 138], 40.00th=[ 153], 50.00th=[ 153], 60.00th=[ 153], | 70.00th=[ 835], 80.00th=[ 835], 90.00th=[ 835], 95.00th=[ 835], | 99.00th=[ 835], 99.50th=[ 835], 99.90th=[ 835], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 8192, max= 8192, per=94.18%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=66.67%, 1000=33.33% cpu : usr=0.26%, sys=9.39%, ctx=62, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11146: Tue Nov 5 07:53:53 2024 write: IOPS=2, BW=11.6MiB/s (12.2MB/s)(12.0MiB/1031msec) clat (msec): min=40, max=918, avg=338.84, stdev=502.07 lat (msec): min=40, max=918, avg=339.14, stdev=502.05 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 41], | 30.00th=[ 41], 40.00th=[ 58], 50.00th=[ 58], 60.00th=[ 58], | 70.00th=[ 919], 80.00th=[ 919], 90.00th=[ 919], 95.00th=[ 919], | 99.00th=[ 919], 99.50th=[ 919], 99.90th=[ 919], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min=16384, max=16384, per=100.00%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 50=33.33%, 100=33.33%, 1000=33.33% cpu : usr=0.00%, sys=6.31%, ctx=63, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3104KiB/s (3179kB/s), 1087KiB/s-2328KiB/s (1113kB/s-2384kB/s), io=16.0MiB (16.8MB), run=3767-5278msec WRITE: bw=8699KiB/s (8907kB/s), 2175KiB/s-11.6MiB/s (2227kB/s-12.2MB/s), io=32.0MiB (33.6MB), run=1031-3767msec rand-rw: (groupid=0, jobs=1): err= 0: pid=11147: Tue Nov 5 07:53:54 2024 read: IOPS=0, BW=2353KiB/s (2410kB/s)(12.0MiB/5222msec) clat (msec): min=124, max=3482, avg=1723.11, stdev=1684.59 lat (msec): min=124, max=3482, avg=1723.11, stdev=1684.59 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 125], 10.00th=[ 125], 20.00th=[ 125], | 30.00th=[ 125], 40.00th=[ 1569], 50.00th=[ 1569], 60.00th=[ 1569], | 70.00th=[ 3473], 80.00th=[ 3473], 90.00th=[ 3473], 95.00th=[ 3473], | 99.00th=[ 3473], 99.50th=[ 3473], 99.90th=[ 3473], 99.95th=[ 3473], | 99.99th=[ 3473] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=2 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=2 lat (msec) : 250=33.33% cpu : usr=0.02%, sys=28.54%, ctx=1623, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11148: Tue Nov 5 07:53:54 2024 read: IOPS=0, BW=998KiB/s (1022kB/s)(4096KiB/4103msec) clat (nsec): min=161918k, max=161918k, avg=161918227.00, stdev= 0.00 lat (nsec): min=161919k, max=161919k, avg=161919400.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 163], 10.00th=[ 163], 20.00th=[ 163], | 30.00th=[ 163], 40.00th=[ 163], 50.00th=[ 163], 60.00th=[ 163], | 70.00th=[ 163], 80.00th=[ 163], 90.00th=[ 163], 95.00th=[ 163], | 99.00th=[ 163], 99.50th=[ 163], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=0, BW=1997KiB/s (2045kB/s)(8192KiB/4103msec) clat (msec): min=190, max=3726, avg=1958.22, stdev=2500.48 lat (msec): min=193, max=3726, avg=1959.90, stdev=2498.37 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 190], 10.00th=[ 190], 20.00th=[ 190], | 30.00th=[ 190], 40.00th=[ 190], 50.00th=[ 190], 60.00th=[ 3742], | 70.00th=[ 3742], 80.00th=[ 3742], 90.00th=[ 3742], 95.00th=[ 3742], | 99.00th=[ 3742], 99.50th=[ 3742], 99.90th=[ 3742], 99.95th=[ 3742], | 99.99th=[ 3742] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67% cpu : usr=0.00%, sys=6.36%, ctx=209, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11149: Tue Nov 5 07:53:54 2024 write: IOPS=0, BW=2259KiB/s (2313kB/s)(12.0MiB/5439msec) clat (msec): min=1066, max=3048, avg=1794.01, stdev=1090.87 lat (msec): min=1067, max=3048, avg=1794.28, stdev=1090.87 clat percentiles (msec): | 1.00th=[ 1070], 5.00th=[ 1070], 10.00th=[ 1070], 20.00th=[ 1070], | 30.00th=[ 1070], 40.00th=[ 1267], 50.00th=[ 1267], 60.00th=[ 1267], | 70.00th=[ 3037], 80.00th=[ 3037], 90.00th=[ 3037], 95.00th=[ 3037], | 99.00th=[ 3037], 99.50th=[ 3037], 99.90th=[ 3037], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 7699, max= 8192, per=100.00%, avg=7945.50, stdev=348.60, samples=2 iops : min= 1, max= 2, avg= 1.50, stdev= 0.71, samples=2 cpu : usr=0.00%, sys=6.31%, ctx=39, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=11150: Tue Nov 5 07:53:54 2024 write: IOPS=0, BW=2999KiB/s (3071kB/s)(12.0MiB/4097msec) clat (msec): min=88, max=3724, avg=1354.14, stdev=2054.21 lat (msec): min=88, max=3724, avg=1354.41, stdev=2054.18 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 89], 10.00th=[ 89], 20.00th=[ 89], | 30.00th=[ 89], 40.00th=[ 251], 50.00th=[ 251], 60.00th=[ 251], | 70.00th=[ 3708], 80.00th=[ 3708], 90.00th=[ 3708], 95.00th=[ 3708], | 99.00th=[ 3708], 99.50th=[ 3708], 99.90th=[ 3708], 99.95th=[ 3708], | 99.99th=[ 3708] bw ( KiB/s): min= 8192, max= 8192, per=100.00%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33% cpu : usr=0.00%, sys=7.35%, ctx=35, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3137KiB/s (3213kB/s), 998KiB/s-2353KiB/s (1022kB/s-2410kB/s), io=16.0MiB (16.8MB), run=4103-5222msec WRITE: bw=6025KiB/s (6169kB/s), 1997KiB/s-2999KiB/s (2045kB/s-3071kB/s), io=32.0MiB (33.6MB), run=4097-5439msec