== sanity test 398b: DIO and buffer IO race ============== 04:07:39 (1738487259) /usr/bin/fio 48+0 records in 48+0 records out 50331648 bytes (50 MB) copied, 1.47328 s, 34.2 MB/s mix direct rw 4096 by fio with 4 jobs... mix buffer rw 4096 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5380: Sun Feb 2 04:07:54 2025 read: IOPS=131, BW=524KiB/s (537kB/s)(5992KiB/11426msec) clat (usec): min=52, max=172111, avg=3581.25, stdev=5541.66 lat (usec): min=52, max=172112, avg=3582.15, stdev=5541.80 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 68], 10.00th=[ 78], 20.00th=[ 1795], | 30.00th=[ 2073], 40.00th=[ 2343], 50.00th=[ 2704], 60.00th=[ 3228], | 70.00th=[ 4047], 80.00th=[ 5211], 90.00th=[ 7046], 95.00th=[ 8586], | 99.00th=[ 15401], 99.50th=[ 19792], 99.90th=[ 53740], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 320, max= 686, per=24.80%, avg=513.59, stdev=102.08, samples=22 iops : min= 80, max= 171, avg=128.32, stdev=25.50, samples=22 write: IOPS=137, BW=551KiB/s (564kB/s)(6296KiB/11426msec) clat (usec): min=421, max=32923, avg=3806.13, stdev=3893.00 lat (usec): min=421, max=32924, avg=3806.93, stdev=3893.04 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 506], 10.00th=[ 529], 20.00th=[ 570], | 30.00th=[ 725], 40.00th=[ 1385], 50.00th=[ 2868], 60.00th=[ 3752], | 70.00th=[ 4883], 80.00th=[ 6390], 90.00th=[ 9110], 95.00th=[10945], | 99.00th=[15795], 99.50th=[21103], 99.90th=[31851], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 359, max= 880, per=24.85%, avg=538.68, stdev=113.15, samples=22 iops : min= 89, max= 220, avg=134.59, stdev=28.35, samples=22 lat (usec) : 100=7.85%, 250=0.59%, 500=1.92%, 750=14.13%, 1000=2.08% lat (msec) : 2=9.02%, 4=30.24%, 10=29.23%, 20=4.43%, 50=0.46% lat (msec) : 100=0.03%, 250=0.03% cpu : usr=0.17%, sys=22.77%, ctx=5371, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5381: Sun Feb 2 04:07:54 2025 read: IOPS=128, BW=514KiB/s (527kB/s)(5940KiB/11550msec) clat (usec): min=52, max=33551, avg=3402.16, stdev=3014.73 lat (usec): min=53, max=33551, avg=3402.75, stdev=3014.69 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 68], 10.00th=[ 77], 20.00th=[ 1745], | 30.00th=[ 2057], 40.00th=[ 2376], 50.00th=[ 2769], 60.00th=[ 3261], | 70.00th=[ 3982], 80.00th=[ 5145], 90.00th=[ 6849], 95.00th=[ 8455], | 99.00th=[13566], 99.50th=[16057], 99.90th=[32113], 99.95th=[33424], | 99.99th=[33424] bw ( KiB/s): min= 336, max= 710, per=24.78%, avg=513.13, stdev=107.38, samples=23 iops : min= 84, max= 177, avg=128.22, stdev=26.78, samples=23 write: IOPS=137, BW=550KiB/s (563kB/s)(6348KiB/11550msec) clat (usec): min=429, max=167583, avg=4049.65, stdev=5614.49 lat (usec): min=430, max=167585, avg=4050.72, stdev=5614.35 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 519], 10.00th=[ 537], 20.00th=[ 603], | 30.00th=[ 881], 40.00th=[ 1926], 50.00th=[ 3163], 60.00th=[ 3982], | 70.00th=[ 5080], 80.00th=[ 6456], 90.00th=[ 9110], 95.00th=[ 10945], | 99.00th=[ 17433], 99.50th=[ 20317], 99.90th=[ 36439], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 352, max= 758, per=25.22%, avg=546.87, stdev=110.37, samples=23 iops : min= 88, max= 189, avg=136.65, stdev=27.54, samples=23 lat (usec) : 100=7.98%, 250=0.55%, 500=1.14%, 750=13.12%, 1000=2.57% lat (msec) : 2=8.56%, 4=31.09%, 10=30.11%, 20=4.36%, 50=0.49% lat (msec) : 250=0.03% cpu : usr=0.18%, sys=22.44%, ctx=5467, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5382: Sun Feb 2 04:07:54 2025 read: IOPS=129, BW=520KiB/s (532kB/s)(6028KiB/11594msec) clat (usec): min=50, max=167539, avg=3460.83, stdev=5292.96 lat (usec): min=50, max=167540, avg=3461.58, stdev=5292.94 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 67], 10.00th=[ 75], 20.00th=[ 1582], | 30.00th=[ 2040], 40.00th=[ 2376], 50.00th=[ 2704], 60.00th=[ 3261], | 70.00th=[ 4178], 80.00th=[ 5276], 90.00th=[ 6783], 95.00th=[ 8225], | 99.00th=[ 12256], 99.50th=[ 16057], 99.90th=[ 63701], 99.95th=[166724], | 99.99th=[166724] bw ( KiB/s): min= 320, max= 888, per=25.00%, avg=517.70, stdev=147.40, samples=23 iops : min= 80, max= 222, avg=129.30, stdev=36.92, samples=23 write: IOPS=134, BW=540KiB/s (553kB/s)(6260KiB/11594msec) clat (usec): min=459, max=55628, avg=4031.10, stdev=4298.03 lat (usec): min=460, max=55629, avg=4031.90, stdev=4297.93 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 510], 10.00th=[ 529], 20.00th=[ 578], | 30.00th=[ 791], 40.00th=[ 1795], 50.00th=[ 2999], 60.00th=[ 3785], | 70.00th=[ 5145], 80.00th=[ 6587], 90.00th=[ 9372], 95.00th=[11994], | 99.00th=[18220], 99.50th=[21890], 99.90th=[36439], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min= 376, max= 936, per=24.73%, avg=536.13, stdev=125.45, samples=23 iops : min= 94, max= 234, avg=133.91, stdev=31.41, samples=23 lat (usec) : 100=8.66%, 250=0.78%, 500=1.66%, 750=13.35%, 1000=2.38% lat (msec) : 2=8.30%, 4=29.92%, 10=29.72%, 20=4.85%, 50=0.29% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.21%, sys=22.57%, ctx=5457, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5383: Sun Feb 2 04:07:54 2025 read: IOPS=131, BW=525KiB/s (537kB/s)(6052KiB/11537msec) clat (usec): min=49, max=65003, avg=3302.96, stdev=3388.09 lat (usec): min=49, max=65003, avg=3303.85, stdev=3388.02 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 67], 10.00th=[ 75], 20.00th=[ 619], | 30.00th=[ 2057], 40.00th=[ 2343], 50.00th=[ 2704], 60.00th=[ 3163], | 70.00th=[ 3949], 80.00th=[ 5276], 90.00th=[ 6652], 95.00th=[ 8160], | 99.00th=[10814], 99.50th=[13566], 99.90th=[61604], 99.95th=[64750], | 99.99th=[64750] bw ( KiB/s): min= 224, max= 752, per=25.32%, avg=524.43, stdev=117.40, samples=23 iops : min= 56, max= 188, avg=131.04, stdev=29.43, samples=23 write: IOPS=135, BW=541KiB/s (553kB/s)(6236KiB/11537msec) clat (usec): min=428, max=212271, avg=4138.18, stdev=6772.40 lat (usec): min=428, max=212273, avg=4138.82, stdev=6772.42 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 515], 10.00th=[ 537], 20.00th=[ 594], | 30.00th=[ 807], 40.00th=[ 1893], 50.00th=[ 3032], 60.00th=[ 3884], | 70.00th=[ 4948], 80.00th=[ 6390], 90.00th=[ 9110], 95.00th=[ 11994], | 99.00th=[ 19530], 99.50th=[ 22152], 99.90th=[ 41681], 99.95th=[212861], | 99.99th=[212861] bw ( KiB/s): min= 248, max= 792, per=24.88%, avg=539.39, stdev=129.00, samples=23 iops : min= 62, max= 198, avg=134.78, stdev=32.31, samples=23 lat (usec) : 50=0.03%, 100=8.72%, 250=0.85%, 500=1.60%, 750=13.31% lat (usec) : 1000=2.15% lat (msec) : 2=8.07%, 4=31.18%, 10=29.13%, 20=4.43%, 50=0.42% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=0.20%, sys=21.95%, ctx=5444, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2071KiB/s (2121kB/s), 514KiB/s-525KiB/s (527kB/s-537kB/s), io=23.4MiB (24.6MB), run=11426-11594msec WRITE: bw=2168KiB/s (2220kB/s), 540KiB/s-551KiB/s (553kB/s-564kB/s), io=24.6MiB (25.7MB), run=11426-11594msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5375: Sun Feb 2 04:08:15 2025 read: IOPS=46, BW=187KiB/s (191kB/s)(5992KiB/32070msec) clat (usec): min=1387, max=173409, avg=3604.90, stdev=5296.31 lat (usec): min=1388, max=173412, avg=3605.54, stdev=5296.34 clat percentiles (usec): | 1.00th=[ 1516], 5.00th=[ 1663], 10.00th=[ 1778], 20.00th=[ 1876], | 30.00th=[ 1975], 40.00th=[ 2114], 50.00th=[ 2507], 60.00th=[ 3032], | 70.00th=[ 3294], 80.00th=[ 3916], 90.00th=[ 7111], 95.00th=[ 9241], | 99.00th=[ 14746], 99.50th=[ 17957], 99.90th=[ 48497], 99.95th=[173016], | 99.99th=[173016] bw ( KiB/s): min= 55, max= 320, per=25.00%, avg=186.77, stdev=71.86, samples=64 iops : min= 13, max= 80, avg=46.61, stdev=17.98, samples=64 write: IOPS=49, BW=196KiB/s (201kB/s)(6296KiB/32070msec) clat (usec): min=8469, max=63295, avg=16875.94, stdev=6808.20 lat (usec): min=8470, max=63296, avg=16876.66, stdev=6808.21 clat percentiles (usec): | 1.00th=[ 9110], 5.00th=[10028], 10.00th=[10552], 20.00th=[11469], | 30.00th=[12649], 40.00th=[14091], 50.00th=[15533], 60.00th=[16909], | 70.00th=[17957], 80.00th=[19792], 90.00th=[26084], 95.00th=[31327], | 99.00th=[39584], 99.50th=[42206], 99.90th=[62129], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min= 48, max= 280, per=25.07%, avg=196.02, stdev=59.99, samples=64 iops : min= 12, max= 70, avg=48.92, stdev=14.97, samples=64 lat (msec) : 2=15.56%, 4=23.67%, 10=10.19%, 20=40.46%, 50=9.93% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.06%, sys=5.40%, ctx=3721, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1498,1574,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5376: Sun Feb 2 04:08:15 2025 read: IOPS=46, BW=185KiB/s (189kB/s)(5940KiB/32141msec) clat (usec): min=1367, max=177778, avg=3511.43, stdev=5244.84 lat (usec): min=1367, max=177779, avg=3512.09, stdev=5244.86 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1696], 10.00th=[ 1778], 20.00th=[ 1893], | 30.00th=[ 1975], 40.00th=[ 2114], 50.00th=[ 2474], 60.00th=[ 3032], | 70.00th=[ 3326], 80.00th=[ 3851], 90.00th=[ 6194], 95.00th=[ 8717], | 99.00th=[ 16057], 99.50th=[ 18482], 99.90th=[ 24511], 99.95th=[177210], | 99.99th=[177210] bw ( KiB/s): min= 16, max= 344, per=24.75%, avg=184.89, stdev=77.91, samples=64 iops : min= 4, max= 86, avg=46.14, stdev=19.49, samples=64 write: IOPS=49, BW=198KiB/s (202kB/s)(6348KiB/32141msec) clat (usec): min=7906, max=77257, avg=16899.18, stdev=6783.60 lat (usec): min=7907, max=77258, avg=16899.88, stdev=6783.61 clat percentiles (usec): | 1.00th=[ 9110], 5.00th=[ 9896], 10.00th=[10552], 20.00th=[11469], | 30.00th=[12518], 40.00th=[14091], 50.00th=[15401], 60.00th=[16712], | 70.00th=[17957], 80.00th=[20579], 90.00th=[26346], 95.00th=[30278], | 99.00th=[40109], 99.50th=[43254], 99.90th=[63177], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min= 64, max= 280, per=25.18%, avg=196.89, stdev=59.29, samples=64 iops : min= 16, max= 70, avg=49.14, stdev=14.82, samples=64 lat (msec) : 2=15.59%, 4=23.86%, 10=9.96%, 20=39.36%, 50=11.07% lat (msec) : 100=0.13%, 250=0.03% cpu : usr=0.07%, sys=5.45%, ctx=3763, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1485,1587,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5377: Sun Feb 2 04:08:15 2025 read: IOPS=47, BW=189KiB/s (193kB/s)(6028KiB/31978msec) clat (usec): min=1321, max=35281, avg=3597.73, stdev=2936.94 lat (usec): min=1321, max=35282, avg=3598.36, stdev=2936.92 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1680], 10.00th=[ 1778], 20.00th=[ 1876], | 30.00th=[ 1991], 40.00th=[ 2180], 50.00th=[ 2638], 60.00th=[ 3097], | 70.00th=[ 3425], 80.00th=[ 4146], 90.00th=[ 7242], 95.00th=[ 9503], | 99.00th=[14746], 99.50th=[17171], 99.90th=[33162], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 48, max= 320, per=25.14%, avg=187.81, stdev=68.55, samples=63 iops : min= 12, max= 80, avg=46.95, stdev=17.14, samples=63 write: IOPS=48, BW=196KiB/s (200kB/s)(6260KiB/31978msec) clat (msec): min=8, max=199, avg=16.90, stdev= 8.26 lat (msec): min=8, max=199, avg=16.90, stdev= 8.26 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 14], 50.00th=[ 16], 60.00th=[ 17], | 70.00th=[ 18], 80.00th=[ 20], 90.00th=[ 27], 95.00th=[ 32], | 99.00th=[ 40], 99.50th=[ 43], 99.90th=[ 70], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 48, max= 280, per=24.91%, avg=194.79, stdev=63.08, samples=63 iops : min= 12, max= 70, avg=48.70, stdev=15.77, samples=63 lat (msec) : 2=15.01%, 4=23.83%, 10=10.32%, 20=41.11%, 50=9.54% lat (msec) : 100=0.16%, 250=0.03% cpu : usr=0.11%, sys=5.39%, ctx=3750, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1507,1565,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5378: Sun Feb 2 04:08:15 2025 read: IOPS=47, BW=189KiB/s (194kB/s)(6052KiB/31962msec) clat (usec): min=1358, max=46009, avg=3580.42, stdev=3250.34 lat (usec): min=1359, max=46010, avg=3581.17, stdev=3250.28 clat percentiles (usec): | 1.00th=[ 1500], 5.00th=[ 1663], 10.00th=[ 1745], 20.00th=[ 1844], | 30.00th=[ 1942], 40.00th=[ 2089], 50.00th=[ 2474], 60.00th=[ 3032], | 70.00th=[ 3359], 80.00th=[ 4015], 90.00th=[ 7111], 95.00th=[ 9503], | 99.00th=[16909], 99.50th=[21627], 99.90th=[33424], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 32, max= 344, per=25.24%, avg=188.57, stdev=76.24, samples=63 iops : min= 8, max= 86, avg=47.14, stdev=19.06, samples=63 write: IOPS=48, BW=195KiB/s (200kB/s)(6236KiB/31962msec) clat (msec): min=8, max=223, avg=16.96, stdev= 8.50 lat (msec): min=8, max=223, avg=16.96, stdev= 8.50 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 11], 20.00th=[ 12], | 30.00th=[ 13], 40.00th=[ 15], 50.00th=[ 16], 60.00th=[ 17], | 70.00th=[ 18], 80.00th=[ 20], 90.00th=[ 27], 95.00th=[ 31], | 99.00th=[ 41], 99.50th=[ 47], 99.90th=[ 70], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 48, max= 272, per=24.83%, avg=194.16, stdev=61.57, samples=63 iops : min= 12, max= 68, avg=48.54, stdev=15.39, samples=63 lat (msec) : 2=16.86%, 4=22.43%, 10=10.19%, 20=40.33%, 50=10.06% lat (msec) : 100=0.10%, 250=0.03% cpu : usr=0.09%, sys=5.43%, ctx=3797, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1513,1559,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=747KiB/s (765kB/s), 185KiB/s-189KiB/s (189kB/s-194kB/s), io=23.4MiB (24.6MB), run=31962-32141msec WRITE: bw=782KiB/s (801kB/s), 195KiB/s-198KiB/s (200kB/s-202kB/s), io=24.6MiB (25.7MB), run=31962-32141msec mix direct rw 16384 by fio with 4 jobs... mix buffer rw 16384 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5422: Sun Feb 2 04:08:18 2025 read: IOPS=157, BW=2520KiB/s (2581kB/s)(5920KiB/2349msec) clat (usec): min=48, max=27223, avg=2677.82, stdev=2858.88 lat (usec): min=49, max=27223, avg=2678.33, stdev=2858.89 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 68], 10.00th=[ 71], 20.00th=[ 80], | 30.00th=[ 116], 40.00th=[ 2057], 50.00th=[ 2409], 60.00th=[ 2868], | 70.00th=[ 3556], 80.00th=[ 4424], 90.00th=[ 5407], 95.00th=[ 6390], | 99.00th=[11863], 99.50th=[24773], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 1632, max= 3136, per=24.72%, avg=2520.00, stdev=642.33, samples=4 iops : min= 102, max= 196, avg=157.50, stdev=40.15, samples=4 write: IOPS=169, BW=2711KiB/s (2776kB/s)(6368KiB/2349msec) clat (usec): min=573, max=26414, avg=3148.11, stdev=3364.37 lat (usec): min=573, max=26416, avg=3149.09, stdev=3364.28 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 709], 10.00th=[ 766], 20.00th=[ 832], | 30.00th=[ 914], 40.00th=[ 1123], 50.00th=[ 1844], 60.00th=[ 2999], | 70.00th=[ 3884], 80.00th=[ 5145], 90.00th=[ 6456], 95.00th=[ 9110], | 99.00th=[20055], 99.50th=[23725], 99.90th=[26346], 99.95th=[26346], | 99.99th=[26346] bw ( KiB/s): min= 1856, max= 3552, per=25.43%, avg=2728.00, stdev=823.08, samples=4 iops : min= 116, max= 222, avg=170.50, stdev=51.44, samples=4 lat (usec) : 50=0.13%, 100=13.80%, 250=1.69%, 500=0.13%, 750=4.30% lat (usec) : 1000=14.71% lat (msec) : 2=9.64%, 4=28.52%, 10=24.48%, 20=1.82%, 50=0.78% cpu : usr=0.17%, sys=27.73%, ctx=1402, majf=0, minf=35 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5424: Sun Feb 2 04:08:18 2025 read: IOPS=160, BW=2570KiB/s (2632kB/s)(5744KiB/2235msec) clat (usec): min=51, max=40324, avg=3073.97, stdev=3298.47 lat (usec): min=51, max=40325, avg=3074.87, stdev=3298.20 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 69], 10.00th=[ 75], 20.00th=[ 93], | 30.00th=[ 2040], 40.00th=[ 2376], 50.00th=[ 2671], 60.00th=[ 3032], | 70.00th=[ 3752], 80.00th=[ 4490], 90.00th=[ 6259], 95.00th=[ 7570], | 99.00th=[15139], 99.50th=[17957], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 2208, max= 3200, per=25.97%, avg=2648.00, stdev=500.79, samples=4 iops : min= 138, max= 200, avg=165.50, stdev=31.30, samples=4 write: IOPS=182, BW=2928KiB/s (2998kB/s)(6544KiB/2235msec) clat (usec): min=575, max=14829, avg=2727.72, stdev=2605.26 lat (usec): min=576, max=14830, avg=2730.03, stdev=2607.03 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 685], 10.00th=[ 717], 20.00th=[ 816], | 30.00th=[ 889], 40.00th=[ 1057], 50.00th=[ 1418], 60.00th=[ 2409], | 70.00th=[ 3228], 80.00th=[ 4752], 90.00th=[ 6456], 95.00th=[ 7898], | 99.00th=[11731], 99.50th=[12387], 99.90th=[14877], 99.95th=[14877], | 99.99th=[14877] bw ( KiB/s): min= 2528, max= 3264, per=27.59%, avg=2960.00, stdev=341.17, samples=4 iops : min= 158, max= 204, avg=185.00, stdev=21.32, samples=4 lat (usec) : 100=10.42%, 250=1.56%, 500=0.13%, 750=6.90%, 1000=12.89% lat (msec) : 2=11.59%, 4=31.12%, 10=23.18%, 20=2.08%, 50=0.13% cpu : usr=0.22%, sys=29.19%, ctx=1454, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5426: Sun Feb 2 04:08:18 2025 read: IOPS=177, BW=2840KiB/s (2908kB/s)(6256KiB/2203msec) clat (usec): min=49, max=25789, avg=2774.17, stdev=2900.77 lat (usec): min=49, max=25789, avg=2774.67, stdev=2900.79 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 68], 20.00th=[ 82], | 30.00th=[ 135], 40.00th=[ 2147], 50.00th=[ 2442], 60.00th=[ 2933], | 70.00th=[ 3818], 80.00th=[ 4555], 90.00th=[ 5604], 95.00th=[ 7046], | 99.00th=[11863], 99.50th=[25560], 99.90th=[25822], 99.95th=[25822], | 99.99th=[25822] bw ( KiB/s): min= 1952, max= 3424, per=27.30%, avg=2784.00, stdev=624.89, samples=4 iops : min= 122, max= 214, avg=174.00, stdev=39.06, samples=4 write: IOPS=171, BW=2738KiB/s (2804kB/s)(6032KiB/2203msec) clat (usec): min=580, max=20820, avg=2684.79, stdev=2614.89 lat (usec): min=581, max=20821, avg=2685.57, stdev=2614.90 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 676], 10.00th=[ 717], 20.00th=[ 791], | 30.00th=[ 873], 40.00th=[ 1012], 50.00th=[ 1336], 60.00th=[ 2573], | 70.00th=[ 3326], 80.00th=[ 4555], 90.00th=[ 6587], 95.00th=[ 7701], | 99.00th=[10421], 99.50th=[19530], 99.90th=[20841], 99.95th=[20841], | 99.99th=[20841] bw ( KiB/s): min= 2400, max= 3296, per=25.13%, avg=2696.00, stdev=410.94, samples=4 iops : min= 150, max= 206, avg=168.50, stdev=25.68, samples=4 lat (usec) : 50=0.13%, 100=13.54%, 250=2.34%, 500=0.52%, 750=7.68% lat (usec) : 1000=11.72% lat (msec) : 2=9.64%, 4=28.78%, 10=24.22%, 20=1.04%, 50=0.39% cpu : usr=0.23%, sys=28.47%, ctx=1344, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5428: Sun Feb 2 04:08:18 2025 read: IOPS=165, BW=2655KiB/s (2719kB/s)(6032KiB/2272msec) clat (usec): min=51, max=16803, avg=2806.68, stdev=2599.81 lat (usec): min=51, max=16804, avg=2807.56, stdev=2599.77 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 63], 10.00th=[ 71], 20.00th=[ 83], | 30.00th=[ 161], 40.00th=[ 2245], 50.00th=[ 2606], 60.00th=[ 3064], | 70.00th=[ 3687], 80.00th=[ 4686], 90.00th=[ 5997], 95.00th=[ 7570], | 99.00th=[10683], 99.50th=[16319], 99.90th=[16909], 99.95th=[16909], | 99.99th=[16909] bw ( KiB/s): min= 1952, max= 3328, per=25.89%, avg=2640.00, stdev=569.14, samples=4 iops : min= 122, max= 208, avg=165.00, stdev=35.57, samples=4 write: IOPS=172, BW=2754KiB/s (2820kB/s)(6256KiB/2272msec) clat (usec): min=588, max=23741, avg=2978.53, stdev=3111.02 lat (usec): min=589, max=23742, avg=2980.23, stdev=3110.53 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 693], 10.00th=[ 734], 20.00th=[ 799], | 30.00th=[ 914], 40.00th=[ 1029], 50.00th=[ 1418], 60.00th=[ 2573], | 70.00th=[ 3556], 80.00th=[ 5145], 90.00th=[ 6980], 95.00th=[ 8225], | 99.00th=[16188], 99.50th=[22414], 99.90th=[23725], 99.95th=[23725], | 99.99th=[23725] bw ( KiB/s): min= 2144, max= 3456, per=25.73%, avg=2760.00, stdev=547.67, samples=4 iops : min= 134, max= 216, avg=172.50, stdev=34.23, samples=4 lat (usec) : 100=12.50%, 250=2.86%, 500=0.52%, 750=6.25%, 1000=13.41% lat (msec) : 2=8.59%, 4=29.04%, 10=24.87%, 20=1.69%, 50=0.26% cpu : usr=0.13%, sys=27.70%, ctx=1527, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=9.96MiB/s (10.4MB/s), 2520KiB/s-2840KiB/s (2581kB/s-2908kB/s), io=23.4MiB (24.5MB), run=2203-2349msec WRITE: bw=10.5MiB/s (10.0MB/s), 2711KiB/s-2928KiB/s (2776kB/s-2998kB/s), io=24.6MiB (25.8MB), run=2203-2349msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5423: Sun Feb 2 04:08:23 2025 read: IOPS=49, BW=797KiB/s (816kB/s)(5920KiB/7428msec) clat (usec): min=1659, max=27020, avg=3214.93, stdev=2718.96 lat (usec): min=1659, max=27021, avg=3215.57, stdev=2718.96 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 1876], 10.00th=[ 1909], 20.00th=[ 1958], | 30.00th=[ 2040], 40.00th=[ 2114], 50.00th=[ 2212], 60.00th=[ 2343], | 70.00th=[ 2540], 80.00th=[ 3228], 90.00th=[ 5932], 95.00th=[ 9503], | 99.00th=[13829], 99.50th=[16319], 99.90th=[27132], 99.95th=[27132], | 99.99th=[27132] bw ( KiB/s): min= 448, max= 1312, per=24.18%, avg=779.43, stdev=250.91, samples=14 iops : min= 28, max= 82, avg=48.71, stdev=15.68, samples=14 write: IOPS=53, BW=857KiB/s (878kB/s)(6368KiB/7428msec) clat (usec): min=9039, max=56981, avg=15623.96, stdev=5449.72 lat (usec): min=9041, max=56983, avg=15625.22, stdev=5449.91 clat percentiles (usec): | 1.00th=[ 9241], 5.00th=[10028], 10.00th=[10290], 20.00th=[10945], | 30.00th=[12256], 40.00th=[13698], 50.00th=[15008], 60.00th=[16057], | 70.00th=[16909], 80.00th=[17957], 90.00th=[22152], 95.00th=[25822], | 99.00th=[37487], 99.50th=[39584], 99.90th=[56886], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min= 416, max= 1024, per=24.87%, avg=843.43, stdev=212.49, samples=14 iops : min= 26, max= 64, avg=52.71, stdev=13.28, samples=14 lat (msec) : 2=12.37%, 4=27.99%, 10=7.81%, 20=44.66%, 50=7.03% lat (msec) : 100=0.13% cpu : usr=0.13%, sys=6.75%, ctx=862, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=370,398,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5425: Sun Feb 2 04:08:23 2025 read: IOPS=48, BW=778KiB/s (797kB/s)(5744KiB/7382msec) clat (usec): min=1757, max=28972, avg=3172.78, stdev=2664.21 lat (usec): min=1758, max=28973, avg=3173.42, stdev=2664.19 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1860], 10.00th=[ 1909], 20.00th=[ 1958], | 30.00th=[ 2024], 40.00th=[ 2073], 50.00th=[ 2180], 60.00th=[ 2311], | 70.00th=[ 2507], 80.00th=[ 3720], 90.00th=[ 6325], 95.00th=[ 8291], | 99.00th=[14353], 99.50th=[17433], 99.90th=[28967], 99.95th=[28967], | 99.99th=[28967] bw ( KiB/s): min= 288, max= 1312, per=24.39%, avg=786.29, stdev=292.67, samples=14 iops : min= 18, max= 82, avg=49.14, stdev=18.29, samples=14 write: IOPS=55, BW=886KiB/s (908kB/s)(6544KiB/7382msec) clat (usec): min=8979, max=45848, avg=15071.20, stdev=4883.33 lat (usec): min=8980, max=45851, avg=15072.37, stdev=4883.34 clat percentiles (usec): | 1.00th=[ 9372], 5.00th=[ 9634], 10.00th=[10028], 20.00th=[10683], | 30.00th=[11863], 40.00th=[13304], 50.00th=[14222], 60.00th=[15401], | 70.00th=[16581], 80.00th=[17957], 90.00th=[21627], 95.00th=[24773], | 99.00th=[29754], 99.50th=[35390], 99.90th=[45876], 99.95th=[45876], | 99.99th=[45876] bw ( KiB/s): min= 480, max= 1120, per=25.88%, avg=877.71, stdev=221.96, samples=14 iops : min= 30, max= 70, avg=54.86, stdev=13.87, samples=14 lat (msec) : 2=12.50%, 4=25.65%, 10=12.11%, 20=42.97%, 50=6.77% cpu : usr=0.18%, sys=6.68%, ctx=873, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=359,409,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5427: Sun Feb 2 04:08:23 2025 read: IOPS=54, BW=867KiB/s (888kB/s)(6256KiB/7214msec) clat (usec): min=1705, max=35361, avg=3097.15, stdev=3158.60 lat (usec): min=1706, max=35362, avg=3097.80, stdev=3158.70 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 1844], 10.00th=[ 1876], 20.00th=[ 1942], | 30.00th=[ 2008], 40.00th=[ 2073], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2409], 80.00th=[ 2802], 90.00th=[ 5080], 95.00th=[ 7832], | 99.00th=[17695], 99.50th=[27132], 99.90th=[35390], 99.95th=[35390], | 99.99th=[35390] bw ( KiB/s): min= 384, max= 1472, per=27.07%, avg=872.79, stdev=373.16, samples=14 iops : min= 24, max= 92, avg=54.43, stdev=23.24, samples=14 write: IOPS=52, BW=836KiB/s (856kB/s)(6032KiB/7214msec) clat (usec): min=9194, max=46766, avg=15597.61, stdev=5256.13 lat (usec): min=9195, max=46769, avg=15598.79, stdev=5256.17 clat percentiles (usec): | 1.00th=[ 9372], 5.00th=[10028], 10.00th=[10421], 20.00th=[11207], | 30.00th=[12125], 40.00th=[13698], 50.00th=[14877], 60.00th=[15926], | 70.00th=[17171], 80.00th=[17957], 90.00th=[22676], 95.00th=[26346], | 99.00th=[31851], 99.50th=[44303], 99.90th=[46924], 99.95th=[46924], | 99.99th=[46924] bw ( KiB/s): min= 384, max= 1024, per=24.45%, avg=829.43, stdev=211.09, samples=14 iops : min= 24, max= 64, avg=51.71, stdev=13.09, samples=14 lat (msec) : 2=14.32%, 4=28.91%, 10=7.68%, 20=41.80%, 50=7.29% cpu : usr=0.10%, sys=6.78%, ctx=874, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=391,377,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5429: Sun Feb 2 04:08:23 2025 read: IOPS=51, BW=824KiB/s (844kB/s)(6032KiB/7319msec) clat (usec): min=1598, max=22932, avg=3076.00, stdev=2658.28 lat (usec): min=1598, max=22933, avg=3076.64, stdev=2658.28 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 1827], 10.00th=[ 1893], 20.00th=[ 1958], | 30.00th=[ 2008], 40.00th=[ 2089], 50.00th=[ 2180], 60.00th=[ 2278], | 70.00th=[ 2474], 80.00th=[ 2900], 90.00th=[ 5276], 95.00th=[ 8717], | 99.00th=[15795], 99.50th=[17171], 99.90th=[22938], 99.95th=[22938], | 99.99th=[22938] bw ( KiB/s): min= 384, max= 1312, per=25.23%, avg=813.36, stdev=249.49, samples=14 iops : min= 24, max= 82, avg=50.71, stdev=15.56, samples=14 write: IOPS=53, BW=855KiB/s (875kB/s)(6256KiB/7319msec) clat (usec): min=8417, max=35804, avg=15439.10, stdev=4714.14 lat (usec): min=8418, max=35805, avg=15440.28, stdev=4714.18 clat percentiles (usec): | 1.00th=[ 9372], 5.00th=[ 9896], 10.00th=[10290], 20.00th=[11076], | 30.00th=[12387], 40.00th=[13829], 50.00th=[15139], 60.00th=[16188], | 70.00th=[16909], 80.00th=[17957], 90.00th=[21365], 95.00th=[25297], | 99.00th=[32900], 99.50th=[34866], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] bw ( KiB/s): min= 416, max= 1024, per=25.06%, avg=849.93, stdev=201.58, samples=14 iops : min= 26, max= 64, avg=53.00, stdev=12.50, samples=14 lat (msec) : 2=13.28%, 4=29.17%, 10=7.94%, 20=43.75%, 50=5.86% cpu : usr=0.14%, sys=6.71%, ctx=858, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=377,391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3225KiB/s (3302kB/s), 778KiB/s-867KiB/s (797kB/s-888kB/s), io=23.4MiB (24.5MB), run=7214-7428msec WRITE: bw=3393KiB/s (3474kB/s), 836KiB/s-886KiB/s (856kB/s-908kB/s), io=24.6MiB (25.8MB), run=7214-7428msec mix direct rw 1048576 by fio with 4 jobs... mix buffer rw 1048576 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5441: Sun Feb 2 04:08:25 2025 read: IOPS=21, BW=21.5MiB/s (22.5MB/s)(7168KiB/326msec) clat (usec): min=417, max=50166, avg=27224.21, stdev=20240.48 lat (usec): min=418, max=50166, avg=27225.12, stdev=20240.62 clat percentiles (usec): | 1.00th=[ 416], 5.00th=[ 416], 10.00th=[ 416], 20.00th=[ 523], | 30.00th=[26870], 40.00th=[26870], 50.00th=[31065], 60.00th=[32900], | 70.00th=[32900], 80.00th=[48497], 90.00th=[50070], 95.00th=[50070], | 99.00th=[50070], 99.50th=[50070], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] write: IOPS=15, BW=15.3MiB/s (16.1MB/s)(5120KiB/326msec) clat (usec): min=15195, max=36121, avg=25364.77, stdev=9616.77 lat (usec): min=15213, max=36164, avg=25395.09, stdev=9629.52 clat percentiles (usec): | 1.00th=[15139], 5.00th=[15139], 10.00th=[15139], 20.00th=[15139], | 30.00th=[16450], 40.00th=[16450], 50.00th=[25297], 60.00th=[25297], | 70.00th=[33817], 80.00th=[33817], 90.00th=[35914], 95.00th=[35914], | 99.00th=[35914], 99.50th=[35914], 99.90th=[35914], 99.95th=[35914], | 99.99th=[35914] lat (usec) : 500=8.33%, 750=8.33% lat (msec) : 20=16.67%, 50=58.33%, 100=8.33% cpu : usr=0.00%, sys=52.31%, ctx=867, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5443: Sun Feb 2 04:08:25 2025 read: IOPS=2, BW=2833KiB/s (2901kB/s)(2048KiB/723msec) clat (usec): min=37332, max=40100, avg=38716.36, stdev=1957.65 lat (usec): min=37333, max=40101, avg=38717.37, stdev=1957.18 clat percentiles (usec): | 1.00th=[37487], 5.00th=[37487], 10.00th=[37487], 20.00th=[37487], | 30.00th=[37487], 40.00th=[37487], 50.00th=[37487], 60.00th=[40109], | 70.00th=[40109], 80.00th=[40109], 90.00th=[40109], 95.00th=[40109], | 99.00th=[40109], 99.50th=[40109], 99.90th=[40109], 99.95th=[40109], | 99.99th=[40109] bw ( KiB/s): min= 4023, max= 4023, per=14.20%, avg=4023.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=13, BW=13.8MiB/s (14.5MB/s)(10.0MiB/723msec) clat (usec): min=31197, max=98855, avg=61192.94, stdev=24113.54 lat (usec): min=31254, max=98896, avg=61238.45, stdev=24103.71 clat percentiles (usec): | 1.00th=[31327], 5.00th=[31327], 10.00th=[31327], 20.00th=[31851], | 30.00th=[38536], 40.00th=[51119], 50.00th=[56886], 60.00th=[57934], | 70.00th=[85459], 80.00th=[85459], 90.00th=[87557], 95.00th=[99091], | 99.00th=[99091], 99.50th=[99091], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min=12070, max=12070, per=30.44%, avg=12070.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=58.33% cpu : usr=0.14%, sys=34.07%, ctx=186, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5446: Sun Feb 2 04:08:25 2025 read: IOPS=12, BW=12.3MiB/s (12.9MB/s)(6144KiB/488msec) clat (usec): min=323, max=23545, avg=7808.34, stdev=11486.68 lat (usec): min=324, max=23546, avg=7808.95, stdev=11486.63 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 383], | 30.00th=[ 383], 40.00th=[ 424], 50.00th=[ 424], 60.00th=[ 482], | 70.00th=[21627], 80.00th=[21627], 90.00th=[23462], 95.00th=[23462], | 99.00th=[23462], 99.50th=[23462], 99.90th=[23462], 99.95th=[23462], | 99.99th=[23462] write: IOPS=12, BW=12.3MiB/s (12.9MB/s)(6144KiB/488msec) clat (msec): min=22, max=210, avg=70.13, stdev=70.86 lat (msec): min=22, max=210, avg=70.17, stdev=70.86 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 29], | 30.00th=[ 29], 40.00th=[ 36], 50.00th=[ 36], 60.00th=[ 57], | 70.00th=[ 68], 80.00th=[ 68], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] lat (usec) : 500=33.33% lat (msec) : 50=41.67%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=27.72%, ctx=164, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5447: Sun Feb 2 04:08:25 2025 read: IOPS=8, BW=8232KiB/s (8429kB/s)(5120KiB/622msec) clat (usec): min=327, max=32947, avg=23741.95, stdev=13637.75 lat (usec): min=328, max=32947, avg=23742.67, stdev=13637.78 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 326], 10.00th=[ 326], 20.00th=[ 326], | 30.00th=[23725], 40.00th=[23725], 50.00th=[28967], 60.00th=[28967], | 70.00th=[32900], 80.00th=[32900], 90.00th=[32900], 95.00th=[32900], | 99.00th=[32900], 99.50th=[32900], 99.90th=[32900], 99.95th=[32900], | 99.99th=[32900] bw ( KiB/s): min= 4096, max= 4096, per=14.46%, avg=4096.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 write: IOPS=11, BW=11.3MiB/s (11.8MB/s)(7168KiB/622msec) clat (msec): min=25, max=192, avg=66.55, stdev=57.45 lat (msec): min=25, max=192, avg=66.61, stdev=57.45 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 26], 10.00th=[ 26], 20.00th=[ 32], | 30.00th=[ 41], 40.00th=[ 41], 50.00th=[ 52], 60.00th=[ 62], | 70.00th=[ 62], 80.00th=[ 64], 90.00th=[ 192], 95.00th=[ 192], | 99.00th=[ 192], 99.50th=[ 192], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 8192, max= 8192, per=20.66%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 8, max= 8, avg= 8.00, stdev= 0.00, samples=1 lat (usec) : 500=8.33% lat (msec) : 50=58.33%, 100=25.00%, 250=8.33% cpu : usr=0.00%, sys=32.05%, ctx=542, majf=0, minf=33 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=27.7MiB/s (29.0MB/s), 2833KiB/s-21.5MiB/s (2901kB/s-22.5MB/s), io=20.0MiB (20.0MB), run=326-723msec WRITE: bw=38.7MiB/s (40.6MB/s), 11.3MiB/s-15.3MiB/s (11.8MB/s-16.1MB/s), io=28.0MiB (29.4MB), run=326-723msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5440: Sun Feb 2 04:08:25 2025 read: IOPS=12, BW=12.5MiB/s (13.1MB/s)(7168KiB/559msec) clat (msec): min=12, max=147, avg=46.07, stdev=47.09 lat (msec): min=12, max=147, avg=46.07, stdev=47.09 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 12], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 23], 40.00th=[ 23], 50.00th=[ 36], 60.00th=[ 41], | 70.00th=[ 41], 80.00th=[ 51], 90.00th=[ 148], 95.00th=[ 148], | 99.00th=[ 148], 99.50th=[ 148], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min=10219, max=10219, per=37.67%, avg=10219.00, stdev= 0.00, samples=1 iops : min= 9, max= 9, avg= 9.00, stdev= 0.00, samples=1 write: IOPS=8, BW=9159KiB/s (9379kB/s)(5120KiB/559msec) clat (usec): min=26785, max=65669, avg=40612.42, stdev=14794.58 lat (usec): min=26868, max=65697, avg=40675.21, stdev=14774.49 clat percentiles (usec): | 1.00th=[26870], 5.00th=[26870], 10.00th=[26870], 20.00th=[26870], | 30.00th=[33817], 40.00th=[33817], 50.00th=[38011], 60.00th=[38011], | 70.00th=[38536], 80.00th=[38536], 90.00th=[65799], 95.00th=[65799], | 99.00th=[65799], 99.50th=[65799], 99.90th=[65799], 99.95th=[65799], | 99.99th=[65799] bw ( KiB/s): min= 8175, max= 8175, per=21.53%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 20=16.67%, 50=58.33%, 100=16.67%, 250=8.33% cpu : usr=0.00%, sys=15.05%, ctx=46, majf=0, minf=34 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=7,5,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5442: Sun Feb 2 04:08:25 2025 read: IOPS=2, BW=3061KiB/s (3135kB/s)(2048KiB/669msec) clat (usec): min=16191, max=41565, avg=28878.25, stdev=17941.80 lat (usec): min=16193, max=41565, avg=28879.57, stdev=17941.03 clat percentiles (usec): | 1.00th=[16188], 5.00th=[16188], 10.00th=[16188], 20.00th=[16188], | 30.00th=[16188], 40.00th=[16188], 50.00th=[16188], 60.00th=[41681], | 70.00th=[41681], 80.00th=[41681], 90.00th=[41681], 95.00th=[41681], | 99.00th=[41681], 99.50th=[41681], 99.90th=[41681], 99.95th=[41681], | 99.99th=[41681] bw ( KiB/s): min= 4063, max= 4063, per=14.98%, avg=4063.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=14, BW=14.9MiB/s (15.7MB/s)(10.0MiB/669msec) clat (msec): min=33, max=115, avg=59.87, stdev=26.17 lat (msec): min=33, max=115, avg=59.93, stdev=26.17 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 34], 10.00th=[ 34], 20.00th=[ 35], | 30.00th=[ 38], 40.00th=[ 46], 50.00th=[ 50], 60.00th=[ 56], | 70.00th=[ 77], 80.00th=[ 77], 90.00th=[ 86], 95.00th=[ 115], | 99.00th=[ 115], 99.50th=[ 115], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min=12190, max=12190, per=32.10%, avg=12190.00, stdev= 0.00, samples=1 iops : min= 11, max= 11, avg=11.00, stdev= 0.00, samples=1 lat (msec) : 20=8.33%, 50=50.00%, 100=33.33%, 250=8.33% cpu : usr=0.15%, sys=14.07%, ctx=40, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2,10,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5444: Sun Feb 2 04:08:25 2025 read: IOPS=7, BW=8138KiB/s (8333kB/s)(6144KiB/755msec) clat (usec): min=23319, max=70135, avg=50916.58, stdev=20748.70 lat (usec): min=23320, max=70135, avg=50917.41, stdev=20748.75 clat percentiles (usec): | 1.00th=[23200], 5.00th=[23200], 10.00th=[23200], 20.00th=[25822], | 30.00th=[25822], 40.00th=[59507], 50.00th=[59507], 60.00th=[61604], | 70.00th=[65799], 80.00th=[65799], 90.00th=[69731], 95.00th=[69731], | 99.00th=[69731], 99.50th=[69731], 99.90th=[69731], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min= 4087, max= 4087, per=15.07%, avg=4087.00, stdev= 0.00, samples=1 iops : min= 3, max= 3, avg= 3.00, stdev= 0.00, samples=1 write: IOPS=7, BW=8138KiB/s (8333kB/s)(6144KiB/755msec) clat (msec): min=28, max=178, avg=72.35, stdev=55.77 lat (msec): min=28, max=178, avg=72.41, stdev=55.78 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 29], 10.00th=[ 29], 20.00th=[ 36], | 30.00th=[ 36], 40.00th=[ 47], 50.00th=[ 47], 60.00th=[ 59], | 70.00th=[ 88], 80.00th=[ 88], 90.00th=[ 178], 95.00th=[ 178], | 99.00th=[ 178], 99.50th=[ 178], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 8175, max= 8175, per=21.53%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=41.67%, 100=50.00%, 250=8.33% cpu : usr=0.00%, sys=11.27%, ctx=46, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=6,6,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5445: Sun Feb 2 04:08:25 2025 read: IOPS=7, BW=7665KiB/s (7849kB/s)(5120KiB/668msec) clat (usec): min=22405, max=62034, avg=33128.99, stdev=16319.24 lat (usec): min=22406, max=62035, avg=33129.77, stdev=16319.16 clat percentiles (usec): | 1.00th=[22414], 5.00th=[22414], 10.00th=[22414], 20.00th=[22414], | 30.00th=[25560], 40.00th=[25560], 50.00th=[27395], 60.00th=[27395], | 70.00th=[28443], 80.00th=[28443], 90.00th=[62129], 95.00th=[62129], | 99.00th=[62129], 99.50th=[62129], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min= 8126, max= 8126, per=29.96%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 write: IOPS=10, BW=10.5MiB/s (10.0MB/s)(7168KiB/668msec) clat (msec): min=32, max=137, avg=66.92, stdev=36.42 lat (msec): min=32, max=138, avg=66.98, stdev=36.44 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 33], 10.00th=[ 33], 20.00th=[ 42], | 30.00th=[ 51], 40.00th=[ 51], 50.00th=[ 52], 60.00th=[ 66], | 70.00th=[ 66], 80.00th=[ 90], 90.00th=[ 138], 95.00th=[ 138], | 99.00th=[ 138], 99.50th=[ 138], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 8126, max= 8126, per=21.40%, avg=8126.00, stdev= 0.00, samples=1 iops : min= 7, max= 7, avg= 7.00, stdev= 0.00, samples=1 lat (msec) : 50=50.00%, 100=41.67%, 250=8.33% cpu : usr=0.15%, sys=14.69%, ctx=168, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=5,7,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=26.5MiB/s (27.8MB/s), 3061KiB/s-12.5MiB/s (3135kB/s-13.1MB/s), io=20.0MiB (20.0MB), run=559-755msec WRITE: bw=37.1MiB/s (38.9MB/s), 8138KiB/s-14.9MiB/s (8333kB/s-15.7MB/s), io=28.0MiB (29.4MB), run=559-755msec mix direct rw 4194304 by fio with 4 jobs... mix buffer rw 4194304 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5455: Sun Feb 2 04:08:26 2025 read: IOPS=8, BW=33.7MiB/s (35.3MB/s)(12.0MiB/356msec) clat (msec): min=71, max=142, avg=106.89, stdev=35.09 lat (msec): min=71, max=142, avg=106.89, stdev=35.09 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 72], 10.00th=[ 72], 20.00th=[ 72], | 30.00th=[ 72], 40.00th=[ 107], 50.00th=[ 107], 60.00th=[ 107], | 70.00th=[ 142], 80.00th=[ 142], 90.00th=[ 142], 95.00th=[ 142], | 99.00th=[ 142], 99.50th=[ 142], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=32.11%, ctx=1426, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5457: Sun Feb 2 04:08:26 2025 read: IOPS=1, BW=7543KiB/s (7724kB/s)(4096KiB/543msec) clat (nsec): min=118754k, max=118754k, avg=118754108.00, stdev= 0.00 lat (nsec): min=118755k, max=118755k, avg=118755190.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 120], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 8159, max= 8159, per=27.04%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 write: IOPS=3, BW=14.7MiB/s (15.4MB/s)(8192KiB/543msec) clat (msec): min=129, max=287, avg=208.55, stdev=111.63 lat (msec): min=129, max=287, avg=208.85, stdev=111.64 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 130], 10.00th=[ 130], 20.00th=[ 130], | 30.00th=[ 130], 40.00th=[ 130], 50.00th=[ 130], 60.00th=[ 288], | 70.00th=[ 288], 80.00th=[ 288], 90.00th=[ 288], 95.00th=[ 288], | 99.00th=[ 288], 99.50th=[ 288], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 8159, max= 8159, per=13.89%, avg=8159.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=34.50%, ctx=846, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5458: Sun Feb 2 04:08:26 2025 write: IOPS=5, BW=21.5MiB/s (22.6MB/s)(12.0MiB/558msec) clat (msec): min=71, max=246, avg=175.48, stdev=92.33 lat (msec): min=71, max=247, avg=175.73, stdev=92.34 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 71], 10.00th=[ 71], 20.00th=[ 71], | 30.00th=[ 71], 40.00th=[ 209], 50.00th=[ 209], 60.00th=[ 209], | 70.00th=[ 247], 80.00th=[ 247], 90.00th=[ 247], 95.00th=[ 247], | 99.00th=[ 247], 99.50th=[ 247], 99.90th=[ 247], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min=16384, max=16384, per=27.90%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=66.67% cpu : usr=0.00%, sys=34.11%, ctx=60, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5460: Sun Feb 2 04:08:26 2025 write: IOPS=5, BW=21.8MiB/s (22.8MB/s)(12.0MiB/551msec) clat (msec): min=106, max=249, avg=167.11, stdev=73.65 lat (msec): min=106, max=249, avg=167.37, stdev=73.64 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 146], 50.00th=[ 146], 60.00th=[ 146], | 70.00th=[ 249], 80.00th=[ 249], 90.00th=[ 249], 95.00th=[ 249], | 99.00th=[ 249], 99.50th=[ 249], 99.90th=[ 249], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min=16384, max=16384, per=27.90%, avg=16384.00, stdev= 0.00, samples=1 iops : min= 4, max= 4, avg= 4.00, stdev= 0.00, samples=1 lat (msec) : 250=100.00% cpu : usr=0.00%, sys=43.45%, ctx=86, majf=0, minf=28 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=29.5MiB/s (30.9MB/s), 7543KiB/s-33.7MiB/s (7724kB/s-35.3MB/s), io=16.0MiB (16.8MB), run=356-543msec WRITE: bw=57.3MiB/s (60.1MB/s), 14.7MiB/s-21.8MiB/s (15.4MB/s-22.8MB/s), io=32.0MiB (33.6MB), run=543-558msec rand-rw: (groupid=0, jobs=1): err= 0: pid=5454: Sun Feb 2 04:08:27 2025 read: IOPS=10, BW=40.0MiB/s (41.9MB/s)(12.0MiB/300msec) clat (msec): min=52, max=132, avg=83.32, stdev=42.88 lat (msec): min=52, max=132, avg=83.32, stdev=42.88 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 53], 10.00th=[ 53], 20.00th=[ 53], | 30.00th=[ 53], 40.00th=[ 66], 50.00th=[ 66], 60.00th=[ 66], | 70.00th=[ 133], 80.00th=[ 133], 90.00th=[ 133], 95.00th=[ 133], | 99.00th=[ 133], 99.50th=[ 133], 99.90th=[ 133], 99.95th=[ 133], | 99.99th=[ 133] lat (msec) : 100=66.67%, 250=33.33% cpu : usr=0.00%, sys=23.08%, ctx=40, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=3,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5456: Sun Feb 2 04:08:27 2025 read: IOPS=1, BW=5737KiB/s (5874kB/s)(4096KiB/714msec) clat (nsec): min=349745k, max=349745k, avg=349744707.00, stdev= 0.00 lat (nsec): min=349746k, max=349746k, avg=349746226.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 351], 10.00th=[ 351], 20.00th=[ 351], | 30.00th=[ 351], 40.00th=[ 351], 50.00th=[ 351], 60.00th=[ 351], | 70.00th=[ 351], 80.00th=[ 351], 90.00th=[ 351], 95.00th=[ 351], | 99.00th=[ 351], 99.50th=[ 351], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 8192, max= 8192, per=35.70%, avg=8192.00, stdev= 0.00, samples=1 iops : min= 2, max= 2, avg= 2.00, stdev= 0.00, samples=1 write: IOPS=2, BW=11.2MiB/s (11.7MB/s)(8192KiB/714msec) clat (msec): min=124, max=200, avg=162.78, stdev=53.58 lat (msec): min=125, max=200, avg=162.99, stdev=53.48 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 126], 10.00th=[ 126], 20.00th=[ 126], | 30.00th=[ 126], 40.00th=[ 126], 50.00th=[ 126], 60.00th=[ 201], | 70.00th=[ 201], 80.00th=[ 201], 90.00th=[ 201], 95.00th=[ 201], | 99.00th=[ 201], 99.50th=[ 201], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.00%, sys=12.90%, ctx=42, majf=0, minf=32 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,2,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5459: Sun Feb 2 04:08:27 2025 write: IOPS=4, BW=18.6MiB/s (19.5MB/s)(12.0MiB/644msec) clat (msec): min=53, max=323, avg=202.13, stdev=137.04 lat (msec): min=53, max=323, avg=202.39, stdev=136.87 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 54], 10.00th=[ 54], 20.00th=[ 54], | 30.00th=[ 54], 40.00th=[ 230], 50.00th=[ 230], 60.00th=[ 230], | 70.00th=[ 326], 80.00th=[ 326], 90.00th=[ 326], 95.00th=[ 326], | 99.00th=[ 326], 99.50th=[ 326], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 8175, max= 8175, per=18.96%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 100=33.33%, 250=33.33%, 500=33.33% cpu : usr=0.16%, sys=16.64%, ctx=99, majf=0, minf=31 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5461: Sun Feb 2 04:08:27 2025 write: IOPS=3, BW=15.8MiB/s (16.6MB/s)(12.0MiB/760msec) clat (msec): min=117, max=400, avg=243.75, stdev=144.14 lat (msec): min=117, max=400, avg=244.02, stdev=144.13 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 117], 10.00th=[ 117], 20.00th=[ 117], | 30.00th=[ 117], 40.00th=[ 213], 50.00th=[ 213], 60.00th=[ 213], | 70.00th=[ 401], 80.00th=[ 401], 90.00th=[ 401], 95.00th=[ 401], | 99.00th=[ 401], 99.50th=[ 401], 99.90th=[ 401], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 8175, max= 8175, per=18.96%, avg=8175.00, stdev= 0.00, samples=1 iops : min= 1, max= 1, avg= 1.00, stdev= 0.00, samples=1 lat (msec) : 250=66.67%, 500=33.33% cpu : usr=0.26%, sys=12.25%, ctx=99, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=22.4MiB/s (23.5MB/s), 5737KiB/s-40.0MiB/s (5874kB/s-41.9MB/s), io=16.0MiB (16.8MB), run=300-714msec WRITE: bw=42.1MiB/s (44.1MB/s), 11.2MiB/s-18.6MiB/s (11.7MB/s-19.5MB/s), io=32.0MiB (33.6MB), run=644-760msec