== sanity test 398c: run fio to test AIO ================= 18:15:36 (1788819336) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.454546 s, 92.3 MB/s osc.lustre-OST0000-osc-ffff9559ca103800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179912: Mon Sep 7 18:16:12 2026 write: IOPS=95, BW=383KiB/s (393kB/s)(10.0MiB/26715msec); 0 zone resets slat (usec): min=22, max=20221, avg=211.61, stdev=843.81 clat (msec): min=14, max=316, avg=165.68, stdev=44.59 lat (msec): min=14, max=316, avg=165.89, stdev=44.61 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 111], 10.00th=[ 121], 20.00th=[ 130], | 30.00th=[ 142], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 169], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 232], 95.00th=[ 253], | 99.00th=[ 288], 99.50th=[ 305], 99.90th=[ 313], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 255, max= 512, per=24.89%, avg=381.57, stdev=71.46, samples=53 iops : min= 63, max= 128, avg=95.04, stdev=17.88, samples=53 lat (msec) : 20=0.04%, 50=0.20%, 100=2.54%, 250=92.03%, 500=5.20% cpu : usr=0.22%, sys=0.97%, ctx=2347, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179913: Mon Sep 7 18:16:12 2026 write: IOPS=95, BW=384KiB/s (393kB/s)(10.0MiB/26678msec); 0 zone resets slat (usec): min=21, max=17916, avg=237.37, stdev=855.24 clat (msec): min=17, max=321, avg=166.30, stdev=45.97 lat (msec): min=18, max=321, avg=166.54, stdev=45.93 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 108], 10.00th=[ 121], 20.00th=[ 133], | 30.00th=[ 142], 40.00th=[ 150], 50.00th=[ 159], 60.00th=[ 169], | 70.00th=[ 184], 80.00th=[ 199], 90.00th=[ 230], 95.00th=[ 259], | 99.00th=[ 296], 99.50th=[ 305], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 321] bw ( KiB/s): min= 255, max= 576, per=24.97%, avg=382.73, stdev=74.32, samples=52 iops : min= 63, max= 144, avg=95.33, stdev=18.57, samples=52 lat (msec) : 20=0.12%, 50=0.55%, 100=3.09%, 250=89.80%, 500=6.45% cpu : usr=0.20%, sys=1.18%, ctx=2862, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179914: Mon Sep 7 18:16:12 2026 write: IOPS=95, BW=383KiB/s (393kB/s)(10.0MiB/26703msec); 0 zone resets slat (usec): min=22, max=16341, avg=203.47, stdev=835.50 clat (msec): min=31, max=337, avg=165.74, stdev=43.93 lat (msec): min=31, max=337, avg=165.94, stdev=43.95 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 110], 10.00th=[ 120], 20.00th=[ 133], | 30.00th=[ 142], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 169], | 70.00th=[ 184], 80.00th=[ 194], 90.00th=[ 228], 95.00th=[ 259], | 99.00th=[ 284], 99.50th=[ 300], 99.90th=[ 334], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 240, max= 518, per=24.87%, avg=381.28, stdev=75.89, samples=53 iops : min= 60, max= 129, avg=94.98, stdev=18.99, samples=53 lat (msec) : 50=0.08%, 100=2.81%, 250=91.21%, 500=5.90% cpu : usr=0.16%, sys=0.87%, ctx=2498, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179915: Mon Sep 7 18:16:12 2026 write: IOPS=96, BW=384KiB/s (394kB/s)(10.0MiB/26634msec); 0 zone resets slat (usec): min=20, max=22856, avg=253.24, stdev=1100.61 clat (msec): min=28, max=337, avg=165.61, stdev=46.18 lat (msec): min=28, max=337, avg=165.86, stdev=46.18 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 105], 10.00th=[ 118], 20.00th=[ 132], | 30.00th=[ 142], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 171], | 70.00th=[ 182], 80.00th=[ 197], 90.00th=[ 230], 95.00th=[ 257], | 99.00th=[ 300], 99.50th=[ 326], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 338] bw ( KiB/s): min= 239, max= 565, per=25.03%, avg=383.75, stdev=79.82, samples=52 iops : min= 59, max= 141, avg=95.58, stdev=19.94, samples=52 lat (msec) : 50=0.74%, 100=3.44%, 250=89.88%, 500=5.94% cpu : usr=0.26%, sys=1.21%, ctx=2789, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1533KiB/s (1570kB/s), 383KiB/s-384KiB/s (393kB/s-394kB/s), io=40.0MiB (41.9MB), run=26634-26715msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179928: Mon Sep 7 18:16:56 2026 read: IOPS=32, BW=131KiB/s (135kB/s)(5048KiB/38425msec) slat (usec): min=36, max=187318, avg=17481.12, stdev=11728.19 clat (msec): min=38, max=492, avg=228.12, stdev=70.71 lat (msec): min=49, max=527, avg=245.61, stdev=73.76 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 118], 10.00th=[ 144], 20.00th=[ 174], | 30.00th=[ 190], 40.00th=[ 207], 50.00th=[ 222], 60.00th=[ 239], | 70.00th=[ 257], 80.00th=[ 279], 90.00th=[ 321], 95.00th=[ 363], | 99.00th=[ 435], 99.50th=[ 451], 99.90th=[ 489], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 70, max= 320, per=25.21%, avg=131.35, stdev=40.54, samples=75 iops : min= 17, max= 80, avg=32.33, stdev=10.18, samples=75 write: IOPS=33, BW=135KiB/s (138kB/s)(5192KiB/38425msec); 0 zone resets slat (usec): min=29, max=20233, avg=338.72, stdev=1220.79 clat (msec): min=49, max=481, avg=232.79, stdev=70.52 lat (msec): min=49, max=481, avg=233.13, stdev=70.52 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 122], 10.00th=[ 150], 20.00th=[ 180], | 30.00th=[ 197], 40.00th=[ 211], 50.00th=[ 226], 60.00th=[ 243], | 70.00th=[ 262], 80.00th=[ 288], 90.00th=[ 330], 95.00th=[ 363], | 99.00th=[ 435], 99.50th=[ 456], 99.90th=[ 472], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 39, max= 287, per=24.97%, avg=135.36, stdev=47.18, samples=75 iops : min= 9, max= 71, avg=33.36, stdev=11.88, samples=75 lat (msec) : 50=0.12%, 100=1.68%, 250=63.63%, 500=34.57% cpu : usr=0.28%, sys=2.03%, ctx=4256, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179929: Mon Sep 7 18:16:56 2026 read: IOPS=32, BW=129KiB/s (132kB/s)(4976KiB/38494msec) slat (usec): min=38, max=83227, avg=17583.75, stdev=11184.93 clat (msec): min=68, max=518, avg=230.17, stdev=74.09 lat (msec): min=74, max=542, avg=247.77, stdev=77.68 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 115], 10.00th=[ 142], 20.00th=[ 171], | 30.00th=[ 190], 40.00th=[ 205], 50.00th=[ 222], 60.00th=[ 239], | 70.00th=[ 259], 80.00th=[ 288], 90.00th=[ 334], 95.00th=[ 372], | 99.00th=[ 435], 99.50th=[ 460], 99.90th=[ 498], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 40, max= 279, per=24.68%, avg=128.56, stdev=39.96, samples=75 iops : min= 10, max= 69, avg=31.64, stdev=10.02, samples=75 write: IOPS=34, BW=137KiB/s (140kB/s)(5264KiB/38494msec); 0 zone resets slat (usec): min=23, max=21759, avg=293.78, stdev=1186.75 clat (msec): min=67, max=474, avg=232.02, stdev=70.77 lat (msec): min=72, max=475, avg=232.32, stdev=70.76 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 121], 10.00th=[ 146], 20.00th=[ 176], | 30.00th=[ 194], 40.00th=[ 211], 50.00th=[ 226], 60.00th=[ 241], | 70.00th=[ 262], 80.00th=[ 288], 90.00th=[ 330], 95.00th=[ 372], | 99.00th=[ 418], 99.50th=[ 430], 99.90th=[ 468], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 63, max= 304, per=25.28%, avg=137.04, stdev=47.35, samples=75 iops : min= 15, max= 76, avg=33.77, stdev=11.92, samples=75 lat (msec) : 100=1.91%, 250=63.20%, 500=34.84%, 750=0.04% cpu : usr=0.31%, sys=1.75%, ctx=4088, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179930: Mon Sep 7 18:16:56 2026 read: IOPS=32, BW=130KiB/s (133kB/s)(4984KiB/38425msec) slat (usec): min=29, max=95370, avg=17294.18, stdev=11326.14 clat (msec): min=54, max=517, avg=229.81, stdev=70.39 lat (msec): min=67, max=526, avg=247.11, stdev=73.60 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 117], 10.00th=[ 144], 20.00th=[ 174], | 30.00th=[ 192], 40.00th=[ 209], 50.00th=[ 226], 60.00th=[ 243], | 70.00th=[ 262], 80.00th=[ 288], 90.00th=[ 321], 95.00th=[ 355], | 99.00th=[ 418], 99.50th=[ 430], 99.90th=[ 510], 99.95th=[ 518], | 99.99th=[ 518] bw ( KiB/s): min= 31, max= 255, per=24.90%, avg=129.73, stdev=41.45, samples=75 iops : min= 7, max= 63, avg=31.93, stdev=10.44, samples=75 write: IOPS=34, BW=137KiB/s (140kB/s)(5256KiB/38425msec); 0 zone resets slat (usec): min=25, max=26072, avg=255.35, stdev=888.12 clat (msec): min=71, max=494, avg=232.68, stdev=72.70 lat (msec): min=71, max=494, avg=232.94, stdev=72.70 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 115], 10.00th=[ 140], 20.00th=[ 176], | 30.00th=[ 190], 40.00th=[ 213], 50.00th=[ 230], 60.00th=[ 249], | 70.00th=[ 266], 80.00th=[ 292], 90.00th=[ 330], 95.00th=[ 363], | 99.00th=[ 414], 99.50th=[ 439], 99.90th=[ 477], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 56, max= 368, per=25.16%, avg=136.37, stdev=47.09, samples=75 iops : min= 14, max= 92, avg=33.61, stdev=11.84, samples=75 lat (msec) : 100=2.46%, 250=60.23%, 500=37.19%, 750=0.12% cpu : usr=0.32%, sys=1.64%, ctx=3963, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179931: Mon Sep 7 18:16:56 2026 read: IOPS=32, BW=131KiB/s (135kB/s)(5060KiB/38504msec) slat (usec): min=2483, max=91001, avg=17841.89, stdev=11178.17 clat (msec): min=21, max=462, avg=225.84, stdev=71.66 lat (msec): min=41, max=508, avg=243.69, stdev=74.85 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 113], 10.00th=[ 138], 20.00th=[ 169], | 30.00th=[ 188], 40.00th=[ 205], 50.00th=[ 222], 60.00th=[ 241], | 70.00th=[ 257], 80.00th=[ 279], 90.00th=[ 321], 95.00th=[ 355], | 99.00th=[ 430], 99.50th=[ 443], 99.90th=[ 460], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 55, max= 288, per=25.16%, avg=131.09, stdev=44.17, samples=75 iops : min= 13, max= 72, avg=32.28, stdev=11.12, samples=75 write: IOPS=33, BW=135KiB/s (138kB/s)(5180KiB/38504msec); 0 zone resets slat (usec): min=27, max=13214, avg=326.05, stdev=1022.61 clat (msec): min=41, max=486, avg=234.54, stdev=68.70 lat (msec): min=41, max=486, avg=234.87, stdev=68.69 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 116], 10.00th=[ 148], 20.00th=[ 180], | 30.00th=[ 201], 40.00th=[ 218], 50.00th=[ 230], 60.00th=[ 249], | 70.00th=[ 266], 80.00th=[ 288], 90.00th=[ 326], 95.00th=[ 355], | 99.00th=[ 409], 99.50th=[ 426], 99.90th=[ 472], 99.95th=[ 485], | 99.99th=[ 485] bw ( KiB/s): min= 47, max= 312, per=24.72%, avg=133.96, stdev=43.64, samples=75 iops : min= 11, max= 78, avg=32.99, stdev=10.98, samples=75 lat (msec) : 50=0.23%, 100=1.91%, 250=61.60%, 500=36.25% cpu : usr=0.28%, sys=1.86%, ctx=4211, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=521KiB/s (534kB/s), 129KiB/s-131KiB/s (132kB/s-135kB/s), io=19.6MiB (20.5MB), run=38425-38504msec WRITE: bw=543KiB/s (556kB/s), 135KiB/s-137KiB/s (138kB/s-140kB/s), io=20.4MiB (21.4MB), run=38425-38504msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179941: Mon Sep 7 18:17:06 2026 read: IOPS=2, BW=89.3MiB/s (93.6MB/s)(40.0MiB/448msec) slat (nsec): min=24490k, max=24490k, avg=24489703.00, stdev= 0.00 clat (nsec): min=406094k, max=406094k, avg=406094247.00, stdev= 0.00 lat (nsec): min=430589k, max=430589k, avg=430589117.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.24%, ctx=11, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=89.3MiB/s (93.6MB/s), 89.3MiB/s-89.3MiB/s (93.6MB/s-93.6MB/s), io=40.0MiB (41.9MB), run=448-448msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout