== sanity test 398c: run fio to test AIO ================= 14:58:00 (1780340280) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.809169 s, 51.8 MB/s osc.lustre-OST0000-osc-ffff92f108450000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180675: Mon Jun 1 14:59:00 2026 write: IOPS=53, BW=215KiB/s (220kB/s)(10.0MiB/47610msec); 0 zone resets slat (usec): min=38, max=30097, avg=437.82, stdev=1460.14 clat (msec): min=18, max=3070, avg=295.83, stdev=238.57 lat (msec): min=20, max=3070, avg=296.27, stdev=238.57 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 148], 10.00th=[ 167], 20.00th=[ 194], | 30.00th=[ 222], 40.00th=[ 247], 50.00th=[ 264], 60.00th=[ 279], | 70.00th=[ 296], 80.00th=[ 338], 90.00th=[ 409], 95.00th=[ 502], | 99.00th=[ 1267], 99.50th=[ 2601], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 7, max= 446, per=26.36%, avg=226.67, stdev=79.13, samples=89 iops : min= 1, max= 111, avg=56.21, stdev=19.88, samples=89 lat (msec) : 20=0.04%, 50=0.12%, 100=0.66%, 250=40.94%, 500=53.24% lat (msec) : 750=3.36%, 1000=0.31%, 2000=0.70%, >=2000=0.62% cpu : usr=0.15%, sys=0.85%, ctx=2394, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180676: Mon Jun 1 14:59:00 2026 write: IOPS=53, BW=215KiB/s (221kB/s)(10.0MiB/47550msec); 0 zone resets slat (usec): min=48, max=20915, avg=395.89, stdev=1285.52 clat (msec): min=43, max=3096, avg=296.14, stdev=231.89 lat (msec): min=43, max=3097, avg=296.54, stdev=231.88 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 142], 10.00th=[ 165], 20.00th=[ 194], | 30.00th=[ 220], 40.00th=[ 243], 50.00th=[ 262], 60.00th=[ 279], | 70.00th=[ 300], 80.00th=[ 338], 90.00th=[ 422], 95.00th=[ 493], | 99.00th=[ 1368], 99.50th=[ 2366], 99.90th=[ 2970], 99.95th=[ 2970], | 99.99th=[ 3104] bw ( KiB/s): min= 24, max= 471, per=26.36%, avg=226.71, stdev=84.86, samples=89 iops : min= 6, max= 117, avg=56.22, stdev=21.27, samples=89 lat (msec) : 50=0.04%, 250=43.55%, 500=52.03%, 750=2.70%, 1000=0.12% lat (msec) : 2000=0.94%, >=2000=0.62% cpu : usr=0.12%, sys=0.79%, ctx=2030, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180677: Mon Jun 1 14:59:00 2026 write: IOPS=53, BW=215KiB/s (220kB/s)(10.0MiB/47591msec); 0 zone resets slat (usec): min=38, max=24910, avg=436.61, stdev=1333.56 clat (msec): min=54, max=2652, avg=296.82, stdev=230.82 lat (msec): min=55, max=2652, avg=297.26, stdev=230.78 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 144], 10.00th=[ 163], 20.00th=[ 192], | 30.00th=[ 220], 40.00th=[ 243], 50.00th=[ 262], 60.00th=[ 279], | 70.00th=[ 300], 80.00th=[ 338], 90.00th=[ 414], 95.00th=[ 502], | 99.00th=[ 1301], 99.50th=[ 2567], 99.90th=[ 2567], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 486, per=26.66%, avg=229.24, stdev=84.91, samples=88 iops : min= 1, max= 121, avg=56.83, stdev=21.30, samples=88 lat (msec) : 100=0.59%, 250=41.88%, 500=52.11%, 750=3.44%, 1000=0.12% lat (msec) : 2000=1.25%, >=2000=0.62% cpu : usr=0.13%, sys=0.86%, ctx=2113, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180678: Mon Jun 1 14:59:00 2026 write: IOPS=53, BW=215KiB/s (220kB/s)(10.0MiB/47607msec); 0 zone resets slat (usec): min=48, max=33365, avg=493.30, stdev=1721.68 clat (msec): min=21, max=2816, avg=295.80, stdev=236.98 lat (msec): min=21, max=2816, avg=296.30, stdev=236.94 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 148], 10.00th=[ 165], 20.00th=[ 197], | 30.00th=[ 220], 40.00th=[ 245], 50.00th=[ 266], 60.00th=[ 279], | 70.00th=[ 300], 80.00th=[ 330], 90.00th=[ 405], 95.00th=[ 481], | 99.00th=[ 1452], 99.50th=[ 2601], 99.90th=[ 2802], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 31, max= 471, per=26.36%, avg=226.66, stdev=80.56, samples=89 iops : min= 7, max= 117, avg=56.24, stdev=20.20, samples=89 lat (msec) : 50=0.08%, 100=0.78%, 250=41.37%, 500=53.71%, 750=2.46% lat (msec) : 1000=0.12%, 2000=0.86%, >=2000=0.62% cpu : usr=0.14%, sys=0.95%, ctx=2469, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=860KiB/s (881kB/s), 215KiB/s-215KiB/s (220kB/s-221kB/s), io=40.0MiB (41.9MB), run=47550-47610msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180696: Mon Jun 1 14:59:51 2026 read: IOPS=32, BW=128KiB/s (131kB/s)(5048KiB/39357msec) slat (usec): min=60, max=59929, avg=16645.15, stdev=8405.23 clat (msec): min=24, max=436, avg=232.37, stdev=49.77 lat (msec): min=45, max=445, avg=249.03, stdev=50.95 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 192], | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 230], 60.00th=[ 243], | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 321], | 99.00th=[ 355], 99.50th=[ 376], 99.90th=[ 426], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 71, max= 198, per=25.03%, avg=127.42, stdev=30.92, samples=78 iops : min= 17, max= 49, avg=31.32, stdev= 7.73, samples=78 write: IOPS=32, BW=132KiB/s (135kB/s)(5192KiB/39357msec); 0 zone resets slat (usec): min=57, max=26365, avg=505.28, stdev=1567.18 clat (msec): min=129, max=476, avg=240.83, stdev=45.91 lat (msec): min=129, max=476, avg=241.34, stdev=45.95 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 169], 10.00th=[ 184], 20.00th=[ 203], | 30.00th=[ 215], 40.00th=[ 228], 50.00th=[ 241], 60.00th=[ 249], | 70.00th=[ 259], 80.00th=[ 275], 90.00th=[ 300], 95.00th=[ 321], | 99.00th=[ 363], 99.50th=[ 376], 99.90th=[ 439], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 53, max= 215, per=24.74%, avg=131.14, stdev=31.69, samples=78 iops : min= 13, max= 53, avg=32.26, stdev= 7.90, samples=78 lat (msec) : 50=0.08%, 100=0.04%, 250=63.05%, 500=36.84% cpu : usr=0.28%, sys=1.92%, ctx=4603, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180697: Mon Jun 1 14:59:51 2026 read: IOPS=31, BW=127KiB/s (130kB/s)(4976KiB/39098msec) slat (usec): min=58, max=60859, avg=17085.40, stdev=8610.98 clat (msec): min=111, max=508, avg=232.26, stdev=46.66 lat (msec): min=111, max=530, avg=249.35, stdev=48.19 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 165], 10.00th=[ 178], 20.00th=[ 194], | 30.00th=[ 207], 40.00th=[ 218], 50.00th=[ 228], 60.00th=[ 243], | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 292], 95.00th=[ 309], | 99.00th=[ 351], 99.50th=[ 393], 99.90th=[ 477], 99.95th=[ 510], | 99.99th=[ 510] bw ( KiB/s): min= 71, max= 200, per=24.92%, avg=126.83, stdev=27.09, samples=77 iops : min= 17, max= 50, avg=31.17, stdev= 6.74, samples=77 write: IOPS=33, BW=135KiB/s (138kB/s)(5264KiB/39098msec); 0 zone resets slat (usec): min=55, max=12985, avg=468.89, stdev=1124.56 clat (msec): min=75, max=443, avg=238.62, stdev=45.97 lat (msec): min=75, max=443, avg=239.09, stdev=45.92 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 171], 10.00th=[ 182], 20.00th=[ 203], | 30.00th=[ 215], 40.00th=[ 226], 50.00th=[ 239], 60.00th=[ 249], | 70.00th=[ 264], 80.00th=[ 279], 90.00th=[ 300], 95.00th=[ 317], | 99.00th=[ 342], 99.50th=[ 351], 99.90th=[ 388], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 63, max= 215, per=25.41%, avg=134.65, stdev=32.36, samples=77 iops : min= 15, max= 53, avg=33.12, stdev= 8.07, samples=77 lat (msec) : 100=0.51%, 250=63.83%, 500=35.62%, 750=0.04% cpu : usr=0.30%, sys=2.07%, ctx=4368, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180698: Mon Jun 1 14:59:51 2026 read: IOPS=31, BW=127KiB/s (130kB/s)(4984KiB/39355msec) slat (usec): min=4029, max=59801, avg=17108.27, stdev=8472.10 clat (msec): min=42, max=407, avg=230.40, stdev=45.28 lat (msec): min=64, max=420, avg=247.51, stdev=46.21 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 163], 10.00th=[ 174], 20.00th=[ 192], | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 230], 60.00th=[ 243], | 70.00th=[ 253], 80.00th=[ 271], 90.00th=[ 288], 95.00th=[ 305], | 99.00th=[ 347], 99.50th=[ 359], 99.90th=[ 388], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 54, max= 199, per=24.79%, avg=126.19, stdev=32.81, samples=78 iops : min= 13, max= 49, avg=31.01, stdev= 8.22, samples=78 write: IOPS=33, BW=134KiB/s (137kB/s)(5256KiB/39355msec); 0 zone resets slat (usec): min=54, max=14715, avg=441.73, stdev=1070.37 clat (msec): min=121, max=395, avg=241.35, stdev=43.99 lat (msec): min=121, max=395, avg=241.79, stdev=44.02 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 174], 10.00th=[ 186], 20.00th=[ 203], | 30.00th=[ 218], 40.00th=[ 230], 50.00th=[ 239], 60.00th=[ 251], | 70.00th=[ 264], 80.00th=[ 279], 90.00th=[ 300], 95.00th=[ 317], | 99.00th=[ 351], 99.50th=[ 372], 99.90th=[ 397], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 61, max= 199, per=24.98%, avg=132.37, stdev=29.63, samples=78 iops : min= 15, max= 49, avg=32.55, stdev= 7.41, samples=78 lat (msec) : 50=0.04%, 250=63.24%, 500=36.72% cpu : usr=0.25%, sys=2.08%, ctx=4471, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180699: Mon Jun 1 14:59:51 2026 read: IOPS=32, BW=129KiB/s (132kB/s)(5060KiB/39363msec) slat (usec): min=3561, max=55663, avg=17139.04, stdev=7982.03 clat (msec): min=65, max=423, avg=230.72, stdev=45.96 lat (msec): min=84, max=432, avg=247.87, stdev=47.17 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 161], 10.00th=[ 176], 20.00th=[ 192], | 30.00th=[ 205], 40.00th=[ 215], 50.00th=[ 228], 60.00th=[ 241], | 70.00th=[ 253], 80.00th=[ 271], 90.00th=[ 292], 95.00th=[ 313], | 99.00th=[ 351], 99.50th=[ 359], 99.90th=[ 376], 99.95th=[ 422], | 99.99th=[ 422] bw ( KiB/s): min= 63, max= 192, per=25.08%, avg=127.68, stdev=32.12, samples=78 iops : min= 15, max= 48, avg=31.40, stdev= 8.06, samples=78 write: IOPS=32, BW=132KiB/s (135kB/s)(5180KiB/39363msec); 0 zone resets slat (usec): min=55, max=11147, avg=447.35, stdev=977.10 clat (msec): min=52, max=446, avg=241.32, stdev=44.70 lat (msec): min=53, max=447, avg=241.77, stdev=44.67 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 174], 10.00th=[ 184], 20.00th=[ 205], | 30.00th=[ 218], 40.00th=[ 228], 50.00th=[ 239], 60.00th=[ 253], | 70.00th=[ 264], 80.00th=[ 279], 90.00th=[ 300], 95.00th=[ 317], | 99.00th=[ 342], 99.50th=[ 376], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 64, max= 184, per=24.69%, avg=130.87, stdev=27.97, samples=78 iops : min= 16, max= 46, avg=32.18, stdev= 6.94, samples=78 lat (msec) : 100=0.08%, 250=63.44%, 500=36.48% cpu : usr=0.30%, sys=2.14%, ctx=4732, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=510KiB/s (522kB/s), 127KiB/s-129KiB/s (130kB/s-132kB/s), io=19.6MiB (20.5MB), run=39098-39363msec WRITE: bw=531KiB/s (543kB/s), 132KiB/s-135KiB/s (135kB/s-138kB/s), io=20.4MiB (21.4MB), run=39098-39363msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180709: Mon Jun 1 15:00:03 2026 read: IOPS=1, BW=56.7MiB/s (59.5MB/s)(40.0MiB/705msec) slat (nsec): min=55320k, max=55320k, avg=55320344.00, stdev= 0.00 clat (nsec): min=627818k, max=627818k, avg=627817891.00, stdev= 0.00 lat (nsec): min=690827k, max=690827k, avg=690826785.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 625], 5.00th=[ 625], 10.00th=[ 625], 20.00th=[ 625], | 30.00th=[ 625], 40.00th=[ 625], 50.00th=[ 625], 60.00th=[ 625], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.13%, ctx=12, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=56.7MiB/s (59.5MB/s), 56.7MiB/s-56.7MiB/s (59.5MB/s-59.5MB/s), io=40.0MiB (41.9MB), run=705-705msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout