== sanity test 398c: run fio to test AIO ================= 12:34:32 (1788798872) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.532545 s, 78.8 MB/s osc.lustre-OST0000-osc-ffff92450638d000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179389: Mon Sep 7 12:35:11 2026 write: IOPS=92, BW=369KiB/s (377kB/s)(10.0MiB/27784msec); 0 zone resets slat (usec): min=24, max=16640, avg=202.86, stdev=697.03 clat (msec): min=26, max=341, avg=173.26, stdev=48.46 lat (msec): min=26, max=341, avg=173.46, stdev=48.49 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 114], 10.00th=[ 124], 20.00th=[ 134], | 30.00th=[ 142], 40.00th=[ 150], 50.00th=[ 161], 60.00th=[ 178], | 70.00th=[ 194], 80.00th=[ 211], 90.00th=[ 245], 95.00th=[ 271], | 99.00th=[ 309], 99.50th=[ 313], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 342] bw ( KiB/s): min= 253, max= 506, per=25.03%, avg=366.91, stdev=76.33, samples=55 iops : min= 63, max= 126, avg=91.33, stdev=19.06, samples=55 lat (msec) : 50=0.23%, 100=0.98%, 250=90.20%, 500=8.59% cpu : usr=0.18%, sys=0.96%, ctx=2909, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179390: Mon Sep 7 12:35:11 2026 write: IOPS=91, BW=367KiB/s (375kB/s)(10.0MiB/27938msec); 0 zone resets slat (usec): min=23, max=15392, avg=196.01, stdev=867.74 clat (msec): min=24, max=335, avg=173.46, stdev=46.95 lat (msec): min=30, max=335, avg=173.66, stdev=46.94 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 116], 10.00th=[ 126], 20.00th=[ 134], | 30.00th=[ 144], 40.00th=[ 150], 50.00th=[ 163], 60.00th=[ 178], | 70.00th=[ 194], 80.00th=[ 213], 90.00th=[ 241], 95.00th=[ 264], | 99.00th=[ 309], 99.50th=[ 313], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 232, max= 512, per=24.90%, avg=365.02, stdev=82.09, samples=55 iops : min= 58, max= 128, avg=90.87, stdev=20.49, samples=55 lat (msec) : 50=0.16%, 100=0.90%, 250=91.64%, 500=7.30% cpu : usr=0.18%, sys=0.97%, ctx=2361, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179391: Mon Sep 7 12:35:11 2026 write: IOPS=91, BW=367KiB/s (376kB/s)(10.0MiB/27876msec); 0 zone resets slat (usec): min=21, max=11702, avg=131.64, stdev=501.98 clat (msec): min=60, max=336, avg=173.33, stdev=47.61 lat (msec): min=66, max=336, avg=173.46, stdev=47.62 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 116], 10.00th=[ 123], 20.00th=[ 134], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 178], | 70.00th=[ 192], 80.00th=[ 211], 90.00th=[ 241], 95.00th=[ 264], | 99.00th=[ 317], 99.50th=[ 330], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 136, max= 514, per=24.96%, avg=365.95, stdev=80.72, samples=55 iops : min= 34, max= 128, avg=91.09, stdev=20.16, samples=55 lat (msec) : 100=1.41%, 250=90.12%, 500=8.48% cpu : usr=0.19%, sys=0.76%, ctx=1540, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179392: Mon Sep 7 12:35:11 2026 write: IOPS=91, BW=367KiB/s (375kB/s)(10.0MiB/27933msec); 0 zone resets slat (usec): min=23, max=17031, avg=154.54, stdev=734.92 clat (msec): min=31, max=323, avg=173.58, stdev=47.54 lat (msec): min=32, max=324, avg=173.73, stdev=47.52 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 117], 10.00th=[ 125], 20.00th=[ 134], | 30.00th=[ 142], 40.00th=[ 150], 50.00th=[ 161], 60.00th=[ 176], | 70.00th=[ 194], 80.00th=[ 211], 90.00th=[ 243], 95.00th=[ 275], | 99.00th=[ 296], 99.50th=[ 305], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 200, max= 514, per=24.92%, avg=365.33, stdev=86.04, samples=55 iops : min= 50, max= 128, avg=90.95, stdev=21.52, samples=55 lat (msec) : 50=0.12%, 100=0.35%, 250=92.03%, 500=7.50% cpu : usr=0.14%, sys=0.77%, ctx=1420, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1466KiB/s (1501kB/s), 367KiB/s-369KiB/s (375kB/s-377kB/s), io=40.0MiB (41.9MB), run=27784-27938msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179408: Mon Sep 7 12:35:56 2026 read: IOPS=39, BW=158KiB/s (162kB/s)(5048KiB/31938msec) slat (usec): min=31, max=75011, avg=14031.54, stdev=8299.46 clat (msec): min=77, max=356, avg=188.07, stdev=46.94 lat (msec): min=85, max=372, avg=202.10, stdev=49.21 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 118], 10.00th=[ 130], 20.00th=[ 148], | 30.00th=[ 163], 40.00th=[ 174], 50.00th=[ 184], 60.00th=[ 197], | 70.00th=[ 211], 80.00th=[ 228], 90.00th=[ 251], 95.00th=[ 271], | 99.00th=[ 321], 99.50th=[ 330], 99.90th=[ 351], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 80, max= 280, per=25.49%, avg=157.51, stdev=37.68, samples=63 iops : min= 20, max= 70, avg=38.90, stdev= 9.51, samples=63 write: IOPS=40, BW=163KiB/s (166kB/s)(5192KiB/31938msec); 0 zone resets slat (usec): min=30, max=14125, avg=227.35, stdev=879.62 clat (msec): min=56, max=384, avg=196.11, stdev=47.40 lat (msec): min=57, max=385, avg=196.33, stdev=47.44 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 125], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 220], 80.00th=[ 236], 90.00th=[ 264], 95.00th=[ 279], | 99.00th=[ 313], 99.50th=[ 330], 99.90th=[ 359], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 80, max= 255, per=25.28%, avg=162.52, stdev=37.57, samples=63 iops : min= 20, max= 63, avg=40.17, stdev= 9.36, samples=63 lat (msec) : 100=1.25%, 250=86.68%, 500=12.07% cpu : usr=0.26%, sys=1.78%, ctx=4132, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179409: Mon Sep 7 12:35:56 2026 read: IOPS=38, BW=153KiB/s (157kB/s)(4976KiB/32441msec) slat (usec): min=24, max=356449, avg=14996.10, stdev=16395.56 clat (msec): min=7, max=1147, avg=192.28, stdev=83.87 lat (msec): min=10, max=1164, avg=207.28, stdev=89.23 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 109], 10.00th=[ 127], 20.00th=[ 146], | 30.00th=[ 161], 40.00th=[ 174], 50.00th=[ 186], 60.00th=[ 199], | 70.00th=[ 211], 80.00th=[ 228], 90.00th=[ 253], 95.00th=[ 275], | 99.00th=[ 380], 99.50th=[ 693], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 15, max= 251, per=24.41%, avg=150.83, stdev=43.95, samples=64 iops : min= 3, max= 62, avg=37.22, stdev=10.98, samples=64 write: IOPS=40, BW=162KiB/s (166kB/s)(5264KiB/32441msec); 0 zone resets slat (usec): min=24, max=23486, avg=178.11, stdev=744.90 clat (msec): min=23, max=1192, avg=196.79, stdev=85.83 lat (msec): min=24, max=1192, avg=196.97, stdev=85.82 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 122], 10.00th=[ 136], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 190], 60.00th=[ 201], | 70.00th=[ 213], 80.00th=[ 230], 90.00th=[ 255], 95.00th=[ 275], | 99.00th=[ 326], 99.50th=[ 1099], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 39, max= 288, per=25.27%, avg=162.49, stdev=49.18, samples=63 iops : min= 9, max= 72, avg=40.16, stdev=12.34, samples=63 lat (msec) : 10=0.04%, 20=0.08%, 50=0.62%, 100=2.07%, 250=86.05% lat (msec) : 500=10.43%, 750=0.16%, 1000=0.08%, 2000=0.47% cpu : usr=0.24%, sys=1.59%, ctx=3965, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179410: Mon Sep 7 12:35:56 2026 read: IOPS=38, BW=155KiB/s (158kB/s)(4984KiB/32215msec) slat (usec): min=2279, max=79546, avg=14304.86, stdev=8398.77 clat (msec): min=24, max=425, avg=188.56, stdev=47.69 lat (msec): min=44, max=436, avg=202.87, stdev=49.41 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 117], 10.00th=[ 131], 20.00th=[ 148], | 30.00th=[ 161], 40.00th=[ 174], 50.00th=[ 186], 60.00th=[ 199], | 70.00th=[ 211], 80.00th=[ 226], 90.00th=[ 251], 95.00th=[ 268], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 422], 99.95th=[ 426], | 99.99th=[ 426] bw ( KiB/s): min= 54, max= 304, per=24.94%, avg=154.11, stdev=41.61, samples=64 iops : min= 13, max= 76, avg=38.05, stdev=10.49, samples=64 write: IOPS=40, BW=163KiB/s (167kB/s)(5256KiB/32215msec); 0 zone resets slat (usec): min=24, max=10059, avg=186.85, stdev=570.55 clat (msec): min=77, max=378, avg=196.83, stdev=46.95 lat (msec): min=77, max=378, avg=197.02, stdev=46.93 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 125], 10.00th=[ 140], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 220], 80.00th=[ 236], 90.00th=[ 262], 95.00th=[ 279], | 99.00th=[ 317], 99.50th=[ 342], 99.90th=[ 372], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 38, max= 255, per=25.13%, avg=161.59, stdev=45.23, samples=64 iops : min= 9, max= 63, avg=39.92, stdev=11.29, samples=64 lat (msec) : 50=0.08%, 100=0.86%, 250=87.03%, 500=12.03% cpu : usr=0.20%, sys=1.61%, ctx=4241, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179411: Mon Sep 7 12:35:56 2026 read: IOPS=39, BW=157KiB/s (161kB/s)(5060KiB/32278msec) slat (usec): min=2414, max=70888, avg=14141.39, stdev=8211.47 clat (msec): min=80, max=351, avg=190.70, stdev=48.38 lat (msec): min=87, max=374, avg=204.85, stdev=50.63 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 120], 10.00th=[ 131], 20.00th=[ 146], | 30.00th=[ 163], 40.00th=[ 178], 50.00th=[ 188], 60.00th=[ 201], | 70.00th=[ 213], 80.00th=[ 228], 90.00th=[ 259], 95.00th=[ 279], | 99.00th=[ 313], 99.50th=[ 330], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 47, max= 244, per=25.24%, avg=155.97, stdev=37.44, samples=64 iops : min= 11, max= 61, avg=38.50, stdev= 9.42, samples=64 write: IOPS=40, BW=160KiB/s (164kB/s)(5180KiB/32278msec); 0 zone resets slat (usec): min=26, max=14287, avg=198.69, stdev=678.18 clat (msec): min=81, max=357, avg=195.35, stdev=46.01 lat (msec): min=81, max=357, avg=195.55, stdev=46.00 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 128], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 169], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 215], 80.00th=[ 232], 90.00th=[ 257], 95.00th=[ 279], | 99.00th=[ 313], 99.50th=[ 321], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 31, max= 280, per=24.85%, avg=159.77, stdev=44.76, samples=64 iops : min= 7, max= 70, avg=39.47, stdev=11.23, samples=64 lat (msec) : 100=0.98%, 250=87.03%, 500=11.99% cpu : usr=0.30%, sys=1.68%, ctx=3969, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=619KiB/s (633kB/s), 153KiB/s-158KiB/s (157kB/s-162kB/s), io=19.6MiB (20.5MB), run=31938-32441msec WRITE: bw=644KiB/s (659kB/s), 160KiB/s-163KiB/s (164kB/s-167kB/s), io=20.4MiB (21.4MB), run=31938-32441msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179420: Mon Sep 7 12:36:06 2026 read: IOPS=2, BW=84.2MiB/s (88.3MB/s)(40.0MiB/475msec) slat (nsec): min=33141k, max=33141k, avg=33140511.00, stdev= 0.00 clat (nsec): min=424902k, max=424902k, avg=424902236.00, stdev= 0.00 lat (nsec): min=458048k, max=458048k, avg=458048162.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 426], 10.00th=[ 426], 20.00th=[ 426], | 30.00th=[ 426], 40.00th=[ 426], 50.00th=[ 426], 60.00th=[ 426], | 70.00th=[ 426], 80.00th=[ 426], 90.00th=[ 426], 95.00th=[ 426], | 99.00th=[ 426], 99.50th=[ 426], 99.90th=[ 426], 99.95th=[ 426], | 99.99th=[ 426] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.53%, ctx=5, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=84.2MiB/s (88.3MB/s), 84.2MiB/s-84.2MiB/s (88.3MB/s-88.3MB/s), io=40.0MiB (41.9MB), run=475-475msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout