== sanity test 398c: run fio to test AIO ================= 03:18:55 (1788592735) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.44278 s, 94.7 MB/s osc.lustre-OST0000-osc-ffffa077d895e800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180005: Sat Sep 5 03:19:25 2026 write: IOPS=118, BW=476KiB/s (487kB/s)(10.0MiB/21513msec); 0 zone resets slat (usec): min=23, max=11024, avg=190.23, stdev=577.59 clat (msec): min=24, max=238, avg=133.37, stdev=27.03 lat (msec): min=24, max=238, avg=133.57, stdev=27.03 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 89], 10.00th=[ 100], 20.00th=[ 112], | 30.00th=[ 120], 40.00th=[ 127], 50.00th=[ 134], 60.00th=[ 142], | 70.00th=[ 148], 80.00th=[ 155], 90.00th=[ 167], 95.00th=[ 178], | 99.00th=[ 194], 99.50th=[ 203], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 239] bw ( KiB/s): min= 380, max= 640, per=24.84%, avg=472.62, stdev=72.70, samples=42 iops : min= 95, max= 160, avg=117.83, stdev=18.27, samples=42 lat (msec) : 50=0.12%, 100=10.94%, 250=88.95% cpu : usr=0.20%, sys=1.17%, ctx=2400, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180006: Sat Sep 5 03:19:25 2026 write: IOPS=119, BW=478KiB/s (490kB/s)(10.0MiB/21404msec); 0 zone resets slat (usec): min=28, max=15289, avg=234.47, stdev=673.10 clat (msec): min=14, max=238, avg=133.38, stdev=28.96 lat (msec): min=14, max=238, avg=133.61, stdev=28.95 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 88], 10.00th=[ 99], 20.00th=[ 111], | 30.00th=[ 118], 40.00th=[ 127], 50.00th=[ 134], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 157], 90.00th=[ 169], 95.00th=[ 180], | 99.00th=[ 203], 99.50th=[ 220], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 239] bw ( KiB/s): min= 348, max= 706, per=25.08%, avg=477.29, stdev=78.92, samples=42 iops : min= 87, max= 176, avg=119.00, stdev=19.84, samples=42 lat (msec) : 20=0.16%, 50=0.62%, 100=9.96%, 250=89.26% cpu : usr=0.27%, sys=1.51%, ctx=3132, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180007: Sat Sep 5 03:19:25 2026 write: IOPS=118, BW=476KiB/s (487kB/s)(10.0MiB/21513msec); 0 zone resets slat (usec): min=23, max=18239, avg=198.14, stdev=651.17 clat (msec): min=14, max=237, avg=133.24, stdev=27.69 lat (msec): min=15, max=237, avg=133.44, stdev=27.69 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 89], 10.00th=[ 99], 20.00th=[ 112], | 30.00th=[ 121], 40.00th=[ 127], 50.00th=[ 134], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 157], 90.00th=[ 169], 95.00th=[ 178], | 99.00th=[ 192], 99.50th=[ 203], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 380, max= 637, per=24.88%, avg=473.55, stdev=72.51, samples=42 iops : min= 95, max= 159, avg=118.07, stdev=18.23, samples=42 lat (msec) : 20=0.04%, 100=10.82%, 250=89.14% cpu : usr=0.19%, sys=1.15%, ctx=2443, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180008: Sat Sep 5 03:19:25 2026 write: IOPS=118, BW=476KiB/s (487kB/s)(10.0MiB/21521msec); 0 zone resets slat (usec): min=21, max=20049, avg=184.08, stdev=611.49 clat (msec): min=10, max=220, avg=133.28, stdev=27.28 lat (msec): min=11, max=220, avg=133.46, stdev=27.30 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 89], 10.00th=[ 97], 20.00th=[ 110], | 30.00th=[ 118], 40.00th=[ 127], 50.00th=[ 136], 60.00th=[ 142], | 70.00th=[ 150], 80.00th=[ 157], 90.00th=[ 167], 95.00th=[ 178], | 99.00th=[ 194], 99.50th=[ 207], 99.90th=[ 209], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 380, max= 640, per=24.90%, avg=473.90, stdev=77.55, samples=42 iops : min= 95, max= 160, avg=118.19, stdev=19.51, samples=42 lat (msec) : 20=0.04%, 50=0.20%, 100=11.48%, 250=88.28% cpu : usr=0.17%, sys=1.04%, ctx=2259, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1903KiB/s (1949kB/s), 476KiB/s-478KiB/s (487kB/s-490kB/s), io=40.0MiB (41.9MB), run=21404-21521msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180021: Sat Sep 5 03:19:53 2026 read: IOPS=57, BW=231KiB/s (237kB/s)(5048KiB/21834msec) slat (usec): min=2485, max=37997, avg=9246.40, stdev=4529.98 clat (msec): min=46, max=251, avg=129.51, stdev=31.61 lat (msec): min=55, max=262, avg=138.76, stdev=32.80 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 86], 10.00th=[ 93], 20.00th=[ 103], | 30.00th=[ 111], 40.00th=[ 118], 50.00th=[ 125], 60.00th=[ 133], | 70.00th=[ 144], 80.00th=[ 155], 90.00th=[ 176], 95.00th=[ 190], | 99.00th=[ 211], 99.50th=[ 228], 99.90th=[ 241], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 135, max= 328, per=25.11%, avg=230.23, stdev=47.69, samples=43 iops : min= 33, max= 82, avg=57.14, stdev=11.98, samples=43 write: IOPS=59, BW=238KiB/s (244kB/s)(5192KiB/21834msec); 0 zone resets slat (usec): min=30, max=31256, avg=258.25, stdev=1039.50 clat (msec): min=17, max=266, avg=132.57, stdev=30.51 lat (msec): min=17, max=266, avg=132.83, stdev=30.56 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 92], 10.00th=[ 99], 20.00th=[ 107], | 30.00th=[ 113], 40.00th=[ 121], 50.00th=[ 129], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 174], 95.00th=[ 188], | 99.00th=[ 222], 99.50th=[ 232], 99.90th=[ 266], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 120, max= 327, per=24.79%, avg=236.74, stdev=52.17, samples=43 iops : min= 30, max= 81, avg=58.77, stdev=13.08, samples=43 lat (msec) : 20=0.04%, 50=0.08%, 100=14.30%, 250=85.43%, 500=0.16% cpu : usr=0.27%, sys=2.08%, ctx=4475, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180022: Sat Sep 5 03:19:53 2026 read: IOPS=57, BW=229KiB/s (235kB/s)(4976KiB/21685msec) slat (usec): min=31, max=34175, avg=9530.09, stdev=4690.07 clat (msec): min=36, max=230, avg=128.80, stdev=32.08 lat (msec): min=36, max=245, avg=138.34, stdev=33.59 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 84], 10.00th=[ 91], 20.00th=[ 102], | 30.00th=[ 110], 40.00th=[ 118], 50.00th=[ 126], 60.00th=[ 133], | 70.00th=[ 144], 80.00th=[ 157], 90.00th=[ 174], 95.00th=[ 188], | 99.00th=[ 215], 99.50th=[ 220], 99.90th=[ 230], 99.95th=[ 230], | 99.99th=[ 230] bw ( KiB/s): min= 127, max= 344, per=24.82%, avg=227.58, stdev=53.86, samples=43 iops : min= 31, max= 86, avg=56.49, stdev=13.58, samples=43 write: IOPS=60, BW=243KiB/s (249kB/s)(5264KiB/21685msec); 0 zone resets slat (usec): min=34, max=8507, avg=216.86, stdev=529.88 clat (msec): min=27, max=241, avg=132.34, stdev=31.97 lat (msec): min=27, max=241, avg=132.56, stdev=31.99 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 89], 10.00th=[ 97], 20.00th=[ 108], | 30.00th=[ 115], 40.00th=[ 122], 50.00th=[ 129], 60.00th=[ 138], | 70.00th=[ 148], 80.00th=[ 161], 90.00th=[ 176], 95.00th=[ 188], | 99.00th=[ 213], 99.50th=[ 222], 99.90th=[ 243], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 143, max= 358, per=25.29%, avg=241.56, stdev=55.90, samples=43 iops : min= 35, max= 89, avg=59.95, stdev=14.02, samples=43 lat (msec) : 50=0.98%, 100=14.88%, 250=84.14% cpu : usr=0.34%, sys=2.03%, ctx=4589, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180023: Sat Sep 5 03:19:53 2026 read: IOPS=56, BW=228KiB/s (233kB/s)(4984KiB/21876msec) slat (usec): min=2795, max=39436, avg=9471.16, stdev=4453.35 clat (msec): min=33, max=242, avg=130.63, stdev=32.10 lat (msec): min=38, max=249, avg=140.11, stdev=33.46 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 86], 10.00th=[ 93], 20.00th=[ 104], | 30.00th=[ 111], 40.00th=[ 121], 50.00th=[ 128], 60.00th=[ 136], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 174], 95.00th=[ 192], | 99.00th=[ 228], 99.50th=[ 230], 99.90th=[ 241], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 127, max= 328, per=24.74%, avg=226.86, stdev=45.13, samples=43 iops : min= 31, max= 82, avg=56.30, stdev=11.34, samples=43 write: IOPS=60, BW=240KiB/s (246kB/s)(5256KiB/21876msec); 0 zone resets slat (usec): min=29, max=7104, avg=215.32, stdev=488.15 clat (msec): min=15, max=238, avg=132.01, stdev=30.27 lat (msec): min=16, max=239, avg=132.23, stdev=30.28 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 90], 10.00th=[ 97], 20.00th=[ 107], | 30.00th=[ 113], 40.00th=[ 122], 50.00th=[ 129], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 176], 95.00th=[ 188], | 99.00th=[ 215], 99.50th=[ 222], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 127, max= 344, per=24.94%, avg=238.16, stdev=58.71, samples=43 iops : min= 31, max= 86, avg=59.12, stdev=14.73, samples=43 lat (msec) : 20=0.04%, 50=0.12%, 100=14.53%, 250=85.31% cpu : usr=0.23%, sys=2.11%, ctx=4334, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180024: Sat Sep 5 03:19:53 2026 read: IOPS=57, BW=231KiB/s (237kB/s)(5060KiB/21876msec) slat (usec): min=2339, max=31966, avg=9316.42, stdev=4386.07 clat (msec): min=61, max=236, avg=129.64, stdev=31.92 lat (msec): min=68, max=257, avg=138.96, stdev=33.18 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 86], 10.00th=[ 92], 20.00th=[ 102], | 30.00th=[ 110], 40.00th=[ 118], 50.00th=[ 126], 60.00th=[ 134], | 70.00th=[ 144], 80.00th=[ 155], 90.00th=[ 174], 95.00th=[ 190], | 99.00th=[ 218], 99.50th=[ 228], 99.90th=[ 232], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 127, max= 327, per=25.14%, avg=230.49, stdev=48.16, samples=43 iops : min= 31, max= 81, avg=57.19, stdev=12.12, samples=43 write: IOPS=59, BW=237KiB/s (242kB/s)(5180KiB/21876msec); 0 zone resets slat (usec): min=25, max=5743, avg=200.02, stdev=416.39 clat (msec): min=34, max=234, avg=132.91, stdev=31.14 lat (msec): min=34, max=234, avg=133.11, stdev=31.15 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 90], 10.00th=[ 97], 20.00th=[ 106], | 30.00th=[ 114], 40.00th=[ 122], 50.00th=[ 130], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 159], 90.00th=[ 176], 95.00th=[ 190], | 99.00th=[ 218], 99.50th=[ 222], 99.90th=[ 230], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 136, max= 344, per=24.59%, avg=234.84, stdev=48.84, samples=43 iops : min= 34, max= 86, avg=58.30, stdev=12.26, samples=43 lat (msec) : 50=0.08%, 100=16.02%, 250=83.91% cpu : usr=0.32%, sys=2.00%, ctx=4564, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=917KiB/s (939kB/s), 228KiB/s-231KiB/s (233kB/s-237kB/s), io=19.6MiB (20.5MB), run=21685-21876msec WRITE: bw=955KiB/s (978kB/s), 237KiB/s-243KiB/s (242kB/s-249kB/s), io=20.4MiB (21.4MB), run=21685-21876msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180031: Sat Sep 5 03:20:00 2026 read: IOPS=4, BW=167MiB/s (175MB/s)(40.0MiB/240msec) slat (nsec): min=26614k, max=26614k, avg=26614479.00, stdev= 0.00 clat (nsec): min=194728k, max=194728k, avg=194727953.00, stdev= 0.00 lat (nsec): min=221347k, max=221347k, avg=221346642.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 194], 5.00th=[ 194], 10.00th=[ 194], 20.00th=[ 194], | 30.00th=[ 194], 40.00th=[ 194], 50.00th=[ 194], 60.00th=[ 194], | 70.00th=[ 194], 80.00th=[ 194], 90.00th=[ 194], 95.00th=[ 194], | 99.00th=[ 194], 99.50th=[ 194], 99.90th=[ 194], 99.95th=[ 194], | 99.99th=[ 194] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.02%, ctx=8, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=40.0MiB (41.9MB), run=240-240msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout