== sanity test 398c: run fio to test AIO ================= 12:11:47 (1790007107) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.629285 s, 66.7 MB/s osc.lustre-OST0000-osc-ffff9345479a5800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180059: Mon Sep 21 12:12:28 2026 write: IOPS=93, BW=372KiB/s (381kB/s)(10.0MiB/27519msec); 0 zone resets slat (usec): min=21, max=12314, avg=143.57, stdev=632.58 clat (msec): min=78, max=296, avg=171.74, stdev=45.33 lat (msec): min=78, max=296, avg=171.89, stdev=45.34 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 107], 10.00th=[ 117], 20.00th=[ 133], | 30.00th=[ 144], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 180], | 70.00th=[ 192], 80.00th=[ 213], 90.00th=[ 236], 95.00th=[ 255], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 231, max= 640, per=24.98%, avg=371.74, stdev=90.38, samples=54 iops : min= 57, max= 160, avg=92.54, stdev=22.62, samples=54 lat (msec) : 100=2.85%, 250=90.70%, 500=6.45% cpu : usr=0.16%, sys=0.61%, ctx=1323, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180060: Mon Sep 21 12:12:28 2026 write: IOPS=93, BW=372KiB/s (381kB/s)(10.0MiB/27515msec); 0 zone resets slat (usec): min=22, max=11365, avg=127.40, stdev=493.21 clat (msec): min=45, max=314, avg=171.18, stdev=45.91 lat (msec): min=45, max=314, avg=171.31, stdev=45.89 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 108], 10.00th=[ 117], 20.00th=[ 134], | 30.00th=[ 144], 40.00th=[ 150], 50.00th=[ 163], 60.00th=[ 176], | 70.00th=[ 194], 80.00th=[ 211], 90.00th=[ 234], 95.00th=[ 255], | 99.00th=[ 292], 99.50th=[ 309], 99.90th=[ 309], 99.95th=[ 309], | 99.99th=[ 313] bw ( KiB/s): min= 250, max= 640, per=24.95%, avg=371.26, stdev=83.30, samples=54 iops : min= 62, max= 160, avg=92.43, stdev=20.82, samples=54 lat (msec) : 50=0.04%, 100=3.28%, 250=90.51%, 500=6.17% cpu : usr=0.09%, sys=0.77%, ctx=1745, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180061: Mon Sep 21 12:12:28 2026 write: IOPS=93, BW=372KiB/s (381kB/s)(10.0MiB/27515msec); 0 zone resets slat (usec): min=24, max=14270, avg=161.50, stdev=569.54 clat (msec): min=31, max=328, avg=171.38, stdev=45.94 lat (msec): min=32, max=328, avg=171.55, stdev=45.92 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 108], 10.00th=[ 120], 20.00th=[ 132], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 190], 80.00th=[ 209], 90.00th=[ 239], 95.00th=[ 268], | 99.00th=[ 288], 99.50th=[ 300], 99.90th=[ 305], 99.95th=[ 317], | 99.99th=[ 330] bw ( KiB/s): min= 160, max= 600, per=24.97%, avg=371.56, stdev=88.69, samples=54 iops : min= 40, max= 150, avg=92.52, stdev=22.20, samples=54 lat (msec) : 50=0.08%, 100=2.73%, 250=90.66%, 500=6.52% cpu : usr=0.17%, sys=0.79%, ctx=2807, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180062: Mon Sep 21 12:12:28 2026 write: IOPS=93, BW=372KiB/s (381kB/s)(10.0MiB/27517msec); 0 zone resets slat (usec): min=24, max=9571, avg=186.89, stdev=644.27 clat (msec): min=17, max=304, avg=171.00, stdev=45.67 lat (msec): min=18, max=305, avg=171.19, stdev=45.64 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 110], 10.00th=[ 118], 20.00th=[ 133], | 30.00th=[ 146], 40.00th=[ 153], 50.00th=[ 165], 60.00th=[ 174], | 70.00th=[ 190], 80.00th=[ 209], 90.00th=[ 234], 95.00th=[ 264], | 99.00th=[ 296], 99.50th=[ 296], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 216, max= 536, per=24.91%, avg=370.67, stdev=81.95, samples=54 iops : min= 54, max= 134, avg=92.30, stdev=20.50, samples=54 lat (msec) : 20=0.04%, 50=0.08%, 100=3.20%, 250=90.12%, 500=6.56% cpu : usr=0.15%, sys=0.87%, ctx=2979, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1488KiB/s (1524kB/s), 372KiB/s-372KiB/s (381kB/s-381kB/s), io=40.0MiB (41.9MB), run=27515-27519msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180076: Mon Sep 21 12:13:02 2026 read: IOPS=46, BW=187KiB/s (192kB/s)(5048KiB/26964msec) slat (usec): min=45, max=56585, avg=12139.04, stdev=6998.28 clat (msec): min=12, max=347, avg=159.75, stdev=41.55 lat (msec): min=14, max=355, avg=171.89, stdev=42.89 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 100], 10.00th=[ 112], 20.00th=[ 126], | 30.00th=[ 138], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 167], | 70.00th=[ 178], 80.00th=[ 190], 90.00th=[ 213], 95.00th=[ 230], | 99.00th=[ 275], 99.50th=[ 288], 99.90th=[ 342], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 111, max= 318, per=25.21%, avg=186.55, stdev=45.23, samples=53 iops : min= 27, max= 79, avg=46.21, stdev=11.47, samples=53 write: IOPS=48, BW=193KiB/s (197kB/s)(5192KiB/26964msec); 0 zone resets slat (usec): min=27, max=6066, avg=167.38, stdev=382.45 clat (msec): min=63, max=349, avg=164.67, stdev=38.06 lat (msec): min=64, max=349, avg=164.84, stdev=38.06 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 107], 10.00th=[ 120], 20.00th=[ 134], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 182], 80.00th=[ 194], 90.00th=[ 211], 95.00th=[ 228], | 99.00th=[ 271], 99.50th=[ 309], 99.90th=[ 347], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 111, max= 295, per=24.94%, avg=192.00, stdev=41.33, samples=53 iops : min= 27, max= 73, avg=47.57, stdev=10.37, samples=53 lat (msec) : 20=0.12%, 50=0.12%, 100=4.10%, 250=93.67%, 500=1.99% cpu : usr=0.27%, sys=1.89%, ctx=4473, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180077: Mon Sep 21 12:13:02 2026 read: IOPS=46, BW=184KiB/s (188kB/s)(4976KiB/27039msec) slat (usec): min=2359, max=52126, avg=12041.98, stdev=6762.90 clat (msec): min=21, max=325, avg=160.52, stdev=40.70 lat (msec): min=34, max=336, avg=172.56, stdev=41.85 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 99], 10.00th=[ 111], 20.00th=[ 128], | 30.00th=[ 138], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 169], | 70.00th=[ 180], 80.00th=[ 194], 90.00th=[ 213], 95.00th=[ 232], | 99.00th=[ 266], 99.50th=[ 275], 99.90th=[ 300], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 97, max= 302, per=24.77%, avg=183.28, stdev=44.21, samples=53 iops : min= 24, max= 75, avg=45.36, stdev=11.14, samples=53 write: IOPS=48, BW=195KiB/s (199kB/s)(5264KiB/27039msec); 0 zone resets slat (usec): min=27, max=6581, avg=158.83, stdev=340.98 clat (msec): min=78, max=301, avg=163.69, stdev=37.32 lat (msec): min=78, max=301, avg=163.85, stdev=37.33 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 106], 10.00th=[ 115], 20.00th=[ 131], | 30.00th=[ 142], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 213], 95.00th=[ 226], | 99.00th=[ 264], 99.50th=[ 266], 99.90th=[ 284], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 87, max= 318, per=25.31%, avg=194.85, stdev=47.69, samples=53 iops : min= 21, max= 79, avg=48.28, stdev=12.02, samples=53 lat (msec) : 50=0.12%, 100=4.34%, 250=94.02%, 500=1.52% cpu : usr=0.29%, sys=1.86%, ctx=4625, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180078: Mon Sep 21 12:13:02 2026 read: IOPS=45, BW=184KiB/s (188kB/s)(4984KiB/27107msec) slat (usec): min=2190, max=58605, avg=12030.04, stdev=6826.29 clat (msec): min=27, max=296, avg=158.20, stdev=38.31 lat (msec): min=55, max=321, avg=170.24, stdev=39.52 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 101], 10.00th=[ 112], 20.00th=[ 127], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 155], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 190], 90.00th=[ 209], 95.00th=[ 226], | 99.00th=[ 259], 99.50th=[ 279], 99.90th=[ 296], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 90, max= 333, per=24.89%, avg=184.15, stdev=46.81, samples=53 iops : min= 22, max= 83, avg=45.58, stdev=11.82, samples=53 write: IOPS=48, BW=194KiB/s (199kB/s)(5256KiB/27107msec); 0 zone resets slat (usec): min=26, max=7550, avg=177.95, stdev=508.55 clat (msec): min=66, max=339, avg=166.19, stdev=40.05 lat (msec): min=66, max=339, avg=166.37, stdev=40.09 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 109], 10.00th=[ 121], 20.00th=[ 133], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 218], 95.00th=[ 234], | 99.00th=[ 292], 99.50th=[ 309], 99.90th=[ 334], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 105, max= 308, per=25.03%, avg=192.75, stdev=45.56, samples=53 iops : min= 26, max= 77, avg=47.75, stdev=11.52, samples=53 lat (msec) : 50=0.04%, 100=3.83%, 250=93.87%, 500=2.27% cpu : usr=0.30%, sys=1.92%, ctx=4412, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180079: Mon Sep 21 12:13:02 2026 read: IOPS=46, BW=187KiB/s (191kB/s)(5060KiB/27117msec) slat (usec): min=2257, max=43849, avg=11880.55, stdev=6419.81 clat (msec): min=16, max=334, avg=157.43, stdev=39.47 lat (msec): min=47, max=352, avg=169.31, stdev=40.89 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 95], 10.00th=[ 110], 20.00th=[ 126], | 30.00th=[ 138], 40.00th=[ 144], 50.00th=[ 155], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 190], 90.00th=[ 207], 95.00th=[ 224], | 99.00th=[ 268], 99.50th=[ 275], 99.90th=[ 321], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 75, max= 287, per=25.13%, avg=185.94, stdev=47.45, samples=53 iops : min= 18, max= 71, avg=46.04, stdev=12.01, samples=53 write: IOPS=47, BW=191KiB/s (196kB/s)(5180KiB/27117msec); 0 zone resets slat (usec): min=35, max=10423, avg=200.30, stdev=638.46 clat (msec): min=57, max=306, avg=166.71, stdev=38.40 lat (msec): min=57, max=306, avg=166.92, stdev=38.38 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 106], 10.00th=[ 117], 20.00th=[ 134], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 186], 80.00th=[ 199], 90.00th=[ 213], 95.00th=[ 230], | 99.00th=[ 266], 99.50th=[ 275], 99.90th=[ 292], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 90, max= 270, per=24.62%, avg=189.60, stdev=39.96, samples=53 iops : min= 22, max= 67, avg=46.98, stdev=10.08, samples=53 lat (msec) : 20=0.04%, 50=0.04%, 100=4.57%, 250=93.24%, 500=2.11% cpu : usr=0.27%, sys=2.02%, ctx=4534, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=740KiB/s (758kB/s), 184KiB/s-187KiB/s (188kB/s-192kB/s), io=19.6MiB (20.5MB), run=26964-27117msec WRITE: bw=770KiB/s (789kB/s), 191KiB/s-195KiB/s (196kB/s-199kB/s), io=20.4MiB (21.4MB), run=26964-27117msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180089: Mon Sep 21 12:13:12 2026 read: IOPS=1, BW=65.3MiB/s (68.4MB/s)(40.0MiB/613msec) slat (nsec): min=49057k, max=49057k, avg=49057446.00, stdev= 0.00 clat (nsec): min=529007k, max=529007k, avg=529006580.00, stdev= 0.00 lat (nsec): min=578070k, max=578070k, avg=578069774.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 531], 5.00th=[ 531], 10.00th=[ 531], 20.00th=[ 531], | 30.00th=[ 531], 40.00th=[ 531], 50.00th=[ 531], 60.00th=[ 531], | 70.00th=[ 531], 80.00th=[ 531], 90.00th=[ 531], 95.00th=[ 531], | 99.00th=[ 531], 99.50th=[ 531], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.96%, ctx=5, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=65.3MiB/s (68.4MB/s), 65.3MiB/s-65.3MiB/s (68.4MB/s-68.4MB/s), io=40.0MiB (41.9MB), run=613-613msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout