== sanity test 398c: run fio to test AIO ================= 03:14:51 (1787555691) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.700177 s, 59.9 MB/s osc.lustre-OST0000-osc-ffff8c4b70cab800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179967: Mon Aug 24 03:15:32 2026 write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27644msec); 0 zone resets slat (usec): min=21, max=18069, avg=172.11, stdev=720.19 clat (msec): min=60, max=317, avg=171.73, stdev=42.78 lat (msec): min=60, max=317, avg=171.90, stdev=42.78 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 111], 10.00th=[ 124], 20.00th=[ 133], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 167], 60.00th=[ 180], | 70.00th=[ 192], 80.00th=[ 211], 90.00th=[ 228], 95.00th=[ 245], | 99.00th=[ 292], 99.50th=[ 296], 99.90th=[ 317], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 238, max= 510, per=25.02%, avg=370.54, stdev=80.73, samples=54 iops : min= 59, max= 127, avg=92.22, stdev=20.26, samples=54 lat (msec) : 100=1.45%, 250=95.31%, 500=3.24% cpu : usr=0.14%, sys=0.81%, ctx=1471, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179968: Mon Aug 24 03:15:32 2026 write: IOPS=93, BW=372KiB/s (381kB/s)(10.0MiB/27517msec); 0 zone resets slat (usec): min=23, max=17792, avg=175.37, stdev=625.82 clat (msec): min=60, max=401, avg=171.50, stdev=43.96 lat (msec): min=60, max=401, avg=171.67, stdev=43.97 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 110], 10.00th=[ 120], 20.00th=[ 134], | 30.00th=[ 144], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 190], 80.00th=[ 209], 90.00th=[ 230], 95.00th=[ 253], | 99.00th=[ 292], 99.50th=[ 300], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 401] bw ( KiB/s): min= 206, max= 510, per=25.18%, avg=372.89, stdev=72.52, samples=54 iops : min= 51, max= 127, avg=92.81, stdev=18.15, samples=54 lat (msec) : 100=2.15%, 250=92.27%, 500=5.59% cpu : usr=0.13%, sys=0.90%, ctx=2323, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179969: Mon Aug 24 03:15:32 2026 write: IOPS=92, BW=371KiB/s (380kB/s)(10.0MiB/27579msec); 0 zone resets slat (usec): min=21, max=12498, avg=238.48, stdev=781.09 clat (msec): min=19, max=394, avg=171.24, stdev=43.81 lat (msec): min=20, max=394, avg=171.49, stdev=43.81 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 108], 10.00th=[ 118], 20.00th=[ 134], | 30.00th=[ 146], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 190], 80.00th=[ 211], 90.00th=[ 232], 95.00th=[ 247], | 99.00th=[ 292], 99.50th=[ 305], 99.90th=[ 330], 99.95th=[ 359], | 99.99th=[ 397] bw ( KiB/s): min= 238, max= 518, per=25.13%, avg=372.20, stdev=72.71, samples=54 iops : min= 59, max= 129, avg=92.63, stdev=18.26, samples=54 lat (msec) : 20=0.04%, 50=0.27%, 100=1.80%, 250=93.71%, 500=4.18% cpu : usr=0.21%, sys=1.20%, ctx=3156, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179970: Mon Aug 24 03:15:32 2026 write: IOPS=93, BW=373KiB/s (382kB/s)(10.0MiB/27453msec); 0 zone resets slat (usec): min=23, max=13594, avg=167.91, stdev=682.76 clat (msec): min=22, max=430, avg=171.20, stdev=46.97 lat (msec): min=22, max=430, avg=171.37, stdev=46.94 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 108], 10.00th=[ 120], 20.00th=[ 133], | 30.00th=[ 144], 40.00th=[ 155], 50.00th=[ 165], 60.00th=[ 174], | 70.00th=[ 188], 80.00th=[ 209], 90.00th=[ 230], 95.00th=[ 255], | 99.00th=[ 300], 99.50th=[ 309], 99.90th=[ 430], 99.95th=[ 430], | 99.99th=[ 430] bw ( KiB/s): min= 252, max= 518, per=25.20%, avg=373.24, stdev=73.04, samples=54 iops : min= 63, max= 129, avg=92.91, stdev=18.31, samples=54 lat (msec) : 50=0.16%, 100=1.37%, 250=93.12%, 500=5.35% cpu : usr=0.11%, sys=0.84%, ctx=1735, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1482KiB/s (1517kB/s), 370KiB/s-373KiB/s (379kB/s-382kB/s), io=40.0MiB (41.9MB), run=27453-27644msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179984: Mon Aug 24 03:16:18 2026 read: IOPS=37, BW=150KiB/s (154kB/s)(5048KiB/33671msec) slat (msec): min=2, max=182, avg=14.90, stdev= 9.62 clat (msec): min=9, max=411, avg=197.82, stdev=50.67 lat (msec): min=15, max=444, avg=212.73, stdev=52.29 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 123], 10.00th=[ 138], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 218], 80.00th=[ 234], 90.00th=[ 262], 95.00th=[ 292], | 99.00th=[ 359], 99.50th=[ 368], 99.90th=[ 401], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 68, max= 256, per=24.92%, avg=148.29, stdev=39.06, samples=66 iops : min= 17, max= 64, avg=36.55, stdev= 9.84, samples=66 write: IOPS=38, BW=154KiB/s (158kB/s)(5192KiB/33671msec); 0 zone resets slat (usec): min=33, max=22290, avg=287.82, stdev=1338.68 clat (msec): min=83, max=393, avg=204.70, stdev=48.39 lat (msec): min=83, max=393, avg=204.99, stdev=48.36 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 133], 10.00th=[ 148], 20.00th=[ 169], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 199], 60.00th=[ 211], | 70.00th=[ 224], 80.00th=[ 241], 90.00th=[ 268], 95.00th=[ 296], | 99.00th=[ 347], 99.50th=[ 363], 99.90th=[ 388], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 45, max= 224, per=24.71%, avg=153.18, stdev=38.22, samples=66 iops : min= 11, max= 56, avg=37.76, stdev= 9.61, samples=66 lat (msec) : 10=0.04%, 20=0.04%, 100=0.66%, 250=85.04%, 500=14.22% cpu : usr=0.23%, sys=1.78%, ctx=4675, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179985: Mon Aug 24 03:16:18 2026 read: IOPS=36, BW=148KiB/s (151kB/s)(4976KiB/33674msec) slat (usec): min=2573, max=66400, avg=15288.35, stdev=9100.76 clat (msec): min=34, max=406, avg=199.20, stdev=50.27 lat (msec): min=50, max=444, avg=214.49, stdev=52.13 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 122], 10.00th=[ 140], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 197], 60.00th=[ 205], | 70.00th=[ 220], 80.00th=[ 239], 90.00th=[ 264], 95.00th=[ 292], | 99.00th=[ 334], 99.50th=[ 363], 99.90th=[ 393], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 45, max= 224, per=24.59%, avg=146.29, stdev=33.70, samples=66 iops : min= 11, max= 56, avg=36.05, stdev= 8.49, samples=66 write: IOPS=39, BW=156KiB/s (160kB/s)(5264KiB/33674msec); 0 zone resets slat (usec): min=25, max=14202, avg=217.11, stdev=836.81 clat (msec): min=89, max=399, avg=203.32, stdev=48.47 lat (msec): min=89, max=400, avg=203.54, stdev=48.49 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 126], 10.00th=[ 140], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 224], 80.00th=[ 239], 90.00th=[ 262], 95.00th=[ 288], | 99.00th=[ 347], 99.50th=[ 380], 99.90th=[ 388], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 70, max= 248, per=25.11%, avg=155.71, stdev=39.59, samples=66 iops : min= 17, max= 62, avg=38.41, stdev=10.01, samples=66 lat (msec) : 50=0.08%, 100=0.90%, 250=84.02%, 500=15.00% cpu : usr=0.25%, sys=1.74%, ctx=4447, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179986: Mon Aug 24 03:16:18 2026 read: IOPS=37, BW=148KiB/s (152kB/s)(4984KiB/33600msec) slat (usec): min=33, max=60501, avg=14823.92, stdev=8214.09 clat (msec): min=33, max=377, avg=198.29, stdev=51.31 lat (msec): min=34, max=412, avg=213.12, stdev=53.33 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 124], 10.00th=[ 136], 20.00th=[ 155], | 30.00th=[ 171], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 222], 80.00th=[ 241], 90.00th=[ 264], 95.00th=[ 292], | 99.00th=[ 342], 99.50th=[ 363], 99.90th=[ 372], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 79, max= 240, per=24.85%, avg=147.85, stdev=35.62, samples=66 iops : min= 19, max= 60, avg=36.44, stdev= 8.98, samples=66 write: IOPS=39, BW=156KiB/s (160kB/s)(5256KiB/33600msec); 0 zone resets slat (usec): min=24, max=20273, avg=264.10, stdev=1248.24 clat (msec): min=84, max=388, avg=206.16, stdev=44.96 lat (msec): min=84, max=388, avg=206.43, stdev=44.99 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 138], 10.00th=[ 153], 20.00th=[ 171], | 30.00th=[ 182], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 226], 80.00th=[ 241], 90.00th=[ 262], 95.00th=[ 288], | 99.00th=[ 326], 99.50th=[ 359], 99.90th=[ 384], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 87, max= 248, per=25.02%, avg=155.15, stdev=32.25, samples=66 iops : min= 21, max= 62, avg=38.26, stdev= 8.15, samples=66 lat (msec) : 50=0.08%, 100=0.66%, 250=85.00%, 500=14.26% cpu : usr=0.25%, sys=1.74%, ctx=4349, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179987: Mon Aug 24 03:16:18 2026 read: IOPS=37, BW=150KiB/s (154kB/s)(5060KiB/33650msec) slat (usec): min=2679, max=64307, avg=14720.79, stdev=8574.29 clat (msec): min=21, max=417, avg=197.84, stdev=48.69 lat (msec): min=50, max=478, avg=212.56, stdev=50.30 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 124], 10.00th=[ 142], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 194], 60.00th=[ 203], | 70.00th=[ 215], 80.00th=[ 234], 90.00th=[ 262], 95.00th=[ 284], | 99.00th=[ 338], 99.50th=[ 363], 99.90th=[ 414], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 53, max= 248, per=25.08%, avg=149.21, stdev=36.01, samples=66 iops : min= 13, max= 62, avg=36.76, stdev= 9.10, samples=66 write: IOPS=38, BW=154KiB/s (158kB/s)(5180KiB/33650msec); 0 zone resets slat (usec): min=26, max=51298, avg=303.72, stdev=1910.50 clat (msec): min=79, max=378, avg=204.66, stdev=49.40 lat (msec): min=79, max=379, avg=204.97, stdev=49.41 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 130], 10.00th=[ 142], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 241], 90.00th=[ 266], 95.00th=[ 292], | 99.00th=[ 342], 99.50th=[ 368], 99.90th=[ 372], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 84, max= 239, per=24.62%, avg=152.65, stdev=36.75, samples=66 iops : min= 21, max= 59, avg=37.64, stdev= 9.20, samples=66 lat (msec) : 50=0.04%, 100=0.66%, 250=85.12%, 500=14.18% cpu : usr=0.27%, sys=1.92%, ctx=4312, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=596KiB/s (610kB/s), 148KiB/s-150KiB/s (151kB/s-154kB/s), io=19.6MiB (20.5MB), run=33600-33674msec WRITE: bw=620KiB/s (635kB/s), 154KiB/s-156KiB/s (158kB/s-160kB/s), io=20.4MiB (21.4MB), run=33600-33674msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179997: Mon Aug 24 03:16:32 2026 read: IOPS=1, BW=55.1MiB/s (57.8MB/s)(40.0MiB/726msec) slat (nsec): min=24345k, max=24345k, avg=24345311.00, stdev= 0.00 clat (nsec): min=639340k, max=639340k, avg=639339755.00, stdev= 0.00 lat (nsec): min=663692k, max=663692k, avg=663692061.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 642], 5.00th=[ 642], 10.00th=[ 642], 20.00th=[ 642], | 30.00th=[ 642], 40.00th=[ 642], 50.00th=[ 642], 60.00th=[ 642], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.52%, ctx=3, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=55.1MiB/s (57.8MB/s), 55.1MiB/s-55.1MiB/s (57.8MB/s-57.8MB/s), io=40.0MiB (41.9MB), run=726-726msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout