== sanity test 398c: run fio to test AIO ================= 03:38:40 (1782632320) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.12221 s, 343 MB/s osc.lustre-OST0000-osc-ffff8f5950590000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=160814: Sun Jun 28 03:39:13 2026 write: IOPS=85, BW=340KiB/s (348kB/s)(10.0MiB/30097msec); 0 zone resets slat (usec): min=49, max=1248, avg=227.80, stdev=117.56 clat (msec): min=52, max=545, avg=187.51, stdev=46.22 lat (msec): min=52, max=545, avg=187.74, stdev=46.23 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 129], 10.00th=[ 146], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 215], 90.00th=[ 230], 95.00th=[ 245], | 99.00th=[ 355], 99.50th=[ 405], 99.90th=[ 531], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 192, max= 682, per=25.10%, avg=341.32, stdev=64.09, samples=59 iops : min= 48, max= 170, avg=85.32, stdev=15.98, samples=59 lat (msec) : 100=3.87%, 250=92.11%, 500=3.87%, 750=0.16% cpu : usr=0.12%, sys=1.52%, ctx=2802, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=160815: Sun Jun 28 03:39:13 2026 write: IOPS=85, BW=342KiB/s (350kB/s)(10.0MiB/29974msec); 0 zone resets slat (usec): min=41, max=1213, avg=224.61, stdev=113.14 clat (msec): min=31, max=469, avg=186.94, stdev=44.93 lat (msec): min=31, max=469, avg=187.16, stdev=44.94 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 129], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 215], 90.00th=[ 232], 95.00th=[ 247], | 99.00th=[ 355], 99.50th=[ 376], 99.90th=[ 456], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 192, max= 714, per=25.14%, avg=341.86, stdev=69.26, samples=59 iops : min= 48, max= 178, avg=85.46, stdev=17.27, samples=59 lat (msec) : 50=0.31%, 100=3.71%, 250=92.07%, 500=3.91% cpu : usr=0.13%, sys=1.47%, ctx=2850, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=160816: Sun Jun 28 03:39:13 2026 write: IOPS=85, BW=341KiB/s (349kB/s)(10.0MiB/30060msec); 0 zone resets slat (usec): min=43, max=1112, avg=226.63, stdev=115.79 clat (msec): min=50, max=526, avg=187.32, stdev=44.28 lat (msec): min=50, max=526, avg=187.55, stdev=44.28 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 129], 10.00th=[ 146], 20.00th=[ 159], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 218], 90.00th=[ 230], 95.00th=[ 245], | 99.00th=[ 355], 99.50th=[ 384], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 527] bw ( KiB/s): min= 216, max= 682, per=25.05%, avg=340.64, stdev=66.77, samples=59 iops : min= 54, max= 170, avg=85.15, stdev=16.65, samples=59 lat (msec) : 100=3.95%, 250=91.91%, 500=4.10%, 750=0.04% cpu : usr=0.12%, sys=1.49%, ctx=2856, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=160817: Sun Jun 28 03:39:13 2026 write: IOPS=85, BW=340KiB/s (348kB/s)(10.0MiB/30097msec); 0 zone resets slat (usec): min=48, max=1653, avg=224.22, stdev=113.74 clat (msec): min=50, max=528, avg=187.59, stdev=45.25 lat (msec): min=50, max=529, avg=187.82, stdev=45.26 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 129], 10.00th=[ 146], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 215], 90.00th=[ 230], 95.00th=[ 245], | 99.00th=[ 355], 99.50th=[ 388], 99.90th=[ 531], 99.95th=[ 531], | 99.99th=[ 531] bw ( KiB/s): min= 192, max= 669, per=25.06%, avg=340.83, stdev=63.99, samples=59 iops : min= 48, max= 167, avg=85.20, stdev=15.98, samples=59 lat (msec) : 100=3.91%, 250=92.07%, 500=3.87%, 750=0.16% cpu : usr=0.13%, sys=1.49%, ctx=2803, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1361KiB/s (1394kB/s), 340KiB/s-342KiB/s (348kB/s-350kB/s), io=40.0MiB (41.9MB), run=29974-30097msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=160831: Sun Jun 28 03:39:37 2026 read: IOPS=60, BW=244KiB/s (250kB/s)(5048KiB/20700msec) slat (usec): min=1122, max=10318, avg=2785.59, stdev=1078.29 clat (msec): min=5, max=208, avg=79.75, stdev=16.65 lat (msec): min=8, max=212, avg=82.54, stdev=16.89 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 57], 10.00th=[ 62], 20.00th=[ 68], | 30.00th=[ 71], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 83], | 70.00th=[ 87], 80.00th=[ 92], 90.00th=[ 100], 95.00th=[ 107], | 99.00th=[ 124], 99.50th=[ 134], 99.90th=[ 207], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 144, max= 368, per=25.23%, avg=244.27, stdev=52.91, samples=41 iops : min= 36, max= 92, avg=61.05, stdev=13.24, samples=41 write: IOPS=62, BW=251KiB/s (257kB/s)(5192KiB/20700msec); 0 zone resets slat (usec): min=64, max=904, avg=211.74, stdev=116.35 clat (msec): min=71, max=325, avg=173.62, stdev=37.81 lat (msec): min=71, max=325, avg=173.83, stdev=37.81 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 112], 10.00th=[ 125], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 205], 90.00th=[ 224], 95.00th=[ 236], | 99.00th=[ 271], 99.50th=[ 288], 99.90th=[ 313], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 200, max= 360, per=24.81%, avg=249.88, stdev=36.12, samples=41 iops : min= 50, max= 90, avg=62.46, stdev= 9.04, samples=41 lat (msec) : 10=0.12%, 20=0.08%, 50=0.74%, 100=44.65%, 250=53.28% lat (msec) : 500=1.13% cpu : usr=0.19%, sys=2.41%, ctx=4813, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=160832: Sun Jun 28 03:39:37 2026 read: IOPS=60, BW=240KiB/s (246kB/s)(4976KiB/20728msec) slat (usec): min=1111, max=8329, avg=2808.40, stdev=1095.56 clat (msec): min=6, max=135, avg=79.60, stdev=15.58 lat (msec): min=8, max=141, avg=82.41, stdev=15.92 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 83], | 70.00th=[ 86], 80.00th=[ 93], 90.00th=[ 100], 95.00th=[ 106], | 99.00th=[ 123], 99.50th=[ 130], 99.90th=[ 133], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 168, max= 360, per=24.76%, avg=239.68, stdev=50.58, samples=41 iops : min= 42, max= 90, avg=59.90, stdev=12.66, samples=41 write: IOPS=63, BW=254KiB/s (260kB/s)(5264KiB/20728msec); 0 zone resets slat (usec): min=53, max=863, avg=212.38, stdev=123.15 clat (msec): min=26, max=282, avg=173.22, stdev=36.96 lat (msec): min=26, max=282, avg=173.43, stdev=36.96 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 110], 10.00th=[ 125], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 205], 90.00th=[ 222], 95.00th=[ 232], | 99.00th=[ 253], 99.50th=[ 266], 99.90th=[ 279], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 192, max= 336, per=25.09%, avg=252.66, stdev=36.98, samples=41 iops : min= 48, max= 84, avg=63.15, stdev= 9.26, samples=41 lat (msec) : 10=0.04%, 20=0.20%, 50=0.59%, 100=44.22%, 250=54.34% lat (msec) : 500=0.62% cpu : usr=0.20%, sys=2.34%, ctx=4832, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=160833: Sun Jun 28 03:39:37 2026 read: IOPS=60, BW=241KiB/s (247kB/s)(4984KiB/20670msec) slat (usec): min=72, max=9822, avg=2786.62, stdev=1090.52 clat (msec): min=3, max=213, avg=80.06, stdev=15.82 lat (msec): min=7, max=216, avg=82.85, stdev=16.07 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 69], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 84], | 70.00th=[ 87], 80.00th=[ 92], 90.00th=[ 100], 95.00th=[ 106], | 99.00th=[ 121], 99.50th=[ 125], 99.90th=[ 133], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 168, max= 384, per=24.95%, avg=241.54, stdev=51.70, samples=41 iops : min= 42, max= 96, avg=60.37, stdev=12.93, samples=41 write: IOPS=63, BW=254KiB/s (260kB/s)(5256KiB/20670msec); 0 zone resets slat (usec): min=54, max=2791, avg=218.84, stdev=156.22 clat (msec): min=57, max=371, avg=172.53, stdev=39.48 lat (msec): min=57, max=371, avg=172.75, stdev=39.49 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 109], 10.00th=[ 123], 20.00th=[ 140], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 207], 90.00th=[ 224], 95.00th=[ 236], | 99.00th=[ 266], 99.50th=[ 296], 99.90th=[ 351], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 160, max= 344, per=25.03%, avg=252.07, stdev=43.82, samples=41 iops : min= 40, max= 86, avg=63.00, stdev=10.95, samples=41 lat (msec) : 4=0.04%, 10=0.04%, 20=0.12%, 50=0.55%, 100=44.77% lat (msec) : 250=53.48%, 500=1.02% cpu : usr=0.17%, sys=2.38%, ctx=4862, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=160834: Sun Jun 28 03:39:37 2026 read: IOPS=61, BW=244KiB/s (250kB/s)(5060KiB/20712msec) slat (usec): min=1093, max=9105, avg=2750.05, stdev=1053.56 clat (msec): min=2, max=134, avg=79.59, stdev=15.95 lat (msec): min=6, max=136, avg=82.34, stdev=16.24 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 56], 10.00th=[ 62], 20.00th=[ 68], | 30.00th=[ 73], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 87], 80.00th=[ 92], 90.00th=[ 100], 95.00th=[ 107], | 99.00th=[ 122], 99.50th=[ 128], 99.90th=[ 131], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 136, max= 424, per=25.36%, avg=245.46, stdev=58.31, samples=41 iops : min= 34, max= 106, avg=61.37, stdev=14.58, samples=41 write: IOPS=62, BW=250KiB/s (256kB/s)(5180KiB/20712msec); 0 zone resets slat (usec): min=46, max=948, avg=208.74, stdev=118.45 clat (msec): min=71, max=289, avg=174.16, stdev=38.14 lat (msec): min=71, max=289, avg=174.37, stdev=38.14 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 112], 10.00th=[ 124], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 205], 90.00th=[ 226], 95.00th=[ 241], | 99.00th=[ 271], 99.50th=[ 284], 99.90th=[ 288], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 184, max= 336, per=24.70%, avg=248.71, stdev=35.68, samples=41 iops : min= 46, max= 84, avg=62.17, stdev= 8.93, samples=41 lat (msec) : 4=0.04%, 10=0.12%, 20=0.12%, 50=0.62%, 100=44.53% lat (msec) : 250=53.12%, 500=1.45% cpu : usr=0.17%, sys=2.36%, ctx=4849, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=968KiB/s (991kB/s), 240KiB/s-244KiB/s (246kB/s-250kB/s), io=19.6MiB (20.5MB), run=20670-20728msec WRITE: bw=1008KiB/s (1032kB/s), 250KiB/s-254KiB/s (256kB/s-260kB/s), io=20.4MiB (21.4MB), run=20670-20728msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=160840: Sun Jun 28 03:39:40 2026 read: IOPS=7, BW=301MiB/s (315MB/s)(40.0MiB/133msec) slat (nsec): min=8675.3k, max=8675.3k, avg=8675252.00, stdev= 0.00 clat (nsec): min=119135k, max=119135k, avg=119135388.00, stdev= 0.00 lat (nsec): min=127813k, max=127813k, avg=127813406.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 120], 10.00th=[ 120], 20.00th=[ 120], | 30.00th=[ 120], 40.00th=[ 120], 50.00th=[ 120], 60.00th=[ 120], | 70.00th=[ 120], 80.00th=[ 120], 90.00th=[ 120], 95.00th=[ 120], | 99.00th=[ 120], 99.50th=[ 120], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.06%, ctx=9, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=301MiB/s (315MB/s), 301MiB/s-301MiB/s (315MB/s-315MB/s), io=40.0MiB (41.9MB), run=133-133msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout