== sanity test 398c: run fio to test AIO ================= 06:58:51 (1789988331) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.62888 s, 66.7 MB/s osc.lustre-OST0000-osc-ffff99ff60f9c000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=189107: Mon Sep 21 06:59:27 2026 write: IOPS=97, BW=390KiB/s (399kB/s)(10.0MiB/26272msec); 0 zone resets slat (usec): min=29, max=7836, avg=213.43, stdev=559.32 clat (msec): min=19, max=330, avg=163.80, stdev=35.31 lat (msec): min=19, max=330, avg=164.02, stdev=35.28 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 116], 10.00th=[ 124], 20.00th=[ 133], | 30.00th=[ 142], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 174], | 70.00th=[ 184], 80.00th=[ 194], 90.00th=[ 211], 95.00th=[ 222], | 99.00th=[ 241], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 330] bw ( KiB/s): min= 263, max= 508, per=24.95%, avg=388.02, stdev=63.59, samples=52 iops : min= 65, max= 127, avg=96.63, stdev=15.97, samples=52 lat (msec) : 20=0.04%, 50=0.55%, 100=1.21%, 250=97.50%, 500=0.70% cpu : usr=0.22%, sys=0.94%, ctx=2566, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=189108: Mon Sep 21 06:59:27 2026 write: IOPS=97, BW=389KiB/s (398kB/s)(10.0MiB/26338msec); 0 zone resets slat (usec): min=23, max=7336, avg=172.40, stdev=472.88 clat (msec): min=37, max=272, avg=164.21, stdev=35.73 lat (msec): min=37, max=272, avg=164.39, stdev=35.71 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 116], 10.00th=[ 123], 20.00th=[ 132], | 30.00th=[ 142], 40.00th=[ 150], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 215], 95.00th=[ 228], | 99.00th=[ 241], 99.50th=[ 245], 99.90th=[ 262], 99.95th=[ 262], | 99.99th=[ 271] bw ( KiB/s): min= 255, max= 512, per=24.81%, avg=385.83, stdev=69.31, samples=52 iops : min= 63, max= 128, avg=96.12, stdev=17.38, samples=52 lat (msec) : 50=0.66%, 100=0.82%, 250=98.24%, 500=0.27% cpu : usr=0.15%, sys=0.80%, ctx=1495, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=189109: Mon Sep 21 06:59:27 2026 write: IOPS=97, BW=389KiB/s (399kB/s)(10.0MiB/26306msec); 0 zone resets slat (usec): min=30, max=21108, avg=216.86, stdev=693.38 clat (msec): min=20, max=267, avg=163.69, stdev=36.02 lat (msec): min=21, max=267, avg=163.91, stdev=35.99 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 114], 10.00th=[ 123], 20.00th=[ 133], | 30.00th=[ 142], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 213], 95.00th=[ 224], | 99.00th=[ 241], 99.50th=[ 245], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 252, max= 588, per=24.96%, avg=388.08, stdev=72.84, samples=52 iops : min= 63, max= 147, avg=96.65, stdev=18.28, samples=52 lat (msec) : 50=0.74%, 100=1.64%, 250=97.19%, 500=0.43% cpu : usr=0.20%, sys=1.18%, ctx=2756, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=189110: Mon Sep 21 06:59:27 2026 write: IOPS=97, BW=389KiB/s (398kB/s)(10.0MiB/26331msec); 0 zone resets slat (usec): min=22, max=20098, avg=198.37, stdev=762.43 clat (msec): min=22, max=283, avg=163.92, stdev=35.24 lat (msec): min=22, max=283, avg=164.12, stdev=35.21 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 115], 10.00th=[ 122], 20.00th=[ 134], | 30.00th=[ 144], 40.00th=[ 150], 50.00th=[ 161], 60.00th=[ 174], | 70.00th=[ 184], 80.00th=[ 194], 90.00th=[ 213], 95.00th=[ 226], | 99.00th=[ 243], 99.50th=[ 249], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 255, max= 510, per=24.85%, avg=386.38, stdev=59.63, samples=52 iops : min= 63, max= 127, avg=96.23, stdev=14.93, samples=52 lat (msec) : 50=0.20%, 100=1.80%, 250=97.58%, 500=0.43% cpu : usr=0.17%, sys=0.93%, ctx=2252, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1555KiB/s (1592kB/s), 389KiB/s-390KiB/s (398kB/s-399kB/s), io=40.0MiB (41.9MB), run=26272-26338msec 0% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=189124: Mon Sep 21 07:00:09 2026 read: IOPS=37, BW=151KiB/s (155kB/s)(5048KiB/33414msec) slat (usec): min=2409, max=61944, avg=15356.27, stdev=8341.58 clat (msec): min=30, max=349, avg=197.29, stdev=46.06 lat (msec): min=41, max=366, avg=212.65, stdev=47.74 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 130], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 194], 60.00th=[ 205], | 70.00th=[ 220], 80.00th=[ 234], 90.00th=[ 259], 95.00th=[ 284], | 99.00th=[ 317], 99.50th=[ 330], 99.90th=[ 347], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 45, max= 231, per=24.91%, avg=149.45, stdev=33.91, samples=66 iops : min= 11, max= 57, avg=36.82, stdev= 8.46, samples=66 write: IOPS=38, BW=155KiB/s (159kB/s)(5192KiB/33414msec); 0 zone resets slat (usec): min=31, max=19814, avg=366.40, stdev=1187.98 clat (msec): min=30, max=364, avg=201.25, stdev=45.20 lat (msec): min=30, max=364, avg=201.61, stdev=45.22 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 138], 10.00th=[ 148], 20.00th=[ 163], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 207], | 70.00th=[ 220], 80.00th=[ 239], 90.00th=[ 266], 95.00th=[ 284], | 99.00th=[ 330], 99.50th=[ 334], 99.90th=[ 351], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 37, max= 246, per=24.69%, avg=154.30, stdev=39.63, samples=66 iops : min= 9, max= 61, avg=38.03, stdev= 9.90, samples=66 lat (msec) : 50=0.16%, 100=0.23%, 250=85.55%, 500=14.06% cpu : usr=0.27%, sys=1.82%, ctx=4465, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=189125: Mon Sep 21 07:00:09 2026 read: IOPS=37, BW=149KiB/s (153kB/s)(4976KiB/33399msec) slat (usec): min=2709, max=83038, avg=15451.49, stdev=8375.56 clat (msec): min=11, max=418, avg=196.61, stdev=47.46 lat (msec): min=25, max=433, avg=212.08, stdev=49.33 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 128], 10.00th=[ 140], 20.00th=[ 157], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 213], 80.00th=[ 234], 90.00th=[ 262], 95.00th=[ 279], | 99.00th=[ 330], 99.50th=[ 355], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 45, max= 247, per=24.60%, avg=147.58, stdev=39.89, samples=66 iops : min= 11, max= 61, avg=36.33, stdev= 9.98, samples=66 write: IOPS=39, BW=158KiB/s (161kB/s)(5264KiB/33399msec); 0 zone resets slat (usec): min=32, max=22653, avg=325.96, stdev=1144.89 clat (msec): min=46, max=417, avg=203.24, stdev=46.19 lat (msec): min=46, max=417, avg=203.57, stdev=46.20 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 138], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 211], | 70.00th=[ 224], 80.00th=[ 239], 90.00th=[ 262], 95.00th=[ 284], | 99.00th=[ 342], 99.50th=[ 372], 99.90th=[ 401], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 70, max= 255, per=25.16%, avg=157.27, stdev=38.68, samples=66 iops : min= 17, max= 63, avg=38.77, stdev= 9.69, samples=66 lat (msec) : 20=0.04%, 50=0.04%, 100=0.08%, 250=86.45%, 500=13.40% cpu : usr=0.36%, sys=2.01%, ctx=4462, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=189126: Mon Sep 21 07:00:09 2026 read: IOPS=37, BW=150KiB/s (153kB/s)(4984KiB/33303msec) slat (usec): min=2875, max=59821, avg=15117.08, stdev=7977.04 clat (msec): min=12, max=367, avg=194.69, stdev=44.78 lat (msec): min=25, max=380, avg=209.82, stdev=45.93 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 129], 10.00th=[ 142], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 213], 80.00th=[ 228], 90.00th=[ 257], 95.00th=[ 279], | 99.00th=[ 321], 99.50th=[ 342], 99.90th=[ 359], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 45, max= 253, per=24.87%, avg=149.23, stdev=37.32, samples=66 iops : min= 11, max= 63, avg=36.76, stdev= 9.39, samples=66 write: IOPS=39, BW=158KiB/s (162kB/s)(5256KiB/33303msec); 0 zone resets slat (usec): min=37, max=17232, avg=303.34, stdev=989.62 clat (msec): min=107, max=410, avg=203.04, stdev=46.19 lat (msec): min=107, max=410, avg=203.34, stdev=46.15 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 138], 10.00th=[ 148], 20.00th=[ 165], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 197], 60.00th=[ 209], | 70.00th=[ 224], 80.00th=[ 243], 90.00th=[ 264], 95.00th=[ 288], | 99.00th=[ 330], 99.50th=[ 342], 99.90th=[ 405], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 68, max= 229, per=25.09%, avg=156.80, stdev=39.21, samples=66 iops : min= 17, max= 57, avg=38.67, stdev= 9.78, samples=66 lat (msec) : 20=0.04%, 100=0.12%, 250=86.37%, 500=13.48% cpu : usr=0.30%, sys=2.02%, ctx=4536, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=189127: Mon Sep 21 07:00:09 2026 read: IOPS=38, BW=153KiB/s (156kB/s)(5060KiB/33137msec) slat (usec): min=72, max=65824, avg=15243.22, stdev=8303.96 clat (msec): min=26, max=381, avg=194.75, stdev=46.09 lat (msec): min=39, max=397, avg=210.01, stdev=47.41 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 126], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 218], 80.00th=[ 232], 90.00th=[ 251], 95.00th=[ 275], | 99.00th=[ 317], 99.50th=[ 342], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 94, max= 223, per=25.33%, avg=151.95, stdev=32.47, samples=65 iops : min= 23, max= 55, avg=37.45, stdev= 8.17, samples=65 write: IOPS=39, BW=156KiB/s (160kB/s)(5180KiB/33137msec); 0 zone resets slat (usec): min=31, max=12739, avg=252.33, stdev=729.13 clat (msec): min=24, max=404, avg=203.41, stdev=44.42 lat (msec): min=37, max=404, avg=203.67, stdev=44.37 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 142], 10.00th=[ 153], 20.00th=[ 165], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 199], 60.00th=[ 209], | 70.00th=[ 222], 80.00th=[ 239], 90.00th=[ 259], 95.00th=[ 284], | 99.00th=[ 334], 99.50th=[ 355], 99.90th=[ 380], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 88, max= 255, per=24.86%, avg=155.38, stdev=37.55, samples=65 iops : min= 22, max= 63, avg=38.29, stdev= 9.41, samples=65 lat (msec) : 50=0.31%, 100=0.16%, 250=87.70%, 500=11.84% cpu : usr=0.30%, sys=1.80%, ctx=4516, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=601KiB/s (615kB/s), 149KiB/s-153KiB/s (153kB/s-156kB/s), io=19.6MiB (20.5MB), run=33137-33414msec WRITE: bw=625KiB/s (640kB/s), 155KiB/s-158KiB/s (159kB/s-162kB/s), io=20.4MiB (21.4MB), run=33137-33414msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=189136: Mon Sep 21 07:00:20 2026 read: IOPS=1, BW=70.8MiB/s (74.2MB/s)(40.0MiB/565msec) slat (nsec): min=81784k, max=81784k, avg=81783763.00, stdev= 0.00 clat (nsec): min=461751k, max=461751k, avg=461751463.00, stdev= 0.00 lat (nsec): min=543542k, max=543542k, avg=543541646.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 464], 10.00th=[ 464], 20.00th=[ 464], | 30.00th=[ 464], 40.00th=[ 464], 50.00th=[ 464], 60.00th=[ 464], | 70.00th=[ 464], 80.00th=[ 464], 90.00th=[ 464], 95.00th=[ 464], | 99.00th=[ 464], 99.50th=[ 464], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.84%, ctx=13, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=70.8MiB/s (74.2MB/s), 70.8MiB/s-70.8MiB/s (74.2MB/s-74.2MB/s), io=40.0MiB (41.9MB), run=565-565msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout