== sanity test 398c: run fio to test AIO ================= 23:36:50 (1789270610) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.505183 s, 83.0 MB/s osc.lustre-OST0000-osc-ffff8dc1b1af9000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=183827: Sat Sep 12 23:37:27 2026 write: IOPS=87, BW=350KiB/s (358kB/s)(10.0MiB/29256msec); 0 zone resets slat (usec): min=21, max=15383, avg=236.23, stdev=889.46 clat (msec): min=44, max=374, avg=182.01, stdev=54.20 lat (msec): min=44, max=374, avg=182.25, stdev=54.21 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 102], 10.00th=[ 114], 20.00th=[ 132], | 30.00th=[ 155], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 192], | 70.00th=[ 207], 80.00th=[ 224], 90.00th=[ 255], 95.00th=[ 284], | 99.00th=[ 326], 99.50th=[ 363], 99.90th=[ 376], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 173, max= 564, per=24.95%, avg=348.86, stdev=86.58, samples=57 iops : min= 43, max= 141, avg=86.81, stdev=21.70, samples=57 lat (msec) : 50=0.04%, 100=4.77%, 250=83.75%, 500=11.45% cpu : usr=0.15%, sys=0.87%, ctx=1428, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183828: Sat Sep 12 23:37:27 2026 write: IOPS=87, BW=352KiB/s (360kB/s)(10.0MiB/29111msec); 0 zone resets slat (usec): min=22, max=19541, avg=264.44, stdev=947.61 clat (msec): min=21, max=400, avg=181.47, stdev=54.57 lat (msec): min=22, max=405, avg=181.74, stdev=54.59 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 102], 10.00th=[ 114], 20.00th=[ 131], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 190], | 70.00th=[ 207], 80.00th=[ 224], 90.00th=[ 253], 95.00th=[ 284], | 99.00th=[ 326], 99.50th=[ 355], 99.90th=[ 397], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 144, max= 572, per=25.08%, avg=350.60, stdev=96.92, samples=57 iops : min= 36, max= 143, avg=87.23, stdev=24.28, samples=57 lat (msec) : 50=0.23%, 100=4.45%, 250=84.73%, 500=10.59% cpu : usr=0.26%, sys=1.15%, ctx=2192, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183829: Sat Sep 12 23:37:27 2026 write: IOPS=87, BW=350KiB/s (358kB/s)(10.0MiB/29285msec); 0 zone resets slat (usec): min=21, max=16482, avg=204.40, stdev=798.38 clat (msec): min=16, max=406, avg=181.66, stdev=55.94 lat (msec): min=16, max=406, avg=181.87, stdev=55.96 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 101], 10.00th=[ 112], 20.00th=[ 133], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 180], 60.00th=[ 190], | 70.00th=[ 207], 80.00th=[ 222], 90.00th=[ 257], 95.00th=[ 284], | 99.00th=[ 330], 99.50th=[ 397], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 159, max= 532, per=24.88%, avg=347.88, stdev=88.43, samples=57 iops : min= 39, max= 133, avg=86.56, stdev=22.17, samples=57 lat (msec) : 20=0.04%, 50=0.04%, 100=4.84%, 250=83.75%, 500=11.33% cpu : usr=0.22%, sys=0.94%, ctx=1335, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183830: Sat Sep 12 23:37:27 2026 write: IOPS=87, BW=350KiB/s (358kB/s)(10.0MiB/29291msec); 0 zone resets slat (usec): min=22, max=21520, avg=203.81, stdev=833.36 clat (msec): min=51, max=404, avg=181.89, stdev=54.68 lat (msec): min=51, max=404, avg=182.09, stdev=54.71 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 104], 10.00th=[ 114], 20.00th=[ 132], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 209], 80.00th=[ 224], 90.00th=[ 257], 95.00th=[ 279], | 99.00th=[ 342], 99.50th=[ 388], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 143, max= 512, per=24.97%, avg=349.04, stdev=96.71, samples=57 iops : min= 35, max= 128, avg=86.86, stdev=24.24, samples=57 lat (msec) : 100=4.30%, 250=84.84%, 500=10.86% cpu : usr=0.18%, sys=1.01%, ctx=1583, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1398KiB/s (1432kB/s), 350KiB/s-352KiB/s (358kB/s-360kB/s), io=40.0MiB (41.9MB), run=29111-29291msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=183844: Sat Sep 12 23:38:12 2026 read: IOPS=35, BW=141KiB/s (144kB/s)(5048KiB/35820msec) slat (msec): min=3, max=304, avg=16.72, stdev=13.00 clat (msec): min=23, max=573, avg=211.47, stdev=71.32 lat (msec): min=44, max=587, avg=228.21, stdev=74.28 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 120], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 230], 80.00th=[ 253], 90.00th=[ 300], 95.00th=[ 359], | 99.00th=[ 472], 99.50th=[ 498], 99.90th=[ 542], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 56, max= 231, per=25.02%, avg=140.13, stdev=38.73, samples=70 iops : min= 14, max= 57, avg=34.56, stdev= 9.68, samples=70 write: IOPS=36, BW=145KiB/s (148kB/s)(5192KiB/35820msec); 0 zone resets slat (usec): min=32, max=28503, avg=309.70, stdev=1205.38 clat (msec): min=52, max=540, avg=215.71, stdev=66.60 lat (msec): min=53, max=540, avg=216.02, stdev=66.63 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 127], 10.00th=[ 140], 20.00th=[ 165], | 30.00th=[ 182], 40.00th=[ 194], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 234], 80.00th=[ 257], 90.00th=[ 300], 95.00th=[ 342], | 99.00th=[ 464], 99.50th=[ 489], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 39, max= 269, per=24.83%, avg=144.76, stdev=46.87, samples=70 iops : min= 9, max= 67, avg=35.69, stdev=11.73, samples=70 lat (msec) : 50=0.04%, 100=1.68%, 250=76.21%, 500=21.68%, 750=0.39% cpu : usr=0.34%, sys=1.94%, ctx=4161, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183845: Sat Sep 12 23:38:12 2026 read: IOPS=35, BW=140KiB/s (144kB/s)(4976KiB/35499msec) slat (usec): min=3573, max=71197, avg=16357.53, stdev=9191.37 clat (msec): min=16, max=621, avg=210.42, stdev=66.59 lat (msec): min=22, max=643, avg=226.79, stdev=69.37 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 121], 10.00th=[ 138], 20.00th=[ 161], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 213], | 70.00th=[ 232], 80.00th=[ 259], 90.00th=[ 300], 95.00th=[ 334], | 99.00th=[ 401], 99.50th=[ 464], 99.90th=[ 550], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 46, max= 238, per=24.91%, avg=139.49, stdev=44.26, samples=70 iops : min= 11, max= 59, avg=34.39, stdev=11.06, samples=70 write: IOPS=37, BW=148KiB/s (152kB/s)(5264KiB/35499msec); 0 zone resets slat (usec): min=25, max=26395, avg=357.23, stdev=1341.21 clat (msec): min=54, max=567, avg=213.66, stdev=63.57 lat (msec): min=54, max=567, avg=214.02, stdev=63.67 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 120], 10.00th=[ 140], 20.00th=[ 167], | 30.00th=[ 182], 40.00th=[ 194], 50.00th=[ 205], 60.00th=[ 218], | 70.00th=[ 234], 80.00th=[ 257], 90.00th=[ 300], 95.00th=[ 330], | 99.00th=[ 409], 99.50th=[ 439], 99.90th=[ 523], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 47, max= 293, per=25.40%, avg=148.10, stdev=46.61, samples=70 iops : min= 11, max= 73, avg=36.56, stdev=11.68, samples=70 lat (msec) : 20=0.04%, 50=0.04%, 100=1.21%, 250=75.70%, 500=22.77% lat (msec) : 750=0.23% cpu : usr=0.35%, sys=2.01%, ctx=3927, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183846: Sat Sep 12 23:38:12 2026 read: IOPS=35, BW=142KiB/s (145kB/s)(4984KiB/35222msec) slat (usec): min=37, max=77217, avg=16563.30, stdev=9829.14 clat (msec): min=20, max=645, avg=210.06, stdev=68.68 lat (msec): min=20, max=662, avg=226.63, stdev=71.61 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 123], 10.00th=[ 138], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 234], 80.00th=[ 259], 90.00th=[ 296], 95.00th=[ 330], | 99.00th=[ 451], 99.50th=[ 489], 99.90th=[ 617], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 79, max= 215, per=25.35%, avg=141.97, stdev=37.67, samples=69 iops : min= 19, max= 53, avg=35.00, stdev= 9.41, samples=69 write: IOPS=37, BW=149KiB/s (153kB/s)(5256KiB/35222msec); 0 zone resets slat (usec): min=28, max=17331, avg=325.20, stdev=1062.74 clat (msec): min=73, max=619, avg=212.95, stdev=62.02 lat (msec): min=73, max=619, avg=213.28, stdev=62.05 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 128], 10.00th=[ 144], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 205], 60.00th=[ 220], | 70.00th=[ 234], 80.00th=[ 257], 90.00th=[ 296], 95.00th=[ 330], | 99.00th=[ 397], 99.50th=[ 426], 99.90th=[ 592], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 55, max= 279, per=25.55%, avg=148.99, stdev=46.81, samples=69 iops : min= 13, max= 69, avg=36.75, stdev=11.74, samples=69 lat (msec) : 50=0.23%, 100=1.09%, 250=75.90%, 500=22.50%, 750=0.27% cpu : usr=0.31%, sys=2.09%, ctx=4426, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183848: Sat Sep 12 23:38:12 2026 read: IOPS=35, BW=142KiB/s (146kB/s)(5060KiB/35577msec) slat (usec): min=3336, max=70162, avg=16558.85, stdev=9876.00 clat (msec): min=31, max=473, avg=209.65, stdev=64.65 lat (msec): min=35, max=493, avg=226.22, stdev=67.95 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 118], 10.00th=[ 136], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 232], 80.00th=[ 257], 90.00th=[ 300], 95.00th=[ 334], | 99.00th=[ 401], 99.50th=[ 409], 99.90th=[ 451], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 55, max= 230, per=25.38%, avg=142.10, stdev=40.79, samples=70 iops : min= 13, max= 57, avg=35.04, stdev=10.20, samples=70 write: IOPS=36, BW=146KiB/s (149kB/s)(5180KiB/35577msec); 0 zone resets slat (usec): min=35, max=15350, avg=303.43, stdev=961.54 clat (msec): min=39, max=480, avg=215.21, stdev=63.56 lat (msec): min=39, max=480, avg=215.51, stdev=63.62 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 125], 10.00th=[ 142], 20.00th=[ 167], | 30.00th=[ 182], 40.00th=[ 192], 50.00th=[ 205], 60.00th=[ 220], | 70.00th=[ 236], 80.00th=[ 259], 90.00th=[ 309], 95.00th=[ 338], | 99.00th=[ 397], 99.50th=[ 439], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 39, max= 293, per=24.95%, avg=145.49, stdev=42.97, samples=70 iops : min= 9, max= 73, avg=35.91, stdev=10.77, samples=70 lat (msec) : 50=0.12%, 100=1.37%, 250=75.86%, 500=22.66% cpu : usr=0.26%, sys=2.15%, ctx=3972, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=560KiB/s (574kB/s), 140KiB/s-142KiB/s (144kB/s-146kB/s), io=19.6MiB (20.5MB), run=35222-35820msec WRITE: bw=583KiB/s (597kB/s), 145KiB/s-149KiB/s (148kB/s-153kB/s), io=20.4MiB (21.4MB), run=35222-35820msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=183857: Sat Sep 12 23:38:25 2026 read: IOPS=1, BW=72.1MiB/s (75.6MB/s)(40.0MiB/555msec) slat (nsec): min=40190k, max=40190k, avg=40190465.00, stdev= 0.00 clat (nsec): min=470265k, max=470265k, avg=470264748.00, stdev= 0.00 lat (nsec): min=510461k, max=510461k, avg=510460933.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 472], 10.00th=[ 472], 20.00th=[ 472], | 30.00th=[ 472], 40.00th=[ 472], 50.00th=[ 472], 60.00th=[ 472], | 70.00th=[ 472], 80.00th=[ 472], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.89%, ctx=3, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=72.1MiB/s (75.6MB/s), 72.1MiB/s-72.1MiB/s (75.6MB/s-75.6MB/s), io=40.0MiB (41.9MB), run=555-555msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout