== sanity test 398c: run fio to test AIO ================= 12:41:52 (1776271312) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.362264 s, 116 MB/s osc.lustre-OST0000-osc-ffff9bd0439b4000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148984: Wed Apr 15 12:42:12 2026 write: IOPS=179, BW=718KiB/s (735kB/s)(10.0MiB/14257msec); 0 zone resets slat (usec): min=20, max=7476, avg=153.01, stdev=405.95 clat (msec): min=17, max=173, avg=88.70, stdev=20.60 lat (msec): min=18, max=173, avg=88.85, stdev=20.60 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 59], 10.00th=[ 64], 20.00th=[ 73], | 30.00th=[ 80], 40.00th=[ 84], 50.00th=[ 89], 60.00th=[ 92], | 70.00th=[ 97], 80.00th=[ 104], 90.00th=[ 112], 95.00th=[ 125], | 99.00th=[ 150], 99.50th=[ 169], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 563, max= 896, per=24.81%, avg=712.64, stdev=82.73, samples=28 iops : min= 140, max= 224, avg=177.93, stdev=20.75, samples=28 lat (msec) : 20=0.04%, 50=2.27%, 100=72.66%, 250=25.04% cpu : usr=0.23%, sys=1.62%, ctx=909, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148985: Wed Apr 15 12:42:12 2026 write: IOPS=180, BW=720KiB/s (738kB/s)(10.0MiB/14213msec); 0 zone resets slat (usec): min=20, max=10422, avg=174.76, stdev=491.38 clat (msec): min=11, max=171, avg=88.42, stdev=21.12 lat (msec): min=12, max=171, avg=88.60, stdev=21.12 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 59], 10.00th=[ 64], 20.00th=[ 72], | 30.00th=[ 79], 40.00th=[ 84], 50.00th=[ 88], 60.00th=[ 92], | 70.00th=[ 97], 80.00th=[ 105], 90.00th=[ 114], 95.00th=[ 128], | 99.00th=[ 148], 99.50th=[ 155], 99.90th=[ 169], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 605, max= 904, per=25.06%, avg=719.64, stdev=70.70, samples=28 iops : min= 151, max= 226, avg=179.68, stdev=17.72, samples=28 lat (msec) : 20=0.66%, 50=1.37%, 100=73.20%, 250=24.77% cpu : usr=0.32%, sys=1.95%, ctx=1209, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148986: Wed Apr 15 12:42:12 2026 write: IOPS=180, BW=721KiB/s (739kB/s)(10.0MiB/14196msec); 0 zone resets slat (usec): min=20, max=7822, avg=170.27, stdev=398.28 clat (msec): min=10, max=173, avg=88.40, stdev=20.77 lat (msec): min=12, max=173, avg=88.57, stdev=20.77 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 71], | 30.00th=[ 79], 40.00th=[ 84], 50.00th=[ 88], 60.00th=[ 92], | 70.00th=[ 97], 80.00th=[ 105], 90.00th=[ 113], 95.00th=[ 122], | 99.00th=[ 148], 99.50th=[ 163], 99.90th=[ 171], 99.95th=[ 171], | 99.99th=[ 174] bw ( KiB/s): min= 524, max= 896, per=25.12%, avg=721.46, stdev=89.57, samples=28 iops : min= 131, max= 224, avg=180.14, stdev=22.45, samples=28 lat (msec) : 20=0.59%, 50=1.72%, 100=71.68%, 250=26.02% cpu : usr=0.27%, sys=1.86%, ctx=1235, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148987: Wed Apr 15 12:42:12 2026 write: IOPS=179, BW=719KiB/s (736kB/s)(10.0MiB/14246msec); 0 zone resets slat (usec): min=20, max=9070, avg=173.89, stdev=501.19 clat (msec): min=8, max=172, avg=88.77, stdev=20.76 lat (msec): min=9, max=172, avg=88.95, stdev=20.77 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 59], 10.00th=[ 64], 20.00th=[ 72], | 30.00th=[ 79], 40.00th=[ 83], 50.00th=[ 88], 60.00th=[ 93], | 70.00th=[ 97], 80.00th=[ 105], 90.00th=[ 114], 95.00th=[ 125], | 99.00th=[ 150], 99.50th=[ 155], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 588, max= 904, per=25.00%, avg=717.86, stdev=73.25, samples=28 iops : min= 147, max= 226, avg=179.21, stdev=18.39, samples=28 lat (msec) : 10=0.04%, 50=1.68%, 100=72.07%, 250=26.21% cpu : usr=0.34%, sys=1.97%, ctx=1265, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2873KiB/s (2942kB/s), 718KiB/s-721KiB/s (735kB/s-739kB/s), io=40.0MiB (41.9MB), run=14196-14257msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148999: Wed Apr 15 12:42:29 2026 read: IOPS=97, BW=389KiB/s (398kB/s)(5048KiB/12978msec) slat (usec): min=1157, max=24547, avg=5573.99, stdev=3208.93 clat (msec): min=2, max=158, avg=76.76, stdev=22.28 lat (msec): min=6, max=164, avg=82.35, stdev=23.20 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 43], 10.00th=[ 51], 20.00th=[ 59], | 30.00th=[ 65], 40.00th=[ 70], 50.00th=[ 75], 60.00th=[ 82], | 70.00th=[ 88], 80.00th=[ 94], 90.00th=[ 105], 95.00th=[ 116], | 99.00th=[ 136], 99.50th=[ 142], 99.90th=[ 157], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 255, max= 552, per=25.19%, avg=389.24, stdev=72.83, samples=25 iops : min= 63, max= 138, avg=97.00, stdev=18.19, samples=25 write: IOPS=100, BW=400KiB/s (410kB/s)(5192KiB/12978msec); 0 zone resets slat (usec): min=22, max=8241, avg=194.69, stdev=522.94 clat (msec): min=23, max=167, avg=78.78, stdev=22.48 lat (msec): min=23, max=167, avg=78.98, stdev=22.48 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 45], 10.00th=[ 52], 20.00th=[ 61], | 30.00th=[ 66], 40.00th=[ 72], 50.00th=[ 78], 60.00th=[ 84], | 70.00th=[ 89], 80.00th=[ 97], 90.00th=[ 107], 95.00th=[ 120], | 99.00th=[ 144], 99.50th=[ 146], 99.90th=[ 159], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 296, max= 607, per=25.02%, avg=402.32, stdev=87.69, samples=25 iops : min= 74, max= 151, avg=100.20, stdev=21.91, samples=25 lat (msec) : 4=0.08%, 20=0.08%, 50=9.06%, 100=75.51%, 250=15.27% cpu : usr=0.52%, sys=3.48%, ctx=2504, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149000: Wed Apr 15 12:42:29 2026 read: IOPS=95, BW=383KiB/s (392kB/s)(4976KiB/12987msec) slat (usec): min=1109, max=35673, avg=5625.96, stdev=3428.02 clat (msec): min=7, max=169, avg=76.46, stdev=22.09 lat (msec): min=12, max=173, avg=82.09, stdev=23.24 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 44], 10.00th=[ 51], 20.00th=[ 59], | 30.00th=[ 65], 40.00th=[ 69], 50.00th=[ 74], 60.00th=[ 80], | 70.00th=[ 86], 80.00th=[ 94], 90.00th=[ 105], 95.00th=[ 114], | 99.00th=[ 140], 99.50th=[ 150], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 247, max= 591, per=24.88%, avg=384.40, stdev=93.15, samples=25 iops : min= 61, max= 147, avg=95.76, stdev=23.20, samples=25 write: IOPS=101, BW=405KiB/s (415kB/s)(5264KiB/12987msec); 0 zone resets slat (usec): min=21, max=9580, avg=186.29, stdev=493.69 clat (msec): min=26, max=146, avg=79.35, stdev=20.63 lat (msec): min=26, max=146, avg=79.53, stdev=20.64 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 48], 10.00th=[ 54], 20.00th=[ 62], | 30.00th=[ 68], 40.00th=[ 73], 50.00th=[ 79], 60.00th=[ 84], | 70.00th=[ 90], 80.00th=[ 96], 90.00th=[ 106], 95.00th=[ 115], | 99.00th=[ 130], 99.50th=[ 136], 99.90th=[ 144], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 288, max= 560, per=25.30%, avg=406.80, stdev=70.26, samples=25 iops : min= 72, max= 140, avg=101.40, stdev=17.69, samples=25 lat (msec) : 10=0.04%, 50=8.44%, 100=76.84%, 250=14.69% cpu : usr=0.55%, sys=3.44%, ctx=2434, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149001: Wed Apr 15 12:42:29 2026 read: IOPS=95, BW=384KiB/s (393kB/s)(4984KiB/12983msec) slat (usec): min=898, max=21790, avg=5564.80, stdev=3113.79 clat (msec): min=14, max=153, avg=76.01, stdev=21.27 lat (msec): min=18, max=161, avg=81.59, stdev=22.34 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 42], 10.00th=[ 50], 20.00th=[ 58], | 30.00th=[ 65], 40.00th=[ 70], 50.00th=[ 75], 60.00th=[ 82], | 70.00th=[ 87], 80.00th=[ 95], 90.00th=[ 104], 95.00th=[ 110], | 99.00th=[ 129], 99.50th=[ 136], 99.90th=[ 153], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 239, max= 639, per=24.93%, avg=385.24, stdev=84.71, samples=25 iops : min= 59, max= 159, avg=96.00, stdev=21.15, samples=25 write: IOPS=101, BW=405KiB/s (415kB/s)(5256KiB/12983msec); 0 zone resets slat (usec): min=22, max=19783, avg=204.61, stdev=772.33 clat (msec): min=15, max=166, avg=79.77, stdev=21.11 lat (msec): min=15, max=166, avg=79.98, stdev=21.16 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 48], 10.00th=[ 55], 20.00th=[ 64], | 30.00th=[ 68], 40.00th=[ 73], 50.00th=[ 79], 60.00th=[ 84], | 70.00th=[ 89], 80.00th=[ 97], 90.00th=[ 107], 95.00th=[ 115], | 99.00th=[ 142], 99.50th=[ 150], 99.90th=[ 159], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 239, max= 626, per=25.07%, avg=403.20, stdev=92.67, samples=25 iops : min= 59, max= 156, avg=100.48, stdev=23.22, samples=25 lat (msec) : 20=0.12%, 50=8.20%, 100=76.33%, 250=15.35% cpu : usr=0.62%, sys=3.30%, ctx=2458, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149002: Wed Apr 15 12:42:29 2026 read: IOPS=98, BW=393KiB/s (403kB/s)(5060KiB/12866msec) slat (usec): min=43, max=30418, avg=5534.30, stdev=3295.25 clat (msec): min=11, max=168, avg=76.64, stdev=22.47 lat (msec): min=13, max=177, avg=82.18, stdev=23.50 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 43], 10.00th=[ 50], 20.00th=[ 59], | 30.00th=[ 65], 40.00th=[ 70], 50.00th=[ 75], 60.00th=[ 81], | 70.00th=[ 87], 80.00th=[ 94], 90.00th=[ 107], 95.00th=[ 116], | 99.00th=[ 138], 99.50th=[ 155], 99.90th=[ 167], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 288, max= 550, per=25.59%, avg=395.32, stdev=74.89, samples=25 iops : min= 72, max= 137, avg=98.52, stdev=18.71, samples=25 write: IOPS=100, BW=403KiB/s (412kB/s)(5180KiB/12866msec); 0 zone resets slat (usec): min=21, max=5569, avg=189.32, stdev=432.20 clat (msec): min=22, max=192, avg=78.23, stdev=22.37 lat (msec): min=22, max=192, avg=78.43, stdev=22.40 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 44], 10.00th=[ 52], 20.00th=[ 61], | 30.00th=[ 67], 40.00th=[ 72], 50.00th=[ 78], 60.00th=[ 84], | 70.00th=[ 89], 80.00th=[ 95], 90.00th=[ 105], 95.00th=[ 115], | 99.00th=[ 146], 99.50th=[ 155], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 247, max= 646, per=25.13%, avg=404.16, stdev=83.93, samples=25 iops : min= 61, max= 161, avg=100.72, stdev=21.02, samples=25 lat (msec) : 20=0.20%, 50=9.38%, 100=76.02%, 250=14.41% cpu : usr=0.43%, sys=3.57%, ctx=2457, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1545KiB/s (1582kB/s), 383KiB/s-393KiB/s (392kB/s-403kB/s), io=19.6MiB (20.5MB), run=12866-12987msec WRITE: bw=1609KiB/s (1647kB/s), 400KiB/s-405KiB/s (410kB/s-415kB/s), io=20.4MiB (21.4MB), run=12866-12987msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=149009: Wed Apr 15 12:42:36 2026 read: IOPS=2, BW=96.2MiB/s (101MB/s)(40.0MiB/416msec) slat (nsec): min=41665k, max=41665k, avg=41664589.00, stdev= 0.00 clat (nsec): min=346354k, max=346354k, avg=346353633.00, stdev= 0.00 lat (nsec): min=388024k, max=388024k, avg=388023534.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 347], | 30.00th=[ 347], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.89%, ctx=2, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=96.2MiB/s (101MB/s), 96.2MiB/s-96.2MiB/s (101MB/s-101MB/s), io=40.0MiB (41.9MB), run=416-416msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck