== sanity test 398c: run fio to test AIO ================= 15:09:43 (1784228983) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.237302 s, 177 MB/s osc.lustre-OST0000-osc-ffff92d00967c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=183152: Thu Jul 16 15:10:01 2026 write: IOPS=197, BW=790KiB/s (809kB/s)(10.0MiB/12965msec); 0 zone resets slat (usec): min=33, max=30800, avg=206.63, stdev=675.26 clat (msec): min=10, max=143, avg=80.43, stdev=18.08 lat (msec): min=10, max=143, avg=80.64, stdev=18.05 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 50], 10.00th=[ 58], 20.00th=[ 67], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 82], 60.00th=[ 86], | 70.00th=[ 90], 80.00th=[ 95], 90.00th=[ 103], 95.00th=[ 110], | 99.00th=[ 125], 99.50th=[ 132], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 144] bw ( KiB/s): min= 628, max= 1135, per=24.86%, avg=784.28, stdev=107.98, samples=25 iops : min= 157, max= 283, avg=195.88, stdev=26.90, samples=25 lat (msec) : 20=0.16%, 50=5.12%, 100=82.15%, 250=12.58% cpu : usr=0.27%, sys=2.74%, ctx=2796, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183153: Thu Jul 16 15:10:01 2026 write: IOPS=197, BW=789KiB/s (808kB/s)(10.0MiB/12978msec); 0 zone resets slat (usec): min=36, max=6555, avg=207.73, stdev=299.13 clat (msec): min=13, max=141, avg=80.46, stdev=18.15 lat (msec): min=13, max=141, avg=80.66, stdev=18.16 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 50], 10.00th=[ 58], 20.00th=[ 67], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 82], 60.00th=[ 85], | 70.00th=[ 90], 80.00th=[ 95], 90.00th=[ 103], 95.00th=[ 110], | 99.00th=[ 124], 99.50th=[ 131], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 142] bw ( KiB/s): min= 588, max= 1156, per=24.78%, avg=781.92, stdev=111.76, samples=25 iops : min= 147, max= 289, avg=195.32, stdev=27.96, samples=25 lat (msec) : 20=0.20%, 50=4.96%, 100=82.97%, 250=11.88% cpu : usr=0.31%, sys=2.91%, ctx=2902, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183154: Thu Jul 16 15:10:01 2026 write: IOPS=198, BW=793KiB/s (812kB/s)(10.0MiB/12914msec); 0 zone resets slat (usec): min=34, max=12354, avg=177.09, stdev=317.59 clat (msec): min=18, max=142, avg=80.44, stdev=18.11 lat (msec): min=18, max=142, avg=80.62, stdev=18.12 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 50], 10.00th=[ 59], 20.00th=[ 66], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 82], 60.00th=[ 85], | 70.00th=[ 90], 80.00th=[ 95], 90.00th=[ 103], 95.00th=[ 110], | 99.00th=[ 129], 99.50th=[ 134], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 144] bw ( KiB/s): min= 588, max= 1238, per=24.92%, avg=786.16, stdev=127.82, samples=25 iops : min= 147, max= 309, avg=196.36, stdev=31.90, samples=25 lat (msec) : 20=0.04%, 50=6.52%, 100=81.48%, 250=11.95% cpu : usr=0.25%, sys=2.22%, ctx=2595, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183155: Thu Jul 16 15:10:01 2026 write: IOPS=197, BW=789KiB/s (808kB/s)(10.0MiB/12979msec); 0 zone resets slat (usec): min=33, max=3768, avg=179.94, stdev=240.93 clat (msec): min=7, max=139, avg=80.44, stdev=18.09 lat (msec): min=8, max=139, avg=80.63, stdev=18.08 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 47], 10.00th=[ 59], 20.00th=[ 66], | 30.00th=[ 71], 40.00th=[ 77], 50.00th=[ 82], 60.00th=[ 85], | 70.00th=[ 90], 80.00th=[ 96], 90.00th=[ 103], 95.00th=[ 109], | 99.00th=[ 125], 99.50th=[ 138], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 620, max= 1064, per=24.71%, avg=779.56, stdev=106.79, samples=25 iops : min= 155, max= 266, avg=194.76, stdev=26.69, samples=25 lat (msec) : 10=0.04%, 50=5.31%, 100=82.23%, 250=12.42% cpu : usr=0.27%, sys=2.23%, ctx=2698, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3156KiB/s (3232kB/s), 789KiB/s-793KiB/s (808kB/s-812kB/s), io=40.0MiB (41.9MB), run=12914-12979msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=183167: Thu Jul 16 15:10:17 2026 read: IOPS=96, BW=388KiB/s (397kB/s)(5048KiB/13017msec) slat (usec): min=1474, max=20792, avg=5384.00, stdev=2421.96 clat (msec): min=30, max=131, avg=76.66, stdev=14.97 lat (msec): min=36, max=137, avg=82.05, stdev=15.31 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 55], 10.00th=[ 59], 20.00th=[ 64], | 30.00th=[ 68], 40.00th=[ 72], 50.00th=[ 77], 60.00th=[ 80], | 70.00th=[ 85], 80.00th=[ 90], 90.00th=[ 96], 95.00th=[ 102], | 99.00th=[ 117], 99.50th=[ 122], 99.90th=[ 130], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 288, max= 493, per=25.19%, avg=388.12, stdev=50.30, samples=25 iops : min= 72, max= 123, avg=96.80, stdev=12.49, samples=25 write: IOPS=99, BW=399KiB/s (408kB/s)(5192KiB/13017msec); 0 zone resets slat (usec): min=43, max=4558, avg=215.13, stdev=349.97 clat (msec): min=38, max=119, avg=79.41, stdev=13.52 lat (msec): min=39, max=120, avg=79.63, stdev=13.52 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 59], 10.00th=[ 63], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 86], 80.00th=[ 91], 90.00th=[ 99], 95.00th=[ 105], | 99.00th=[ 114], 99.50th=[ 115], 99.90th=[ 120], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 288, max= 480, per=24.91%, avg=399.56, stdev=41.91, samples=25 iops : min= 72, max= 120, avg=99.68, stdev=10.46, samples=25 lat (msec) : 50=1.45%, 100=91.45%, 250=7.11% cpu : usr=0.37%, sys=3.33%, ctx=4824, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183168: Thu Jul 16 15:10:17 2026 read: IOPS=95, BW=384KiB/s (393kB/s)(4976KiB/12959msec) slat (usec): min=56, max=19263, avg=5417.90, stdev=2445.86 clat (msec): min=9, max=164, avg=76.47, stdev=16.41 lat (msec): min=9, max=174, avg=81.89, stdev=16.75 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 53], 10.00th=[ 57], 20.00th=[ 63], | 30.00th=[ 68], 40.00th=[ 72], 50.00th=[ 77], 60.00th=[ 81], | 70.00th=[ 85], 80.00th=[ 89], 90.00th=[ 96], 95.00th=[ 104], | 99.00th=[ 120], 99.50th=[ 123], 99.90th=[ 163], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min= 264, max= 472, per=24.97%, avg=384.72, stdev=57.57, samples=25 iops : min= 66, max= 118, avg=95.96, stdev=14.28, samples=25 write: IOPS=101, BW=406KiB/s (416kB/s)(5264KiB/12959msec); 0 zone resets slat (usec): min=38, max=4923, avg=188.43, stdev=258.01 clat (msec): min=17, max=135, avg=79.72, stdev=13.93 lat (msec): min=17, max=136, avg=79.90, stdev=13.94 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 59], 10.00th=[ 64], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 87], 80.00th=[ 92], 90.00th=[ 99], 95.00th=[ 104], | 99.00th=[ 113], 99.50th=[ 117], 99.90th=[ 125], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 319, max= 508, per=25.35%, avg=406.68, stdev=57.46, samples=25 iops : min= 79, max= 127, avg=101.48, stdev=14.46, samples=25 lat (msec) : 10=0.04%, 20=0.23%, 50=1.52%, 100=90.74%, 250=7.46% cpu : usr=0.32%, sys=3.30%, ctx=4695, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183169: Thu Jul 16 15:10:17 2026 read: IOPS=96, BW=385KiB/s (394kB/s)(4984KiB/12949msec) slat (usec): min=1528, max=20748, avg=5402.17, stdev=2474.95 clat (msec): min=10, max=146, avg=76.36, stdev=15.71 lat (msec): min=14, max=150, avg=81.77, stdev=16.04 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 53], 10.00th=[ 58], 20.00th=[ 64], | 30.00th=[ 68], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 80], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 96], 95.00th=[ 105], | 99.00th=[ 117], 99.50th=[ 123], 99.90th=[ 138], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 288, max= 496, per=25.10%, avg=386.80, stdev=53.28, samples=25 iops : min= 72, max= 124, avg=96.48, stdev=13.41, samples=25 write: IOPS=101, BW=406KiB/s (416kB/s)(5256KiB/12949msec); 0 zone resets slat (usec): min=36, max=5190, avg=207.56, stdev=330.04 clat (msec): min=37, max=133, avg=79.14, stdev=13.60 lat (msec): min=37, max=133, avg=79.35, stdev=13.59 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 83], | 70.00th=[ 86], 80.00th=[ 90], 90.00th=[ 97], 95.00th=[ 103], | 99.00th=[ 112], 99.50th=[ 120], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 288, max= 488, per=25.26%, avg=405.20, stdev=47.02, samples=25 iops : min= 72, max= 122, avg=101.08, stdev=11.62, samples=25 lat (msec) : 20=0.08%, 50=1.91%, 100=90.94%, 250=7.07% cpu : usr=0.34%, sys=3.31%, ctx=4870, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183170: Thu Jul 16 15:10:17 2026 read: IOPS=97, BW=389KiB/s (398kB/s)(5060KiB/13021msec) slat (usec): min=1528, max=18035, avg=5365.88, stdev=2441.97 clat (msec): min=11, max=129, avg=76.57, stdev=15.14 lat (msec): min=15, max=137, avg=81.94, stdev=15.41 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 54], 10.00th=[ 58], 20.00th=[ 64], | 30.00th=[ 69], 40.00th=[ 72], 50.00th=[ 77], 60.00th=[ 81], | 70.00th=[ 84], 80.00th=[ 88], 90.00th=[ 97], 95.00th=[ 104], | 99.00th=[ 115], 99.50th=[ 120], 99.90th=[ 126], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 280, max= 480, per=25.16%, avg=387.72, stdev=49.93, samples=25 iops : min= 70, max= 120, avg=96.72, stdev=12.47, samples=25 write: IOPS=99, BW=398KiB/s (407kB/s)(5180KiB/13021msec); 0 zone resets slat (usec): min=41, max=5220, avg=205.10, stdev=289.35 clat (msec): min=33, max=128, avg=79.67, stdev=14.15 lat (msec): min=33, max=128, avg=79.88, stdev=14.16 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 58], 10.00th=[ 62], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 87], 80.00th=[ 92], 90.00th=[ 100], 95.00th=[ 105], | 99.00th=[ 113], 99.50th=[ 116], 99.90th=[ 126], 99.95th=[ 129], | 99.99th=[ 129] bw ( KiB/s): min= 320, max= 504, per=24.68%, avg=395.92, stdev=45.85, samples=25 iops : min= 80, max= 126, avg=98.76, stdev=11.41, samples=25 lat (msec) : 20=0.04%, 50=1.99%, 100=89.88%, 250=8.09% cpu : usr=0.35%, sys=3.26%, ctx=4785, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1541KiB/s (1578kB/s), 384KiB/s-389KiB/s (393kB/s-398kB/s), io=19.6MiB (20.5MB), run=12949-13021msec WRITE: bw=1604KiB/s (1643kB/s), 398KiB/s-406KiB/s (407kB/s-416kB/s), io=20.4MiB (21.4MB), run=12949-13021msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=183176: Thu Jul 16 15:10:21 2026 read: IOPS=5, BW=214MiB/s (224MB/s)(40.0MiB/187msec) slat (nsec): min=16953k, max=16953k, avg=16952903.00, stdev= 0.00 clat (nsec): min=160762k, max=160762k, avg=160761591.00, stdev= 0.00 lat (nsec): min=177719k, max=177719k, avg=177719104.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 161], 10.00th=[ 161], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 161], 50.00th=[ 161], 60.00th=[ 161], | 70.00th=[ 161], 80.00th=[ 161], 90.00th=[ 161], 95.00th=[ 161], | 99.00th=[ 161], 99.50th=[ 161], 99.90th=[ 161], 99.95th=[ 161], | 99.99th=[ 161] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.45%, ctx=9, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=214MiB/s (224MB/s), 214MiB/s-214MiB/s (224MB/s-224MB/s), io=40.0MiB (41.9MB), run=187-187msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout