== sanity test 398c: run fio to test AIO ================= 20:41:46 (1772761306) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.230682 s, 182 MB/s osc.lustre-OST0000-osc-ffff8c4b88002800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=192669: Thu Mar 5 20:42:10 2026 write: IOPS=140, BW=561KiB/s (575kB/s)(10.0MiB/18248msec); 0 zone resets slat (usec): min=40, max=8539, avg=292.38, stdev=420.78 clat (msec): min=21, max=197, avg=113.03, stdev=19.22 lat (msec): min=21, max=197, avg=113.32, stdev=19.23 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 82], 10.00th=[ 89], 20.00th=[ 97], | 30.00th=[ 104], 40.00th=[ 109], 50.00th=[ 113], 60.00th=[ 118], | 70.00th=[ 124], 80.00th=[ 129], 90.00th=[ 136], 95.00th=[ 144], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 199], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 439, max= 766, per=24.93%, avg=559.50, stdev=62.77, samples=36 iops : min= 109, max= 191, avg=139.56, stdev=15.69, samples=36 lat (msec) : 50=0.20%, 100=23.87%, 250=75.94% cpu : usr=0.35%, sys=2.37%, ctx=2589, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=192670: Thu Mar 5 20:42:10 2026 write: IOPS=140, BW=563KiB/s (577kB/s)(10.0MiB/18174msec); 0 zone resets slat (usec): min=39, max=7804, avg=323.03, stdev=484.12 clat (msec): min=9, max=190, avg=113.15, stdev=19.23 lat (msec): min=11, max=190, avg=113.47, stdev=19.22 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 83], 10.00th=[ 89], 20.00th=[ 99], | 30.00th=[ 104], 40.00th=[ 109], 50.00th=[ 113], 60.00th=[ 117], | 70.00th=[ 123], 80.00th=[ 129], 90.00th=[ 138], 95.00th=[ 146], | 99.00th=[ 159], 99.50th=[ 169], 99.90th=[ 190], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min= 455, max= 742, per=24.97%, avg=560.33, stdev=65.92, samples=36 iops : min= 113, max= 185, avg=139.75, stdev=16.52, samples=36 lat (msec) : 10=0.04%, 20=0.04%, 50=0.08%, 100=24.80%, 250=75.04% cpu : usr=0.36%, sys=2.70%, ctx=2795, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=192671: Thu Mar 5 20:42:10 2026 write: IOPS=140, BW=562KiB/s (575kB/s)(10.0MiB/18226msec); 0 zone resets slat (usec): min=39, max=26925, avg=348.45, stdev=704.60 clat (msec): min=10, max=191, avg=113.02, stdev=19.34 lat (msec): min=10, max=192, avg=113.37, stdev=19.31 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 84], 10.00th=[ 89], 20.00th=[ 99], | 30.00th=[ 104], 40.00th=[ 108], 50.00th=[ 112], 60.00th=[ 117], | 70.00th=[ 123], 80.00th=[ 130], 90.00th=[ 138], 95.00th=[ 146], | 99.00th=[ 157], 99.50th=[ 174], 99.90th=[ 182], 99.95th=[ 184], | 99.99th=[ 192] bw ( KiB/s): min= 477, max= 758, per=25.00%, avg=560.89, stdev=60.60, samples=36 iops : min= 119, max= 189, avg=139.89, stdev=15.16, samples=36 lat (msec) : 20=0.12%, 50=0.12%, 100=23.63%, 250=76.13% cpu : usr=0.36%, sys=2.91%, ctx=2930, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=192672: Thu Mar 5 20:42:10 2026 write: IOPS=140, BW=561KiB/s (575kB/s)(10.0MiB/18249msec); 0 zone resets slat (usec): min=42, max=8445, avg=304.74, stdev=457.05 clat (msec): min=22, max=188, avg=113.11, stdev=19.03 lat (msec): min=23, max=188, avg=113.42, stdev=19.05 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 83], 10.00th=[ 89], 20.00th=[ 99], | 30.00th=[ 104], 40.00th=[ 108], 50.00th=[ 112], 60.00th=[ 117], | 70.00th=[ 123], 80.00th=[ 130], 90.00th=[ 138], 95.00th=[ 146], | 99.00th=[ 157], 99.50th=[ 165], 99.90th=[ 178], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 464, max= 766, per=24.93%, avg=559.50, stdev=62.96, samples=36 iops : min= 116, max= 191, avg=139.56, stdev=15.75, samples=36 lat (msec) : 50=0.04%, 100=23.79%, 250=76.17% cpu : usr=0.36%, sys=2.60%, ctx=2721, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2245KiB/s (2298kB/s), 561KiB/s-563KiB/s (575kB/s-577kB/s), io=40.0MiB (41.9MB), run=18174-18249msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=192684: Thu Mar 5 20:42:31 2026 read: IOPS=73, BW=295KiB/s (303kB/s)(5048KiB/17086msec) slat (usec): min=67, max=24979, avg=7032.83, stdev=2918.61 clat (msec): min=55, max=199, avg=101.44, stdev=20.54 lat (msec): min=61, max=203, avg=108.48, stdev=21.19 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 72], 10.00th=[ 78], 20.00th=[ 85], | 30.00th=[ 90], 40.00th=[ 95], 50.00th=[ 100], 60.00th=[ 105], | 70.00th=[ 111], 80.00th=[ 118], 90.00th=[ 129], 95.00th=[ 138], | 99.00th=[ 159], 99.50th=[ 167], 99.90th=[ 188], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 200, max= 398, per=25.26%, avg=296.00, stdev=51.36, samples=33 iops : min= 50, max= 99, avg=73.58, stdev=12.74, samples=33 write: IOPS=75, BW=304KiB/s (311kB/s)(5192KiB/17086msec); 0 zone resets slat (usec): min=41, max=5777, avg=332.75, stdev=448.37 clat (msec): min=61, max=208, avg=103.83, stdev=18.59 lat (msec): min=61, max=208, avg=104.17, stdev=18.59 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 75], 10.00th=[ 82], 20.00th=[ 88], | 30.00th=[ 93], 40.00th=[ 99], 50.00th=[ 103], 60.00th=[ 108], | 70.00th=[ 113], 80.00th=[ 120], 90.00th=[ 129], 95.00th=[ 136], | 99.00th=[ 153], 99.50th=[ 159], 99.90th=[ 203], 99.95th=[ 209], | 99.99th=[ 209] bw ( KiB/s): min= 216, max= 430, per=25.04%, avg=305.52, stdev=44.87, samples=33 iops : min= 54, max= 107, avg=75.94, stdev=11.15, samples=33 lat (msec) : 100=47.42%, 250=52.58% cpu : usr=0.60%, sys=3.68%, ctx=4693, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=192685: Thu Mar 5 20:42:31 2026 read: IOPS=73, BW=292KiB/s (299kB/s)(4976KiB/17022msec) slat (usec): min=39, max=25994, avg=7027.43, stdev=2955.05 clat (msec): min=31, max=214, avg=101.17, stdev=20.37 lat (msec): min=37, max=222, avg=108.20, stdev=21.04 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 71], 10.00th=[ 78], 20.00th=[ 85], | 30.00th=[ 90], 40.00th=[ 95], 50.00th=[ 100], 60.00th=[ 105], | 70.00th=[ 111], 80.00th=[ 118], 90.00th=[ 127], 95.00th=[ 136], | 99.00th=[ 159], 99.50th=[ 165], 99.90th=[ 209], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 214, max= 407, per=25.02%, avg=293.27, stdev=46.53, samples=33 iops : min= 53, max= 101, avg=72.88, stdev=11.61, samples=33 write: IOPS=77, BW=309KiB/s (317kB/s)(5264KiB/17022msec); 0 zone resets slat (usec): min=47, max=4906, avg=332.95, stdev=425.00 clat (msec): min=46, max=170, avg=104.00, stdev=18.22 lat (msec): min=47, max=170, avg=104.33, stdev=18.23 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 77], 10.00th=[ 81], 20.00th=[ 89], | 30.00th=[ 94], 40.00th=[ 99], 50.00th=[ 104], 60.00th=[ 109], | 70.00th=[ 113], 80.00th=[ 120], 90.00th=[ 129], 95.00th=[ 136], | 99.00th=[ 153], 99.50th=[ 155], 99.90th=[ 159], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min= 232, max= 422, per=25.40%, avg=309.85, stdev=47.22, samples=33 iops : min= 58, max= 105, avg=77.03, stdev=11.67, samples=33 lat (msec) : 50=0.12%, 100=47.66%, 250=52.23% cpu : usr=0.42%, sys=3.95%, ctx=4617, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=192686: Thu Mar 5 20:42:31 2026 read: IOPS=72, BW=291KiB/s (298kB/s)(4984KiB/17119msec) slat (usec): min=80, max=23773, avg=7200.92, stdev=2899.91 clat (msec): min=16, max=179, avg=101.35, stdev=20.08 lat (msec): min=16, max=186, avg=108.56, stdev=20.82 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 72], 10.00th=[ 77], 20.00th=[ 84], | 30.00th=[ 90], 40.00th=[ 96], 50.00th=[ 101], 60.00th=[ 106], | 70.00th=[ 111], 80.00th=[ 120], 90.00th=[ 129], 95.00th=[ 136], | 99.00th=[ 150], 99.50th=[ 161], 99.90th=[ 176], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 166, max= 438, per=25.00%, avg=293.03, stdev=53.84, samples=33 iops : min= 41, max= 109, avg=72.82, stdev=13.36, samples=33 write: IOPS=76, BW=307KiB/s (314kB/s)(5256KiB/17119msec); 0 zone resets slat (usec): min=59, max=6286, avg=343.35, stdev=484.99 clat (msec): min=42, max=181, avg=104.14, stdev=17.97 lat (msec): min=42, max=181, avg=104.49, stdev=17.97 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 78], 10.00th=[ 84], 20.00th=[ 89], | 30.00th=[ 94], 40.00th=[ 99], 50.00th=[ 103], 60.00th=[ 108], | 70.00th=[ 113], 80.00th=[ 120], 90.00th=[ 129], 95.00th=[ 138], | 99.00th=[ 148], 99.50th=[ 153], 99.90th=[ 178], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 223, max= 447, per=25.09%, avg=306.15, stdev=53.85, samples=33 iops : min= 55, max= 111, avg=76.09, stdev=13.44, samples=33 lat (msec) : 20=0.04%, 50=0.12%, 100=46.29%, 250=53.55% cpu : usr=0.47%, sys=3.81%, ctx=4747, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=192687: Thu Mar 5 20:42:31 2026 read: IOPS=73, BW=296KiB/s (303kB/s)(5060KiB/17115msec) slat (usec): min=2298, max=24065, avg=7110.14, stdev=2861.92 clat (msec): min=29, max=169, avg=100.02, stdev=19.33 lat (msec): min=41, max=176, avg=107.14, stdev=20.07 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 72], 10.00th=[ 77], 20.00th=[ 84], | 30.00th=[ 89], 40.00th=[ 94], 50.00th=[ 100], 60.00th=[ 104], | 70.00th=[ 109], 80.00th=[ 117], 90.00th=[ 128], 95.00th=[ 136], | 99.00th=[ 150], 99.50th=[ 155], 99.90th=[ 165], 99.95th=[ 169], | 99.99th=[ 169] bw ( KiB/s): min= 160, max= 390, per=25.24%, avg=295.76, stdev=51.65, samples=33 iops : min= 40, max= 97, avg=73.52, stdev=12.82, samples=33 write: IOPS=75, BW=303KiB/s (310kB/s)(5180KiB/17115msec); 0 zone resets slat (usec): min=45, max=11820, avg=322.57, stdev=496.62 clat (msec): min=51, max=179, avg=104.90, stdev=18.27 lat (msec): min=52, max=179, avg=105.23, stdev=18.29 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 78], 10.00th=[ 83], 20.00th=[ 89], | 30.00th=[ 95], 40.00th=[ 100], 50.00th=[ 105], 60.00th=[ 109], | 70.00th=[ 114], 80.00th=[ 121], 90.00th=[ 130], 95.00th=[ 136], | 99.00th=[ 150], 99.50th=[ 157], 99.90th=[ 178], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 180, max= 407, per=24.79%, avg=302.42, stdev=44.73, samples=33 iops : min= 45, max= 101, avg=75.18, stdev=11.08, samples=33 lat (msec) : 50=0.08%, 100=46.91%, 250=53.01% cpu : usr=0.45%, sys=3.82%, ctx=4806, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1172KiB/s (1200kB/s), 291KiB/s-296KiB/s (298kB/s-303kB/s), io=19.6MiB (20.5MB), run=17022-17119msec WRITE: bw=1220KiB/s (1250kB/s), 303KiB/s-309KiB/s (310kB/s-317kB/s), io=20.4MiB (21.4MB), run=17022-17119msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=192693: Thu Mar 5 20:42:37 2026 read: IOPS=3, BW=143MiB/s (150MB/s)(40.0MiB/279msec) slat (nsec): min=21846k, max=21846k, avg=21846271.00, stdev= 0.00 clat (nsec): min=240314k, max=240314k, avg=240314498.00, stdev= 0.00 lat (nsec): min=262498k, max=262498k, avg=262497842.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 241], 10.00th=[ 241], 20.00th=[ 241], | 30.00th=[ 241], 40.00th=[ 241], 50.00th=[ 241], 60.00th=[ 241], | 70.00th=[ 241], 80.00th=[ 241], 90.00th=[ 241], 95.00th=[ 241], | 99.00th=[ 241], 99.50th=[ 241], 99.90th=[ 241], 99.95th=[ 241], | 99.99th=[ 241] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.40%, ctx=11, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=40.0MiB (41.9MB), run=279-279msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout