== sanity test 398c: run fio to test AIO ================= 16:57:14 (1784235434) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.146869 s, 286 MB/s osc.lustre-OST0000-osc-ffff95d3593fb800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=189884: Thu Jul 16 16:57:27 2026 write: IOPS=275, BW=1104KiB/s (1130kB/s)(10.0MiB/9277msec); 0 zone resets slat (usec): min=38, max=3353, avg=211.46, stdev=188.46 clat (usec): min=21390, max=99779, avg=57696.83, stdev=14157.78 lat (msec): min=21, max=100, avg=57.91, stdev=14.16 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 35], 10.00th=[ 40], 20.00th=[ 45], | 30.00th=[ 50], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 62], | 70.00th=[ 65], 80.00th=[ 69], 90.00th=[ 78], 95.00th=[ 84], | 99.00th=[ 90], 99.50th=[ 95], 99.90th=[ 97], 99.95th=[ 99], | 99.99th=[ 101] bw ( KiB/s): min= 888, max= 1568, per=24.84%, avg=1092.94, stdev=197.29, samples=18 iops : min= 222, max= 392, avg=273.06, stdev=49.34, samples=18 lat (msec) : 50=32.03%, 100=67.97% cpu : usr=0.50%, sys=3.86%, ctx=2666, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=189885: Thu Jul 16 16:57:27 2026 write: IOPS=275, BW=1101KiB/s (1127kB/s)(10.0MiB/9302msec); 0 zone resets slat (usec): min=52, max=23996, avg=223.25, stdev=509.07 clat (usec): min=10526, max=99439, avg=57578.76, stdev=14319.32 lat (usec): min=10680, max=99506, avg=57803.29, stdev=14301.79 clat percentiles (usec): | 1.00th=[29492], 5.00th=[34866], 10.00th=[39060], 20.00th=[44827], | 30.00th=[49021], 40.00th=[54789], 50.00th=[57934], 60.00th=[61080], | 70.00th=[64226], 80.00th=[68682], 90.00th=[76022], 95.00th=[81265], | 99.00th=[92799], 99.50th=[96994], 99.90th=[99091], 99.95th=[99091], | 99.99th=[99091] bw ( KiB/s): min= 832, max= 1584, per=24.74%, avg=1088.72, stdev=192.31, samples=18 iops : min= 208, max= 396, avg=272.06, stdev=48.15, samples=18 lat (msec) : 20=0.16%, 50=31.76%, 100=68.09% cpu : usr=0.35%, sys=4.26%, ctx=2720, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=189886: Thu Jul 16 16:57:27 2026 write: IOPS=275, BW=1100KiB/s (1127kB/s)(10.0MiB/9305msec); 0 zone resets slat (usec): min=39, max=21267, avg=234.13, stdev=454.57 clat (usec): min=10513, max=98394, avg=57590.87, stdev=14318.80 lat (usec): min=12579, max=99380, avg=57826.31, stdev=14309.06 clat percentiles (usec): | 1.00th=[28181], 5.00th=[34866], 10.00th=[39060], 20.00th=[44827], | 30.00th=[49546], 40.00th=[54789], 50.00th=[57934], 60.00th=[61080], | 70.00th=[64750], 80.00th=[68682], 90.00th=[77071], 95.00th=[82314], | 99.00th=[93848], 99.50th=[96994], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 856, max= 1552, per=24.69%, avg=1086.50, stdev=186.20, samples=18 iops : min= 214, max= 388, avg=271.50, stdev=46.63, samples=18 lat (msec) : 20=0.16%, 50=32.54%, 100=67.30% cpu : usr=0.31%, sys=4.44%, ctx=2744, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=189887: Thu Jul 16 16:57:27 2026 write: IOPS=275, BW=1100KiB/s (1127kB/s)(10.0MiB/9308msec); 0 zone resets slat (usec): min=43, max=2150, avg=197.67, stdev=157.68 clat (usec): min=6970, max=99881, avg=57617.56, stdev=14177.85 lat (msec): min=7, max=100, avg=57.82, stdev=14.18 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 35], 10.00th=[ 40], 20.00th=[ 45], | 30.00th=[ 50], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 62], | 70.00th=[ 65], 80.00th=[ 69], 90.00th=[ 78], 95.00th=[ 83], | 99.00th=[ 94], 99.50th=[ 97], 99.90th=[ 101], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 840, max= 1560, per=24.66%, avg=1085.17, stdev=182.70, samples=18 iops : min= 210, max= 390, avg=271.17, stdev=45.75, samples=18 lat (msec) : 10=0.04%, 20=0.12%, 50=31.80%, 100=68.05% cpu : usr=0.41%, sys=3.99%, ctx=2640, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=4401KiB/s (4506kB/s), 1100KiB/s-1104KiB/s (1127kB/s-1130kB/s), io=40.0MiB (41.9MB), run=9277-9308msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=189898: Thu Jul 16 16:57:40 2026 read: IOPS=137, BW=551KiB/s (564kB/s)(5048KiB/9169msec) slat (usec): min=1082, max=12092, avg=3727.85, stdev=1520.02 clat (msec): min=28, max=112, avg=54.09, stdev=13.24 lat (msec): min=30, max=115, avg=57.82, stdev=13.85 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 43], | 30.00th=[ 46], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 60], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 80], | 99.00th=[ 99], 99.50th=[ 102], 99.90th=[ 107], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 311, max= 710, per=25.07%, avg=548.44, stdev=101.33, samples=18 iops : min= 77, max= 177, avg=136.89, stdev=25.41, samples=18 write: IOPS=141, BW=566KiB/s (580kB/s)(5192KiB/9169msec); 0 zone resets slat (usec): min=58, max=2028, avg=187.52, stdev=147.02 clat (msec): min=6, max=116, avg=56.07, stdev=13.58 lat (msec): min=7, max=116, avg=56.26, stdev=13.59 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 45], | 30.00th=[ 49], 40.00th=[ 52], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 61], 80.00th=[ 66], 90.00th=[ 75], 95.00th=[ 83], | 99.00th=[ 97], 99.50th=[ 103], 99.90th=[ 109], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 423, max= 725, per=24.84%, avg=565.89, stdev=88.96, samples=18 iops : min= 105, max= 181, avg=141.28, stdev=22.28, samples=18 lat (msec) : 10=0.04%, 20=0.08%, 50=38.79%, 100=60.43%, 250=0.66% cpu : usr=0.50%, sys=4.29%, ctx=4521, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=189899: Thu Jul 16 16:57:40 2026 read: IOPS=136, BW=545KiB/s (558kB/s)(4976KiB/9137msec) slat (usec): min=63, max=10981, avg=3691.50, stdev=1507.12 clat (msec): min=6, max=111, avg=54.33, stdev=13.78 lat (msec): min=6, max=117, avg=58.02, stdev=14.39 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 43], | 30.00th=[ 46], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 57], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 81], | 99.00th=[ 97], 99.50th=[ 103], 99.90th=[ 110], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 343, max= 702, per=24.67%, avg=539.72, stdev=89.65, samples=18 iops : min= 85, max= 175, avg=134.72, stdev=22.51, samples=18 write: IOPS=144, BW=576KiB/s (590kB/s)(5264KiB/9137msec); 0 zone resets slat (usec): min=57, max=3385, avg=194.07, stdev=177.79 clat (msec): min=14, max=108, avg=55.89, stdev=13.08 lat (msec): min=14, max=108, avg=56.08, stdev=13.10 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 46], | 30.00th=[ 49], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 57], | 70.00th=[ 61], 80.00th=[ 66], 90.00th=[ 73], 95.00th=[ 82], | 99.00th=[ 95], 99.50th=[ 100], 99.90th=[ 109], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 383, max= 756, per=25.19%, avg=573.72, stdev=103.05, samples=18 iops : min= 95, max= 189, avg=143.28, stdev=25.84, samples=18 lat (msec) : 10=0.04%, 20=0.20%, 50=37.73%, 100=61.48%, 250=0.55% cpu : usr=0.38%, sys=4.50%, ctx=4454, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=189900: Thu Jul 16 16:57:40 2026 read: IOPS=136, BW=546KiB/s (559kB/s)(4984KiB/9136msec) slat (usec): min=1172, max=13167, avg=3707.54, stdev=1450.16 clat (msec): min=10, max=116, avg=53.56, stdev=13.25 lat (msec): min=12, max=119, avg=57.27, stdev=13.80 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 36], 10.00th=[ 40], 20.00th=[ 43], | 30.00th=[ 46], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 64], 90.00th=[ 70], 95.00th=[ 78], | 99.00th=[ 100], 99.50th=[ 105], 99.90th=[ 114], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 384, max= 680, per=24.89%, avg=544.56, stdev=92.46, samples=18 iops : min= 96, max= 170, avg=135.94, stdev=23.20, samples=18 write: IOPS=143, BW=575KiB/s (589kB/s)(5256KiB/9136msec); 0 zone resets slat (usec): min=44, max=4006, avg=197.36, stdev=197.18 clat (msec): min=23, max=114, avg=56.09, stdev=13.44 lat (msec): min=23, max=114, avg=56.29, stdev=13.45 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 45], | 30.00th=[ 49], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 57], | 70.00th=[ 61], 80.00th=[ 66], 90.00th=[ 77], 95.00th=[ 83], | 99.00th=[ 96], 99.50th=[ 100], 99.90th=[ 111], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 295, max= 728, per=25.04%, avg=570.50, stdev=115.99, samples=18 iops : min= 73, max= 182, avg=142.44, stdev=29.08, samples=18 lat (msec) : 20=0.16%, 50=39.80%, 100=59.41%, 250=0.62% cpu : usr=0.36%, sys=4.50%, ctx=4571, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=189901: Thu Jul 16 16:57:40 2026 read: IOPS=138, BW=553KiB/s (566kB/s)(5060KiB/9156msec) slat (usec): min=1097, max=10123, avg=3658.88, stdev=1374.18 clat (msec): min=27, max=105, avg=54.22, stdev=13.30 lat (msec): min=31, max=109, avg=57.89, stdev=13.83 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 44], | 30.00th=[ 47], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 60], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 80], | 99.00th=[ 97], 99.50th=[ 102], 99.90th=[ 106], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 351, max= 725, per=25.16%, avg=550.50, stdev=96.93, samples=18 iops : min= 87, max= 181, avg=137.44, stdev=24.29, samples=18 write: IOPS=141, BW=566KiB/s (579kB/s)(5180KiB/9156msec); 0 zone resets slat (usec): min=44, max=6594, avg=202.73, stdev=266.37 clat (msec): min=29, max=102, avg=55.82, stdev=13.34 lat (msec): min=29, max=102, avg=56.03, stdev=13.36 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 45], | 30.00th=[ 48], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 57], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 75], 95.00th=[ 84], | 99.00th=[ 96], 99.50th=[ 100], 99.90th=[ 101], 99.95th=[ 103], | 99.99th=[ 103] bw ( KiB/s): min= 383, max= 760, per=24.76%, avg=564.11, stdev=103.22, samples=18 iops : min= 95, max= 190, avg=140.83, stdev=25.90, samples=18 lat (msec) : 50=38.71%, 100=60.86%, 250=0.43% cpu : usr=0.46%, sys=4.38%, ctx=4559, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2189KiB/s (2241kB/s), 545KiB/s-553KiB/s (558kB/s-566kB/s), io=19.6MiB (20.5MB), run=9136-9169msec WRITE: bw=2279KiB/s (2333kB/s), 566KiB/s-576KiB/s (579kB/s-590kB/s), io=20.4MiB (21.4MB), run=9136-9169msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=189906: Thu Jul 16 16:57:44 2026 read: IOPS=7, BW=305MiB/s (320MB/s)(40.0MiB/131msec) slat (nsec): min=12312k, max=12312k, avg=12312185.00, stdev= 0.00 clat (nsec): min=110927k, max=110927k, avg=110926526.00, stdev= 0.00 lat (nsec): min=123244k, max=123244k, avg=123243507.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 111], 10.00th=[ 111], 20.00th=[ 111], | 30.00th=[ 111], 40.00th=[ 111], 50.00th=[ 111], 60.00th=[ 111], | 70.00th=[ 111], 80.00th=[ 111], 90.00th=[ 111], 95.00th=[ 111], | 99.00th=[ 111], 99.50th=[ 111], 99.90th=[ 111], 99.95th=[ 111], | 99.99th=[ 111] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.69%, ctx=6, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=305MiB/s (320MB/s), 305MiB/s-305MiB/s (320MB/s-320MB/s), io=40.0MiB (41.9MB), run=131-131msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout