== sanity test 398c: run fio to test AIO ================= 14:03:55 (1783447435) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.267644 s, 157 MB/s osc.lustre-OST0000-osc-ffff9d93a8666800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180053: Tue Jul 7 14:04:17 2026 write: IOPS=148, BW=594KiB/s (609kB/s)(10.0MiB/17231msec); 0 zone resets slat (usec): min=37, max=9152, avg=266.28, stdev=490.26 clat (msec): min=21, max=178, avg=106.94, stdev=21.17 lat (msec): min=21, max=178, avg=107.21, stdev=21.19 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 72], 10.00th=[ 82], 20.00th=[ 90], | 30.00th=[ 96], 40.00th=[ 101], 50.00th=[ 106], 60.00th=[ 111], | 70.00th=[ 117], 80.00th=[ 124], 90.00th=[ 134], 95.00th=[ 144], | 99.00th=[ 165], 99.50th=[ 174], 99.90th=[ 176], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 408, max= 862, per=24.95%, avg=593.06, stdev=92.57, samples=34 iops : min= 102, max= 215, avg=148.12, stdev=23.13, samples=34 lat (msec) : 50=0.04%, 100=38.55%, 250=61.41% cpu : usr=0.28%, sys=1.88%, ctx=2750, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180054: Tue Jul 7 14:04:17 2026 write: IOPS=149, BW=597KiB/s (611kB/s)(10.0MiB/17151msec); 0 zone resets slat (usec): min=40, max=4698, avg=262.17, stdev=416.93 clat (msec): min=21, max=174, avg=106.82, stdev=21.87 lat (msec): min=21, max=175, avg=107.09, stdev=21.87 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 73], 10.00th=[ 80], 20.00th=[ 90], | 30.00th=[ 95], 40.00th=[ 102], 50.00th=[ 107], 60.00th=[ 111], | 70.00th=[ 117], 80.00th=[ 125], 90.00th=[ 136], 95.00th=[ 146], | 99.00th=[ 167], 99.50th=[ 171], 99.90th=[ 176], 99.95th=[ 176], | 99.99th=[ 176] bw ( KiB/s): min= 440, max= 864, per=25.04%, avg=595.21, stdev=96.27, samples=33 iops : min= 110, max= 216, avg=148.64, stdev=24.02, samples=33 lat (msec) : 50=0.43%, 100=37.46%, 250=62.11% cpu : usr=0.23%, sys=1.97%, ctx=2909, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180055: Tue Jul 7 14:04:17 2026 write: IOPS=148, BW=595KiB/s (609kB/s)(10.0MiB/17211msec); 0 zone resets slat (usec): min=39, max=5455, avg=255.50, stdev=404.61 clat (msec): min=8, max=184, avg=106.97, stdev=22.04 lat (msec): min=9, max=184, avg=107.23, stdev=22.06 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 73], 10.00th=[ 80], 20.00th=[ 90], | 30.00th=[ 96], 40.00th=[ 102], 50.00th=[ 107], 60.00th=[ 111], | 70.00th=[ 116], 80.00th=[ 124], 90.00th=[ 138], 95.00th=[ 146], | 99.00th=[ 163], 99.50th=[ 167], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 440, max= 896, per=24.97%, avg=593.47, stdev=92.78, samples=34 iops : min= 110, max= 224, avg=148.21, stdev=23.17, samples=34 lat (msec) : 10=0.04%, 20=0.04%, 50=0.74%, 100=37.03%, 250=62.15% cpu : usr=0.27%, sys=1.93%, ctx=2849, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180056: Tue Jul 7 14:04:17 2026 write: IOPS=148, BW=595KiB/s (609kB/s)(10.0MiB/17212msec); 0 zone resets slat (usec): min=43, max=5459, avg=257.29, stdev=414.67 clat (msec): min=23, max=185, avg=106.89, stdev=21.59 lat (msec): min=24, max=185, avg=107.15, stdev=21.61 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 73], 10.00th=[ 81], 20.00th=[ 89], | 30.00th=[ 95], 40.00th=[ 102], 50.00th=[ 107], 60.00th=[ 111], | 70.00th=[ 116], 80.00th=[ 124], 90.00th=[ 134], 95.00th=[ 146], | 99.00th=[ 167], 99.50th=[ 171], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 456, max= 878, per=24.97%, avg=593.56, stdev=89.62, samples=34 iops : min= 114, max= 219, avg=148.24, stdev=22.37, samples=34 lat (msec) : 50=0.08%, 100=38.20%, 250=61.72% cpu : usr=0.23%, sys=1.96%, ctx=2876, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2377KiB/s (2434kB/s), 594KiB/s-597KiB/s (609kB/s-611kB/s), io=40.0MiB (41.9MB), run=17151-17231msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180068: Tue Jul 7 14:04:40 2026 read: IOPS=68, BW=274KiB/s (281kB/s)(5048KiB/18424msec) slat (usec): min=2133, max=24616, avg=7654.84, stdev=3334.71 clat (msec): min=8, max=186, avg=108.61, stdev=21.63 lat (msec): min=16, max=194, avg=116.27, stdev=22.11 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 78], 10.00th=[ 83], 20.00th=[ 91], | 30.00th=[ 97], 40.00th=[ 103], 50.00th=[ 108], 60.00th=[ 113], | 70.00th=[ 118], 80.00th=[ 126], 90.00th=[ 140], 95.00th=[ 146], | 99.00th=[ 163], 99.50th=[ 169], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 186] bw ( KiB/s): min= 195, max= 359, per=24.93%, avg=271.47, stdev=40.30, samples=36 iops : min= 48, max= 89, avg=67.44, stdev=10.10, samples=36 write: IOPS=70, BW=282KiB/s (289kB/s)(5192KiB/18424msec); 0 zone resets slat (usec): min=50, max=6556, avg=288.32, stdev=482.30 clat (msec): min=8, max=178, avg=112.49, stdev=19.74 lat (msec): min=8, max=178, avg=112.78, stdev=19.72 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 84], 10.00th=[ 90], 20.00th=[ 96], | 30.00th=[ 102], 40.00th=[ 107], 50.00th=[ 111], 60.00th=[ 116], | 70.00th=[ 122], 80.00th=[ 130], 90.00th=[ 140], 95.00th=[ 148], | 99.00th=[ 163], 99.50th=[ 167], 99.90th=[ 176], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 191, max= 375, per=24.76%, avg=280.56, stdev=44.46, samples=36 iops : min= 47, max= 93, avg=69.72, stdev=11.17, samples=36 lat (msec) : 10=0.08%, 20=0.08%, 50=0.16%, 100=31.56%, 250=68.12% cpu : usr=0.31%, sys=2.69%, ctx=4734, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180069: Tue Jul 7 14:04:40 2026 read: IOPS=67, BW=271KiB/s (278kB/s)(4976KiB/18358msec) slat (usec): min=60, max=30874, avg=7628.14, stdev=3538.92 clat (msec): min=17, max=196, avg=108.35, stdev=21.51 lat (msec): min=17, max=202, avg=115.98, stdev=22.16 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 79], 10.00th=[ 84], 20.00th=[ 91], | 30.00th=[ 97], 40.00th=[ 103], 50.00th=[ 108], 60.00th=[ 112], | 70.00th=[ 117], 80.00th=[ 125], 90.00th=[ 138], 95.00th=[ 146], | 99.00th=[ 165], 99.50th=[ 176], 99.90th=[ 194], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 175, max= 352, per=24.74%, avg=269.44, stdev=47.36, samples=36 iops : min= 43, max= 88, avg=66.94, stdev=11.96, samples=36 write: IOPS=71, BW=287KiB/s (294kB/s)(5264KiB/18358msec); 0 zone resets slat (usec): min=57, max=4900, avg=268.30, stdev=413.77 clat (msec): min=52, max=196, avg=112.94, stdev=20.07 lat (msec): min=52, max=196, avg=113.22, stdev=20.07 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 84], 10.00th=[ 89], 20.00th=[ 96], | 30.00th=[ 102], 40.00th=[ 107], 50.00th=[ 112], 60.00th=[ 117], | 70.00th=[ 124], 80.00th=[ 132], 90.00th=[ 140], 95.00th=[ 146], | 99.00th=[ 163], 99.50th=[ 167], 99.90th=[ 176], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 223, max= 367, per=25.35%, avg=287.25, stdev=32.85, samples=36 iops : min= 55, max= 91, avg=71.42, stdev= 8.10, samples=36 lat (msec) : 20=0.08%, 50=0.12%, 100=31.68%, 250=68.12% cpu : usr=0.33%, sys=2.66%, ctx=4700, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180070: Tue Jul 7 14:04:40 2026 read: IOPS=67, BW=271KiB/s (277kB/s)(4984KiB/18401msec) slat (usec): min=2617, max=28353, avg=7606.93, stdev=3194.61 clat (msec): min=8, max=179, avg=108.88, stdev=21.22 lat (msec): min=12, max=192, avg=116.50, stdev=21.68 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 78], 10.00th=[ 84], 20.00th=[ 91], | 30.00th=[ 97], 40.00th=[ 103], 50.00th=[ 108], 60.00th=[ 113], | 70.00th=[ 118], 80.00th=[ 126], 90.00th=[ 138], 95.00th=[ 146], | 99.00th=[ 167], 99.50th=[ 174], 99.90th=[ 178], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 192, max= 359, per=24.87%, avg=270.81, stdev=45.30, samples=36 iops : min= 48, max= 89, avg=67.28, stdev=11.36, samples=36 write: IOPS=71, BW=286KiB/s (292kB/s)(5256KiB/18401msec); 0 zone resets slat (usec): min=47, max=5747, avg=272.16, stdev=437.31 clat (msec): min=63, max=183, avg=111.94, stdev=19.51 lat (msec): min=63, max=183, avg=112.21, stdev=19.53 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 83], 10.00th=[ 89], 20.00th=[ 96], | 30.00th=[ 102], 40.00th=[ 107], 50.00th=[ 111], 60.00th=[ 115], | 70.00th=[ 122], 80.00th=[ 129], 90.00th=[ 138], 95.00th=[ 148], | 99.00th=[ 165], 99.50th=[ 174], 99.90th=[ 184], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 191, max= 375, per=25.08%, avg=284.19, stdev=47.98, samples=36 iops : min= 47, max= 93, avg=70.64, stdev=12.02, samples=36 lat (msec) : 10=0.04%, 100=32.11%, 250=67.85% cpu : usr=0.30%, sys=2.57%, ctx=4685, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180071: Tue Jul 7 14:04:40 2026 read: IOPS=68, BW=275KiB/s (282kB/s)(5060KiB/18385msec) slat (usec): min=2331, max=24486, avg=7620.28, stdev=3236.15 clat (msec): min=23, max=176, avg=109.29, stdev=20.84 lat (msec): min=30, max=186, avg=116.91, stdev=21.35 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 79], 10.00th=[ 85], 20.00th=[ 93], | 30.00th=[ 99], 40.00th=[ 104], 50.00th=[ 108], 60.00th=[ 113], | 70.00th=[ 118], 80.00th=[ 127], 90.00th=[ 138], 95.00th=[ 148], | 99.00th=[ 167], 99.50th=[ 174], 99.90th=[ 176], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 207, max= 344, per=25.23%, avg=274.75, stdev=37.27, samples=36 iops : min= 51, max= 86, avg=68.28, stdev= 9.36, samples=36 write: IOPS=70, BW=282KiB/s (289kB/s)(5180KiB/18385msec); 0 zone resets slat (usec): min=59, max=10184, avg=300.51, stdev=548.26 clat (msec): min=59, max=178, avg=111.37, stdev=19.09 lat (msec): min=59, max=179, avg=111.68, stdev=19.08 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 84], 10.00th=[ 89], 20.00th=[ 95], | 30.00th=[ 101], 40.00th=[ 105], 50.00th=[ 110], 60.00th=[ 115], | 70.00th=[ 121], 80.00th=[ 127], 90.00th=[ 138], 95.00th=[ 148], | 99.00th=[ 161], 99.50th=[ 167], 99.90th=[ 174], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 207, max= 407, per=24.78%, avg=280.78, stdev=48.66, samples=36 iops : min= 51, max= 101, avg=69.78, stdev=12.18, samples=36 lat (msec) : 50=0.04%, 100=31.80%, 250=68.16% cpu : usr=0.29%, sys=2.66%, ctx=4837, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1089KiB/s (1115kB/s), 271KiB/s-275KiB/s (277kB/s-282kB/s), io=19.6MiB (20.5MB), run=18358-18424msec WRITE: bw=1134KiB/s (1161kB/s), 282KiB/s-287KiB/s (289kB/s-294kB/s), io=20.4MiB (21.4MB), run=18358-18424msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180078: Tue Jul 7 14:04:46 2026 read: IOPS=4, BW=198MiB/s (208MB/s)(40.0MiB/202msec) slat (nsec): min=18493k, max=18493k, avg=18492954.00, stdev= 0.00 clat (nsec): min=173615k, max=173615k, avg=173614790.00, stdev= 0.00 lat (nsec): min=192113k, max=192113k, avg=192112731.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 174], 10.00th=[ 174], 20.00th=[ 174], | 30.00th=[ 174], 40.00th=[ 174], 50.00th=[ 174], 60.00th=[ 174], | 70.00th=[ 174], 80.00th=[ 174], 90.00th=[ 174], 95.00th=[ 174], | 99.00th=[ 174], 99.50th=[ 174], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.97%, ctx=10, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=198MiB/s (208MB/s), 198MiB/s-198MiB/s (208MB/s-208MB/s), io=40.0MiB (41.9MB), run=202-202msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout