== sanity test 398c: run fio to test AIO ================= 06:29:16 (1777026556) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.186603 s, 225 MB/s osc.lustre-OST0000-osc-ffffa0c1e973e800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=195182: Fri Apr 24 06:29:31 2026 write: IOPS=227, BW=909KiB/s (931kB/s)(10.0MiB/11262msec); 0 zone resets slat (usec): min=36, max=28637, avg=205.72, stdev=592.42 clat (msec): min=29, max=133, avg=69.86, stdev=13.43 lat (msec): min=29, max=133, avg=70.07, stdev=13.42 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 46], 10.00th=[ 52], 20.00th=[ 60], | 30.00th=[ 64], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 78], 80.00th=[ 81], 90.00th=[ 86], 95.00th=[ 90], | 99.00th=[ 100], 99.50th=[ 102], 99.90th=[ 126], 99.95th=[ 127], | 99.99th=[ 134] bw ( KiB/s): min= 694, max= 1256, per=24.98%, avg=907.45, stdev=126.25, samples=22 iops : min= 173, max= 314, avg=226.73, stdev=31.58, samples=22 lat (msec) : 50=7.58%, 100=91.48%, 250=0.94% cpu : usr=0.28%, sys=3.28%, ctx=2641, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=195183: Fri Apr 24 06:29:31 2026 write: IOPS=227, BW=908KiB/s (930kB/s)(10.0MiB/11274msec); 0 zone resets slat (usec): min=37, max=2826, avg=197.09, stdev=181.74 clat (msec): min=8, max=106, avg=69.84, stdev=12.90 lat (msec): min=8, max=106, avg=70.04, stdev=12.91 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 46], 10.00th=[ 53], 20.00th=[ 59], | 30.00th=[ 64], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 78], 80.00th=[ 81], 90.00th=[ 86], 95.00th=[ 91], | 99.00th=[ 97], 99.50th=[ 100], 99.90th=[ 102], 99.95th=[ 103], | 99.99th=[ 107] bw ( KiB/s): min= 766, max= 1232, per=24.91%, avg=905.14, stdev=109.04, samples=22 iops : min= 191, max= 308, avg=226.14, stdev=27.26, samples=22 lat (msec) : 10=0.04%, 20=0.20%, 50=7.42%, 100=92.15%, 250=0.20% cpu : usr=0.44%, sys=3.11%, ctx=2932, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=195184: Fri Apr 24 06:29:31 2026 write: IOPS=228, BW=913KiB/s (935kB/s)(10.0MiB/11216msec); 0 zone resets slat (usec): min=40, max=2271, avg=212.65, stdev=193.11 clat (msec): min=10, max=140, avg=69.77, stdev=13.67 lat (msec): min=10, max=141, avg=69.99, stdev=13.68 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 47], 10.00th=[ 52], 20.00th=[ 59], | 30.00th=[ 64], 40.00th=[ 67], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 78], 80.00th=[ 81], 90.00th=[ 86], 95.00th=[ 90], | 99.00th=[ 99], 99.50th=[ 102], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 742, max= 1273, per=25.07%, avg=910.91, stdev=133.41, samples=22 iops : min= 185, max= 318, avg=227.59, stdev=33.36, samples=22 lat (msec) : 20=0.04%, 50=8.44%, 100=90.94%, 250=0.59% cpu : usr=0.35%, sys=3.46%, ctx=2668, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=195185: Fri Apr 24 06:29:31 2026 write: IOPS=227, BW=909KiB/s (931kB/s)(10.0MiB/11268msec); 0 zone resets slat (usec): min=43, max=28694, avg=229.01, stdev=605.19 clat (msec): min=8, max=113, avg=69.80, stdev=13.23 lat (msec): min=10, max=113, avg=70.04, stdev=13.20 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 47], 10.00th=[ 52], 20.00th=[ 60], | 30.00th=[ 64], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 78], 80.00th=[ 81], 90.00th=[ 86], 95.00th=[ 90], | 99.00th=[ 97], 99.50th=[ 101], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 114] bw ( KiB/s): min= 750, max= 1224, per=24.93%, avg=905.77, stdev=118.54, samples=22 iops : min= 187, max= 306, avg=226.32, stdev=29.67, samples=22 lat (msec) : 10=0.04%, 20=0.23%, 50=7.38%, 100=91.84%, 250=0.51% cpu : usr=0.36%, sys=3.51%, ctx=2742, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3633KiB/s (3720kB/s), 908KiB/s-913KiB/s (930kB/s-935kB/s), io=40.0MiB (41.9MB), run=11216-11274msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=195196: Fri Apr 24 06:29:45 2026 read: IOPS=125, BW=504KiB/s (516kB/s)(5048KiB/10025msec) slat (usec): min=1476, max=15170, avg=3916.97, stdev=1421.72 clat (usec): min=32510, max=97575, avg=58986.67, stdev=10579.92 lat (msec): min=35, max=102, avg=62.91, stdev=10.87 clat percentiles (usec): | 1.00th=[38536], 5.00th=[43779], 10.00th=[46400], 20.00th=[49546], | 30.00th=[52691], 40.00th=[55313], 50.00th=[57934], 60.00th=[61080], | 70.00th=[63701], 80.00th=[67634], 90.00th=[72877], 95.00th=[78119], | 99.00th=[88605], 99.50th=[91751], 99.90th=[95945], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 439, max= 622, per=25.07%, avg=501.68, stdev=48.70, samples=19 iops : min= 109, max= 155, avg=125.11, stdev=12.22, samples=19 write: IOPS=129, BW=518KiB/s (530kB/s)(5192KiB/10025msec); 0 zone resets slat (usec): min=53, max=2353, avg=200.59, stdev=154.34 clat (msec): min=26, max=106, avg=61.47, stdev=10.05 lat (msec): min=26, max=107, avg=61.68, stdev=10.04 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 47], 10.00th=[ 50], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 61], 60.00th=[ 64], | 70.00th=[ 67], 80.00th=[ 70], 90.00th=[ 74], 95.00th=[ 79], | 99.00th=[ 88], 99.50th=[ 92], 99.90th=[ 106], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 423, max= 632, per=24.83%, avg=517.11, stdev=61.52, samples=19 iops : min= 105, max= 158, avg=128.95, stdev=15.62, samples=19 lat (msec) : 50=16.25%, 100=83.67%, 250=0.08% cpu : usr=0.51%, sys=4.32%, ctx=4537, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=195197: Fri Apr 24 06:29:45 2026 read: IOPS=124, BW=497KiB/s (509kB/s)(4976KiB/10015msec) slat (usec): min=1344, max=12122, avg=3939.71, stdev=1382.48 clat (usec): min=32276, max=97055, avg=58830.99, stdev=10383.11 lat (msec): min=35, max=101, avg=62.77, stdev=10.68 clat percentiles (usec): | 1.00th=[38536], 5.00th=[43779], 10.00th=[46400], 20.00th=[50070], | 30.00th=[52691], 40.00th=[55313], 50.00th=[57934], 60.00th=[60556], | 70.00th=[63701], 80.00th=[67634], 90.00th=[72877], 95.00th=[76022], | 99.00th=[87557], 99.50th=[90702], 99.90th=[95945], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 423, max= 592, per=24.73%, avg=494.79, stdev=48.29, samples=19 iops : min= 105, max= 148, avg=123.37, stdev=12.25, samples=19 write: IOPS=131, BW=526KiB/s (538kB/s)(5264KiB/10015msec); 0 zone resets slat (usec): min=53, max=2402, avg=196.49, stdev=155.30 clat (msec): min=25, max=111, avg=61.66, stdev= 9.92 lat (msec): min=25, max=111, avg=61.85, stdev= 9.92 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 48], 10.00th=[ 51], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 64], | 70.00th=[ 66], 80.00th=[ 69], 90.00th=[ 74], 95.00th=[ 78], | 99.00th=[ 90], 99.50th=[ 99], 99.90th=[ 108], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 400, max= 646, per=25.22%, avg=525.42, stdev=68.37, samples=19 iops : min= 100, max= 161, avg=131.05, stdev=17.17, samples=19 lat (msec) : 50=14.45%, 100=85.39%, 250=0.16% cpu : usr=0.53%, sys=4.28%, ctx=4503, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=195198: Fri Apr 24 06:29:45 2026 read: IOPS=125, BW=500KiB/s (512kB/s)(4984KiB/9959msec) slat (usec): min=38, max=15692, avg=3938.26, stdev=1478.34 clat (usec): min=6561, max=97401, avg=58784.59, stdev=10852.24 lat (msec): min=6, max=102, avg=62.73, stdev=11.19 clat percentiles (usec): | 1.00th=[34341], 5.00th=[43779], 10.00th=[46400], 20.00th=[50594], | 30.00th=[53216], 40.00th=[55313], 50.00th=[57934], 60.00th=[60556], | 70.00th=[63177], 80.00th=[66847], 90.00th=[72877], 95.00th=[78119], | 99.00th=[86508], 99.50th=[90702], 99.90th=[95945], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 376, max= 606, per=25.09%, avg=502.11, stdev=57.05, samples=19 iops : min= 94, max= 151, avg=125.21, stdev=14.38, samples=19 write: IOPS=131, BW=528KiB/s (540kB/s)(5256KiB/9959msec); 0 zone resets slat (usec): min=46, max=4188, avg=204.82, stdev=223.57 clat (msec): min=23, max=107, avg=61.38, stdev=10.37 lat (msec): min=23, max=108, avg=61.58, stdev=10.37 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 47], 10.00th=[ 50], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 61], 60.00th=[ 64], | 70.00th=[ 66], 80.00th=[ 70], 90.00th=[ 74], 95.00th=[ 80], | 99.00th=[ 88], 99.50th=[ 92], 99.90th=[ 107], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 415, max= 630, per=25.18%, avg=524.58, stdev=63.47, samples=19 iops : min= 103, max= 157, avg=130.84, stdev=16.04, samples=19 lat (msec) : 10=0.08%, 20=0.20%, 50=14.88%, 100=84.73%, 250=0.12% cpu : usr=0.59%, sys=4.35%, ctx=4418, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=195199: Fri Apr 24 06:29:45 2026 read: IOPS=126, BW=506KiB/s (518kB/s)(5060KiB/10005msec) slat (usec): min=1652, max=12897, avg=3941.12, stdev=1390.11 clat (msec): min=9, max=101, avg=59.20, stdev=10.61 lat (msec): min=12, max=104, avg=63.15, stdev=10.88 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 44], 10.00th=[ 47], 20.00th=[ 51], | 30.00th=[ 54], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 79], | 99.00th=[ 89], 99.50th=[ 92], 99.90th=[ 101], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 416, max= 632, per=25.15%, avg=503.26, stdev=59.81, samples=19 iops : min= 104, max= 158, avg=125.47, stdev=15.06, samples=19 write: IOPS=129, BW=518KiB/s (530kB/s)(5180KiB/10005msec); 0 zone resets slat (usec): min=44, max=2501, avg=198.28, stdev=176.24 clat (msec): min=10, max=107, avg=61.18, stdev=10.03 lat (msec): min=10, max=107, avg=61.38, stdev=10.04 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 47], 10.00th=[ 50], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 64], | 70.00th=[ 66], 80.00th=[ 69], 90.00th=[ 74], 95.00th=[ 78], | 99.00th=[ 86], 99.50th=[ 100], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 415, max= 616, per=24.78%, avg=516.26, stdev=49.89, samples=19 iops : min= 103, max= 154, avg=128.74, stdev=12.67, samples=19 lat (msec) : 10=0.04%, 20=0.04%, 50=14.84%, 100=84.80%, 250=0.27% cpu : usr=0.50%, sys=4.41%, ctx=4567, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2002KiB/s (2050kB/s), 497KiB/s-506KiB/s (509kB/s-518kB/s), io=19.6MiB (20.5MB), run=9959-10025msec WRITE: bw=2084KiB/s (2134kB/s), 518KiB/s-528KiB/s (530kB/s-540kB/s), io=20.4MiB (21.4MB), run=9959-10025msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=195204: Fri Apr 24 06:29:49 2026 read: IOPS=7, BW=292MiB/s (306MB/s)(40.0MiB/137msec) slat (nsec): min=13636k, max=13636k, avg=13635619.00, stdev= 0.00 clat (nsec): min=116231k, max=116231k, avg=116230657.00, stdev= 0.00 lat (nsec): min=129871k, max=129871k, avg=129870653.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 116], 10.00th=[ 116], 20.00th=[ 116], | 30.00th=[ 116], 40.00th=[ 116], 50.00th=[ 116], 60.00th=[ 116], | 70.00th=[ 116], 80.00th=[ 116], 90.00th=[ 116], 95.00th=[ 116], | 99.00th=[ 116], 99.50th=[ 116], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.09%, ctx=11, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=292MiB/s (306MB/s), 292MiB/s-292MiB/s (306MB/s-306MB/s), io=40.0MiB (41.9MB), run=137-137msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout