== sanity test 398c: run fio to test AIO ================= 03:00:08 (1769760008) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.160665 s, 261 MB/s osc.lustre-OST0000-osc-ffff8f3d528e6000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=154500: Fri Jan 30 03:00:22 2026 write: IOPS=269, BW=1079KiB/s (1105kB/s)(10.0MiB/9487msec); 0 zone resets slat (usec): min=47, max=35312, avg=213.94, stdev=715.88 clat (msec): min=6, max=109, avg=58.67, stdev=12.98 lat (msec): min=6, max=109, avg=58.89, stdev=12.96 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 48], | 30.00th=[ 52], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 65], 80.00th=[ 69], 90.00th=[ 75], 95.00th=[ 84], | 99.00th=[ 96], 99.50th=[ 102], 99.90th=[ 105], 99.95th=[ 106], | 99.99th=[ 110] bw ( KiB/s): min= 884, max= 1368, per=24.87%, avg=1072.06, stdev=123.26, samples=18 iops : min= 221, max= 342, avg=267.94, stdev=30.88, samples=18 lat (msec) : 10=0.12%, 20=0.08%, 50=26.21%, 100=73.01%, 250=0.59% cpu : usr=0.45%, sys=4.19%, ctx=2812, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154501: Fri Jan 30 03:00:22 2026 write: IOPS=269, BW=1078KiB/s (1104kB/s)(10.0MiB/9500msec); 0 zone resets slat (usec): min=34, max=2648, avg=142.38, stdev=112.10 clat (msec): min=13, max=108, avg=58.76, stdev=12.71 lat (msec): min=13, max=109, avg=58.91, stdev=12.71 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 48], | 30.00th=[ 52], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 65], 80.00th=[ 68], 90.00th=[ 77], 95.00th=[ 83], | 99.00th=[ 96], 99.50th=[ 101], 99.90th=[ 105], 99.95th=[ 107], | 99.99th=[ 109] bw ( KiB/s): min= 876, max= 1400, per=24.79%, avg=1068.83, stdev=133.75, samples=18 iops : min= 219, max= 350, avg=267.11, stdev=33.49, samples=18 lat (msec) : 20=0.04%, 50=26.95%, 100=72.34%, 250=0.66% cpu : usr=0.31%, sys=3.21%, ctx=2346, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154502: Fri Jan 30 03:00:22 2026 write: IOPS=270, BW=1084KiB/s (1110kB/s)(10.0MiB/9448msec); 0 zone resets slat (usec): min=36, max=2021, avg=191.78, stdev=141.27 clat (msec): min=21, max=109, avg=58.77, stdev=13.06 lat (msec): min=21, max=109, avg=58.97, stdev=13.07 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 48], | 30.00th=[ 52], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 65], 80.00th=[ 69], 90.00th=[ 75], 95.00th=[ 84], | 99.00th=[ 99], 99.50th=[ 103], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 109] bw ( KiB/s): min= 812, max= 1304, per=25.01%, avg=1078.33, stdev=125.67, samples=18 iops : min= 203, max= 326, avg=269.44, stdev=31.46, samples=18 lat (msec) : 50=25.00%, 100=74.34%, 250=0.66% cpu : usr=0.41%, sys=3.97%, ctx=2648, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=154503: Fri Jan 30 03:00:22 2026 write: IOPS=269, BW=1078KiB/s (1104kB/s)(10.0MiB/9499msec); 0 zone resets slat (usec): min=42, max=5310, avg=192.83, stdev=219.63 clat (msec): min=13, max=109, avg=58.65, stdev=12.93 lat (msec): min=13, max=109, avg=58.84, stdev=12.94 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 48], | 30.00th=[ 52], 40.00th=[ 55], 50.00th=[ 57], 60.00th=[ 61], | 70.00th=[ 65], 80.00th=[ 69], 90.00th=[ 75], 95.00th=[ 83], | 99.00th=[ 97], 99.50th=[ 101], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 110] bw ( KiB/s): min= 828, max= 1320, per=24.73%, avg=1065.94, stdev=120.07, samples=18 iops : min= 207, max= 330, avg=266.39, stdev=30.08, samples=18 lat (msec) : 20=0.20%, 50=26.29%, 100=72.97%, 250=0.55% cpu : usr=0.47%, sys=3.72%, ctx=2585, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=4312KiB/s (4415kB/s), 1078KiB/s-1084KiB/s (1104kB/s-1110kB/s), io=40.0MiB (41.9MB), run=9448-9500msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=154515: Fri Jan 30 03:00:33 2026 read: IOPS=165, BW=661KiB/s (676kB/s)(5048KiB/7642msec) slat (usec): min=58, max=10811, avg=3333.65, stdev=1339.52 clat (usec): min=4286, max=88188, avg=45519.98, stdev=9173.89 lat (usec): min=9043, max=96731, avg=48856.73, stdev=9530.23 clat percentiles (usec): | 1.00th=[28181], 5.00th=[32900], 10.00th=[34866], 20.00th=[38011], | 30.00th=[40109], 40.00th=[42730], 50.00th=[44303], 60.00th=[46400], | 70.00th=[49021], 80.00th=[52167], 90.00th=[57934], 95.00th=[62653], | 99.00th=[71828], 99.50th=[76022], 99.90th=[85459], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 544, max= 832, per=25.16%, avg=657.20, stdev=71.76, samples=15 iops : min= 136, max= 208, avg=164.27, stdev=17.91, samples=15 write: IOPS=169, BW=679KiB/s (696kB/s)(5192KiB/7642msec); 0 zone resets slat (usec): min=47, max=1492, avg=177.75, stdev=112.85 clat (usec): min=13205, max=80115, avg=46369.81, stdev=8787.80 lat (usec): min=13292, max=80257, avg=46548.88, stdev=8797.32 clat percentiles (usec): | 1.00th=[27919], 5.00th=[34341], 10.00th=[36963], 20.00th=[39584], | 30.00th=[41681], 40.00th=[43254], 50.00th=[45351], 60.00th=[47449], | 70.00th=[49546], 80.00th=[52691], 90.00th=[57410], 95.00th=[63177], | 99.00th=[73925], 99.50th=[77071], 99.90th=[79168], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min= 606, max= 816, per=24.92%, avg=677.47, stdev=55.40, samples=15 iops : min= 151, max= 204, avg=169.33, stdev=13.90, samples=15 lat (msec) : 10=0.04%, 20=0.23%, 50=72.19%, 100=27.54% cpu : usr=0.62%, sys=5.39%, ctx=4405, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154516: Fri Jan 30 03:00:33 2026 read: IOPS=162, BW=648KiB/s (664kB/s)(4976KiB/7676msec) slat (usec): min=1162, max=11908, avg=3329.37, stdev=1278.57 clat (usec): min=18797, max=81794, avg=45521.76, stdev=8776.35 lat (usec): min=26782, max=85975, avg=48854.41, stdev=9061.46 clat percentiles (usec): | 1.00th=[28967], 5.00th=[33424], 10.00th=[35390], 20.00th=[38536], | 30.00th=[40633], 40.00th=[42730], 50.00th=[44303], 60.00th=[46400], | 70.00th=[49021], 80.00th=[52167], 90.00th=[56886], 95.00th=[61604], | 99.00th=[72877], 99.50th=[73925], 99.90th=[76022], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 560, max= 776, per=24.78%, avg=647.33, stdev=56.22, samples=15 iops : min= 140, max= 194, avg=161.80, stdev=14.06, samples=15 write: IOPS=171, BW=686KiB/s (702kB/s)(5264KiB/7676msec); 0 zone resets slat (usec): min=55, max=2505, avg=182.39, stdev=123.07 clat (usec): min=28293, max=81452, avg=46469.42, stdev=8409.35 lat (usec): min=28447, max=81726, avg=46652.98, stdev=8419.21 clat percentiles (usec): | 1.00th=[31327], 5.00th=[35390], 10.00th=[36963], 20.00th=[39584], | 30.00th=[41681], 40.00th=[43254], 50.00th=[44827], 60.00th=[46924], | 70.00th=[49546], 80.00th=[52691], 90.00th=[57934], 95.00th=[63177], | 99.00th=[70779], 99.50th=[76022], 99.90th=[77071], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min= 592, max= 792, per=25.24%, avg=686.33, stdev=56.30, samples=15 iops : min= 148, max= 198, avg=171.53, stdev=14.07, samples=15 lat (msec) : 20=0.04%, 50=72.50%, 100=27.46% cpu : usr=0.48%, sys=5.62%, ctx=4409, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154517: Fri Jan 30 03:00:33 2026 read: IOPS=162, BW=649KiB/s (665kB/s)(4984KiB/7680msec) slat (usec): min=1079, max=9767, avg=3292.40, stdev=1269.28 clat (usec): min=22023, max=81806, avg=45627.13, stdev=8913.23 lat (usec): min=25339, max=84643, avg=48922.30, stdev=9205.49 clat percentiles (usec): | 1.00th=[28705], 5.00th=[33817], 10.00th=[35390], 20.00th=[38011], | 30.00th=[40633], 40.00th=[42206], 50.00th=[44303], 60.00th=[46400], | 70.00th=[49021], 80.00th=[52167], 90.00th=[57934], 95.00th=[62129], | 99.00th=[71828], 99.50th=[76022], 99.90th=[78119], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min= 526, max= 744, per=24.81%, avg=648.07, stdev=72.41, samples=15 iops : min= 131, max= 186, avg=161.93, stdev=18.21, samples=15 write: IOPS=171, BW=684KiB/s (701kB/s)(5256KiB/7680msec); 0 zone resets slat (usec): min=57, max=1518, avg=181.60, stdev=118.07 clat (usec): min=15639, max=90512, avg=46345.56, stdev=8572.35 lat (usec): min=15834, max=90610, avg=46528.30, stdev=8576.24 clat percentiles (usec): | 1.00th=[30278], 5.00th=[34866], 10.00th=[36963], 20.00th=[39060], | 30.00th=[41681], 40.00th=[43254], 50.00th=[45351], 60.00th=[46924], | 70.00th=[49021], 80.00th=[52691], 90.00th=[57410], 95.00th=[61604], | 99.00th=[71828], 99.50th=[78119], 99.90th=[86508], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 576, max= 806, per=25.05%, avg=681.00, stdev=63.25, samples=15 iops : min= 144, max= 201, avg=170.20, stdev=15.75, samples=15 lat (msec) : 20=0.04%, 50=72.97%, 100=26.99% cpu : usr=0.39%, sys=5.65%, ctx=4398, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=154518: Fri Jan 30 03:00:33 2026 read: IOPS=164, BW=659KiB/s (674kB/s)(5060KiB/7683msec) slat (usec): min=1311, max=10049, avg=3306.76, stdev=1269.06 clat (usec): min=11450, max=79112, avg=45251.31, stdev=9003.20 lat (usec): min=13203, max=86141, avg=48561.48, stdev=9331.75 clat percentiles (usec): | 1.00th=[27919], 5.00th=[32900], 10.00th=[34866], 20.00th=[38011], | 30.00th=[40109], 40.00th=[41681], 50.00th=[44303], 60.00th=[46400], | 70.00th=[49021], 80.00th=[52167], 90.00th=[57410], 95.00th=[62129], | 99.00th=[70779], 99.50th=[72877], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 560, max= 728, per=25.11%, avg=655.80, stdev=58.29, samples=15 iops : min= 140, max= 182, avg=163.87, stdev=14.62, samples=15 write: IOPS=168, BW=674KiB/s (690kB/s)(5180KiB/7683msec); 0 zone resets slat (usec): min=58, max=2818, avg=182.72, stdev=146.23 clat (usec): min=17786, max=89819, avg=46719.43, stdev=9216.45 lat (usec): min=17933, max=89963, avg=46903.37, stdev=9230.14 clat percentiles (usec): | 1.00th=[29230], 5.00th=[34341], 10.00th=[36439], 20.00th=[39060], | 30.00th=[41157], 40.00th=[43254], 50.00th=[45351], 60.00th=[47449], | 70.00th=[50070], 80.00th=[53216], 90.00th=[60031], 95.00th=[64226], | 99.00th=[71828], 99.50th=[76022], 99.90th=[88605], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 560, max= 792, per=24.60%, avg=669.00, stdev=60.17, samples=15 iops : min= 140, max= 198, avg=167.20, stdev=15.05, samples=15 lat (msec) : 20=0.20%, 50=71.37%, 100=28.44% cpu : usr=0.53%, sys=5.41%, ctx=4444, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2612KiB/s (2675kB/s), 648KiB/s-661KiB/s (664kB/s-676kB/s), io=19.6MiB (20.5MB), run=7642-7683msec WRITE: bw=2719KiB/s (2785kB/s), 674KiB/s-686KiB/s (690kB/s-702kB/s), io=20.4MiB (21.4MB), run=7642-7683msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=154522: Fri Jan 30 03:00:37 2026 read: IOPS=7, BW=305MiB/s (320MB/s)(40.0MiB/131msec) slat (nsec): min=13868k, max=13868k, avg=13868093.00, stdev= 0.00 clat (nsec): min=107888k, max=107888k, avg=107888365.00, stdev= 0.00 lat (nsec): min=121761k, max=121761k, avg=121761148.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 108], 10.00th=[ 108], 20.00th=[ 108], | 30.00th=[ 108], 40.00th=[ 108], 50.00th=[ 108], 60.00th=[ 108], | 70.00th=[ 108], 80.00th=[ 108], 90.00th=[ 108], 95.00th=[ 108], | 99.00th=[ 108], 99.50th=[ 108], 99.90th=[ 108], 99.95th=[ 108], | 99.99th=[ 108] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=9.23%, ctx=11, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=305MiB/s (320MB/s), 305MiB/s-305MiB/s (320MB/s-320MB/s), io=40.0MiB (41.9MB), run=131-131msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout