== sanity test 398c: run fio to test AIO ================= 17:33:17 (1764109997) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.163134 s, 257 MB/s osc.lustre-OST0000-osc-ffff89dc02fa6800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=191388: Tue Nov 25 17:33:30 2025 write: IOPS=285, BW=1142KiB/s (1170kB/s)(10.0MiB/8964msec); 0 zone resets slat (usec): min=29, max=30045, avg=123.41, stdev=597.40 clat (msec): min=8, max=118, avg=55.46, stdev=13.17 lat (msec): min=8, max=118, avg=55.58, stdev=13.15 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 46], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 61], 80.00th=[ 66], 90.00th=[ 73], 95.00th=[ 81], | 99.00th=[ 99], 99.50th=[ 101], 99.90th=[ 103], 99.95th=[ 104], | 99.99th=[ 120] bw ( KiB/s): min= 760, max= 1424, per=25.19%, avg=1151.12, stdev=196.38, samples=17 iops : min= 190, max= 356, avg=287.71, stdev=49.16, samples=17 lat (msec) : 10=0.04%, 20=0.16%, 50=41.05%, 100=58.44%, 250=0.31% cpu : usr=0.48%, sys=2.86%, ctx=2788, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=191389: Tue Nov 25 17:33:30 2025 write: IOPS=286, BW=1147KiB/s (1174kB/s)(10.0MiB/8930msec); 0 zone resets slat (usec): min=21, max=977, avg=115.23, stdev=85.54 clat (msec): min=5, max=137, avg=55.46, stdev=13.60 lat (msec): min=5, max=137, avg=55.58, stdev=13.60 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 45], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 61], 80.00th=[ 66], 90.00th=[ 73], 95.00th=[ 84], | 99.00th=[ 99], 99.50th=[ 103], 99.90th=[ 129], 99.95th=[ 129], | 99.99th=[ 138] bw ( KiB/s): min= 736, max= 1424, per=25.30%, avg=1156.00, stdev=202.20, samples=17 iops : min= 184, max= 356, avg=288.94, stdev=50.60, samples=17 lat (msec) : 10=0.04%, 50=41.80%, 100=57.50%, 250=0.66% cpu : usr=0.44%, sys=2.70%, ctx=2717, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=191390: Tue Nov 25 17:33:30 2025 write: IOPS=287, BW=1148KiB/s (1176kB/s)(10.0MiB/8917msec); 0 zone resets slat (usec): min=25, max=1158, avg=111.52, stdev=81.55 clat (msec): min=11, max=129, avg=55.52, stdev=13.27 lat (msec): min=11, max=129, avg=55.63, stdev=13.27 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 41], 10.00th=[ 43], 20.00th=[ 46], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 61], 80.00th=[ 66], 90.00th=[ 72], 95.00th=[ 83], | 99.00th=[ 99], 99.50th=[ 102], 99.90th=[ 130], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 736, max= 1440, per=25.30%, avg=1155.94, stdev=198.82, samples=17 iops : min= 184, max= 360, avg=288.94, stdev=49.75, samples=17 lat (msec) : 20=0.08%, 50=41.91%, 100=57.34%, 250=0.66% cpu : usr=0.50%, sys=2.45%, ctx=2693, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=191391: Tue Nov 25 17:33:30 2025 write: IOPS=285, BW=1142KiB/s (1170kB/s)(10.0MiB/8964msec); 0 zone resets slat (usec): min=32, max=26082, avg=123.67, stdev=519.44 clat (msec): min=8, max=102, avg=55.46, stdev=12.92 lat (msec): min=9, max=102, avg=55.59, stdev=12.90 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 46], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 55], | 70.00th=[ 61], 80.00th=[ 66], 90.00th=[ 73], 95.00th=[ 82], | 99.00th=[ 97], 99.50th=[ 99], 99.90th=[ 101], 99.95th=[ 103], | 99.99th=[ 104] bw ( KiB/s): min= 704, max= 1440, per=25.15%, avg=1149.00, stdev=206.98, samples=17 iops : min= 176, max= 360, avg=287.18, stdev=51.81, samples=17 lat (msec) : 10=0.04%, 20=0.04%, 50=41.33%, 100=58.40%, 250=0.20% cpu : usr=0.38%, sys=2.92%, ctx=2775, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=4569KiB/s (4679kB/s), 1142KiB/s-1148KiB/s (1170kB/s-1176kB/s), io=40.0MiB (41.9MB), run=8917-8964msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=191402: Tue Nov 25 17:33:42 2025 read: IOPS=156, BW=624KiB/s (639kB/s)(5048KiB/8086msec) slat (usec): min=31, max=12078, avg=3347.18, stdev=1144.78 clat (usec): min=12483, max=88802, avg=48070.75, stdev=10204.58 lat (usec): min=12615, max=92063, avg=51421.12, stdev=10617.41 clat percentiles (usec): | 1.00th=[26608], 5.00th=[34866], 10.00th=[36963], 20.00th=[40109], | 30.00th=[42206], 40.00th=[44303], 50.00th=[46400], 60.00th=[48497], | 70.00th=[51643], 80.00th=[55837], 90.00th=[61604], 95.00th=[67634], | 99.00th=[77071], 99.50th=[81265], 99.90th=[84411], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 512, max= 720, per=25.38%, avg=629.27, stdev=71.26, samples=15 iops : min= 128, max= 180, avg=157.20, stdev=17.87, samples=15 write: IOPS=160, BW=642KiB/s (658kB/s)(5192KiB/8086msec); 0 zone resets slat (usec): min=35, max=1967, avg=111.97, stdev=97.21 clat (usec): min=7770, max=84805, avg=48972.87, stdev=9499.32 lat (usec): min=7872, max=84863, avg=49086.02, stdev=9505.61 clat percentiles (usec): | 1.00th=[31589], 5.00th=[35914], 10.00th=[38536], 20.00th=[41157], | 30.00th=[43254], 40.00th=[45876], 50.00th=[47973], 60.00th=[49546], | 70.00th=[52167], 80.00th=[55837], 90.00th=[62653], 95.00th=[67634], | 99.00th=[77071], 99.50th=[79168], 99.90th=[83362], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min= 439, max= 824, per=25.11%, avg=648.20, stdev=110.65, samples=15 iops : min= 109, max= 206, avg=161.93, stdev=27.80, samples=15 lat (msec) : 10=0.04%, 20=0.35%, 50=62.54%, 100=37.07% cpu : usr=0.58%, sys=4.00%, ctx=4619, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=191403: Tue Nov 25 17:33:42 2025 read: IOPS=153, BW=615KiB/s (630kB/s)(4976KiB/8094msec) slat (usec): min=57, max=10437, avg=3428.68, stdev=1156.87 clat (usec): min=12784, max=89136, avg=48050.85, stdev=9707.75 lat (usec): min=13039, max=93781, avg=51482.74, stdev=10046.81 clat percentiles (usec): | 1.00th=[28443], 5.00th=[34866], 10.00th=[37487], 20.00th=[40633], | 30.00th=[42730], 40.00th=[44827], 50.00th=[46400], 60.00th=[49021], | 70.00th=[51643], 80.00th=[55313], 90.00th=[60556], 95.00th=[65799], | 99.00th=[76022], 99.50th=[78119], 99.90th=[87557], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 464, max= 744, per=24.87%, avg=616.53, stdev=82.61, samples=15 iops : min= 116, max= 186, avg=154.00, stdev=20.74, samples=15 write: IOPS=162, BW=650KiB/s (666kB/s)(5264KiB/8094msec); 0 zone resets slat (usec): min=33, max=2098, avg=117.32, stdev=107.44 clat (usec): min=22161, max=89007, avg=49033.51, stdev=9361.82 lat (usec): min=22262, max=89114, avg=49152.82, stdev=9361.74 clat percentiles (usec): | 1.00th=[31851], 5.00th=[36439], 10.00th=[38536], 20.00th=[41681], | 30.00th=[43254], 40.00th=[45351], 50.00th=[47449], 60.00th=[50070], | 70.00th=[52691], 80.00th=[56361], 90.00th=[62129], 95.00th=[66323], | 99.00th=[76022], 99.50th=[78119], 99.90th=[82314], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min= 518, max= 816, per=25.53%, avg=658.93, stdev=96.93, samples=15 iops : min= 129, max= 204, avg=164.60, stdev=24.34, samples=15 lat (msec) : 20=0.20%, 50=62.07%, 100=37.73% cpu : usr=0.52%, sys=4.07%, ctx=4580, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=191404: Tue Nov 25 17:33:42 2025 read: IOPS=154, BW=618KiB/s (632kB/s)(4984KiB/8071msec) slat (usec): min=34, max=8721, avg=3389.15, stdev=1130.71 clat (usec): min=8157, max=96660, avg=47822.09, stdev=9839.90 lat (msec): min=12, max=100, avg=51.21, stdev=10.18 clat percentiles (usec): | 1.00th=[29754], 5.00th=[34341], 10.00th=[36963], 20.00th=[40109], | 30.00th=[42206], 40.00th=[44303], 50.00th=[46400], 60.00th=[49021], | 70.00th=[51643], 80.00th=[54789], 90.00th=[60031], 95.00th=[66323], | 99.00th=[77071], 99.50th=[80217], 99.90th=[84411], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 416, max= 744, per=25.29%, avg=626.93, stdev=91.99, samples=15 iops : min= 104, max= 186, avg=156.60, stdev=23.09, samples=15 write: IOPS=162, BW=651KiB/s (667kB/s)(5256KiB/8071msec); 0 zone resets slat (usec): min=33, max=3073, avg=115.52, stdev=115.34 clat (usec): min=15236, max=89703, avg=49434.23, stdev=10167.99 lat (usec): min=15433, max=89858, avg=49550.97, stdev=10169.69 clat percentiles (usec): | 1.00th=[29754], 5.00th=[36439], 10.00th=[38536], 20.00th=[41681], | 30.00th=[43779], 40.00th=[45351], 50.00th=[47449], 60.00th=[50070], | 70.00th=[53216], 80.00th=[56886], 90.00th=[63177], 95.00th=[68682], | 99.00th=[80217], 99.50th=[84411], 99.90th=[89654], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 471, max= 776, per=25.27%, avg=652.27, stdev=90.16, samples=15 iops : min= 117, max= 194, avg=162.93, stdev=22.73, samples=15 lat (msec) : 10=0.04%, 20=0.31%, 50=61.29%, 100=38.36% cpu : usr=0.72%, sys=3.95%, ctx=4552, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=191405: Tue Nov 25 17:33:42 2025 read: IOPS=156, BW=625KiB/s (640kB/s)(5060KiB/8094msec) slat (usec): min=32, max=10544, avg=3415.78, stdev=1155.74 clat (usec): min=7376, max=89511, avg=47825.93, stdev=9991.00 lat (usec): min=11790, max=93166, avg=51246.03, stdev=10367.28 clat percentiles (usec): | 1.00th=[27395], 5.00th=[33817], 10.00th=[36439], 20.00th=[40109], | 30.00th=[42730], 40.00th=[44827], 50.00th=[46400], 60.00th=[48497], | 70.00th=[51643], 80.00th=[55313], 90.00th=[61604], 95.00th=[65799], | 99.00th=[76022], 99.50th=[77071], 99.90th=[86508], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 447, max= 720, per=25.34%, avg=628.20, stdev=82.13, samples=15 iops : min= 111, max= 180, avg=156.87, stdev=20.69, samples=15 write: IOPS=159, BW=640KiB/s (655kB/s)(5180KiB/8094msec); 0 zone resets slat (usec): min=33, max=2827, avg=110.82, stdev=102.97 clat (usec): min=15847, max=95933, avg=49290.55, stdev=9792.27 lat (usec): min=15946, max=95988, avg=49402.65, stdev=9786.87 clat percentiles (usec): | 1.00th=[31589], 5.00th=[35914], 10.00th=[38536], 20.00th=[41681], | 30.00th=[43779], 40.00th=[45351], 50.00th=[47449], 60.00th=[50070], | 70.00th=[53216], 80.00th=[56361], 90.00th=[62129], 95.00th=[67634], | 99.00th=[78119], 99.50th=[83362], 99.90th=[91751], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 480, max= 784, per=25.10%, avg=647.87, stdev=87.59, samples=15 iops : min= 120, max= 196, avg=161.80, stdev=22.02, samples=15 lat (msec) : 10=0.04%, 20=0.31%, 50=61.84%, 100=37.81% cpu : usr=0.67%, sys=3.95%, ctx=4601, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2479KiB/s (2539kB/s), 615KiB/s-625KiB/s (630kB/s-640kB/s), io=19.6MiB (20.5MB), run=8071-8094msec WRITE: bw=2581KiB/s (2643kB/s), 640KiB/s-651KiB/s (655kB/s-667kB/s), io=20.4MiB (21.4MB), run=8071-8094msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=191410: Tue Nov 25 17:33:46 2025 read: IOPS=7, BW=282MiB/s (295MB/s)(40.0MiB/142msec) slat (nsec): min=11303k, max=11303k, avg=11302551.00, stdev= 0.00 clat (nsec): min=122191k, max=122191k, avg=122191233.00, stdev= 0.00 lat (nsec): min=133523k, max=133523k, avg=133522646.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 123], 10.00th=[ 123], 20.00th=[ 123], | 30.00th=[ 123], 40.00th=[ 123], 50.00th=[ 123], 60.00th=[ 123], | 70.00th=[ 123], 80.00th=[ 123], 90.00th=[ 123], 95.00th=[ 123], | 99.00th=[ 123], 99.50th=[ 123], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.09%, ctx=12, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=282MiB/s (295MB/s), 282MiB/s-282MiB/s (295MB/s-295MB/s), io=40.0MiB (41.9MB), run=142-142msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout