== sanity test 398c: run fio to test AIO ================= 14:30:33 (1788892233) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.159073 s, 264 MB/s osc.lustre-OST0000-osc-ffff9fbac91ff800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=192586: Tue Sep 8 14:30:44 2026 write: IOPS=342, BW=1370KiB/s (1403kB/s)(10.0MiB/7475msec); 0 zone resets slat (usec): min=20, max=2838, avg=126.73, stdev=123.16 clat (msec): min=20, max=105, avg=46.45, stdev=10.09 lat (msec): min=20, max=106, avg=46.58, stdev=10.09 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 34], 10.00th=[ 37], 20.00th=[ 40], | 30.00th=[ 41], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 47], | 70.00th=[ 50], 80.00th=[ 54], 90.00th=[ 61], 95.00th=[ 65], | 99.00th=[ 82], 99.50th=[ 84], 99.90th=[ 96], 99.95th=[ 96], | 99.99th=[ 107] bw ( KiB/s): min= 992, max= 1752, per=25.06%, avg=1370.21, stdev=186.82, samples=14 iops : min= 248, max= 438, avg=342.50, stdev=46.72, samples=14 lat (msec) : 50=71.41%, 100=28.55%, 250=0.04% cpu : usr=0.48%, sys=3.49%, ctx=2724, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=192587: Tue Sep 8 14:30:44 2026 write: IOPS=341, BW=1367KiB/s (1400kB/s)(10.0MiB/7490msec); 0 zone resets slat (usec): min=21, max=3367, avg=125.30, stdev=141.87 clat (usec): min=10403, max=84725, avg=46265.68, stdev=9755.86 lat (usec): min=10623, max=84827, avg=46392.31, stdev=9764.33 clat percentiles (usec): | 1.00th=[25560], 5.00th=[33162], 10.00th=[36439], 20.00th=[39060], | 30.00th=[40633], 40.00th=[42730], 50.00th=[44827], 60.00th=[46924], | 70.00th=[49546], 80.00th=[53216], 90.00th=[59507], 95.00th=[64226], | 99.00th=[76022], 99.50th=[79168], 99.90th=[83362], 99.95th=[83362], | 99.99th=[84411] bw ( KiB/s): min= 992, max= 1776, per=24.91%, avg=1362.21, stdev=186.88, samples=14 iops : min= 248, max= 444, avg=340.50, stdev=46.74, samples=14 lat (msec) : 20=0.27%, 50=70.86%, 100=28.87% cpu : usr=0.55%, sys=3.36%, ctx=2758, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=192588: Tue Sep 8 14:30:44 2026 write: IOPS=342, BW=1369KiB/s (1401kB/s)(10.0MiB/7482msec); 0 zone resets slat (usec): min=24, max=4073, avg=131.25, stdev=153.16 clat (usec): min=8353, max=87663, avg=46241.73, stdev=9864.67 lat (usec): min=8514, max=87736, avg=46374.22, stdev=9876.93 clat percentiles (usec): | 1.00th=[27395], 5.00th=[33424], 10.00th=[36439], 20.00th=[38536], | 30.00th=[40633], 40.00th=[42206], 50.00th=[44827], 60.00th=[46924], | 70.00th=[49021], 80.00th=[52691], 90.00th=[60031], 95.00th=[64750], | 99.00th=[77071], 99.50th=[80217], 99.90th=[87557], 99.95th=[87557], | 99.99th=[87557] bw ( KiB/s): min= 952, max= 1776, per=24.99%, avg=1366.36, stdev=193.14, samples=14 iops : min= 238, max= 444, avg=341.50, stdev=48.30, samples=14 lat (msec) : 10=0.08%, 20=0.20%, 50=72.46%, 100=27.27% cpu : usr=0.57%, sys=3.72%, ctx=2839, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=192589: Tue Sep 8 14:30:44 2026 write: IOPS=343, BW=1375KiB/s (1408kB/s)(10.0MiB/7445msec); 0 zone resets slat (usec): min=20, max=3429, avg=125.06, stdev=114.61 clat (usec): min=8554, max=89906, avg=46315.96, stdev=10101.55 lat (usec): min=8757, max=90200, avg=46442.34, stdev=10113.48 clat percentiles (usec): | 1.00th=[24773], 5.00th=[33162], 10.00th=[35914], 20.00th=[38536], | 30.00th=[40633], 40.00th=[42730], 50.00th=[44827], 60.00th=[46924], | 70.00th=[49021], 80.00th=[53216], 90.00th=[60556], 95.00th=[64750], | 99.00th=[80217], 99.50th=[83362], 99.90th=[85459], 99.95th=[89654], | 99.99th=[89654] bw ( KiB/s): min= 984, max= 1760, per=25.14%, avg=1374.43, stdev=190.61, samples=14 iops : min= 246, max= 440, avg=343.57, stdev=47.67, samples=14 lat (msec) : 10=0.04%, 20=0.43%, 50=72.07%, 100=27.46% cpu : usr=0.55%, sys=3.40%, ctx=2884, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=5469KiB/s (5600kB/s), 1367KiB/s-1375KiB/s (1400kB/s-1408kB/s), io=40.0MiB (41.9MB), run=7445-7490msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=192600: Tue Sep 8 14:30:56 2026 read: IOPS=155, BW=623KiB/s (638kB/s)(5048KiB/8106msec) slat (usec): min=31, max=13321, avg=3317.87, stdev=1253.77 clat (usec): min=11027, max=90755, avg=48287.66, stdev=10737.81 lat (usec): min=11207, max=96564, avg=51609.22, stdev=11163.47 clat percentiles (usec): | 1.00th=[23725], 5.00th=[32637], 10.00th=[35914], 20.00th=[39584], | 30.00th=[42730], 40.00th=[44827], 50.00th=[47449], 60.00th=[50070], | 70.00th=[53216], 80.00th=[56361], 90.00th=[61080], 95.00th=[67634], | 99.00th=[81265], 99.50th=[84411], 99.90th=[88605], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min= 376, max= 728, per=25.49%, avg=624.60, stdev=92.47, samples=15 iops : min= 94, max= 182, avg=156.07, stdev=23.11, samples=15 write: IOPS=160, BW=641KiB/s (656kB/s)(5192KiB/8106msec); 0 zone resets slat (usec): min=33, max=3853, avg=133.01, stdev=171.85 clat (usec): min=20428, max=97566, avg=49429.27, stdev=10268.26 lat (usec): min=20613, max=97680, avg=49563.67, stdev=10284.91 clat percentiles (usec): | 1.00th=[29754], 5.00th=[34866], 10.00th=[37487], 20.00th=[41157], | 30.00th=[43254], 40.00th=[45876], 50.00th=[47973], 60.00th=[50594], | 70.00th=[53216], 80.00th=[56886], 90.00th=[62129], 95.00th=[67634], | 99.00th=[80217], 99.50th=[88605], 99.90th=[96994], 99.95th=[98042], | 99.99th=[98042] bw ( KiB/s): min= 496, max= 800, per=25.17%, avg=641.87, stdev=98.72, samples=15 iops : min= 124, max= 200, avg=160.40, stdev=24.69, samples=15 lat (msec) : 20=0.39%, 50=57.38%, 100=42.23% cpu : usr=0.53%, sys=4.49%, ctx=4751, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=192601: Tue Sep 8 14:30:56 2026 read: IOPS=152, BW=609KiB/s (624kB/s)(4976KiB/8172msec) slat (usec): min=1150, max=12354, avg=3339.52, stdev=1146.86 clat (usec): min=15595, max=96180, avg=48451.57, stdev=10012.44 lat (msec): min=20, max=100, avg=51.79, stdev=10.32 clat percentiles (usec): | 1.00th=[30278], 5.00th=[33817], 10.00th=[36439], 20.00th=[40109], | 30.00th=[42730], 40.00th=[45351], 50.00th=[47973], 60.00th=[50070], | 70.00th=[52691], 80.00th=[55837], 90.00th=[61080], 95.00th=[64750], | 99.00th=[77071], 99.50th=[87557], 99.90th=[94897], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 416, max= 752, per=24.81%, avg=607.94, stdev=90.68, samples=16 iops : min= 104, max= 188, avg=151.88, stdev=22.63, samples=16 write: IOPS=161, BW=644KiB/s (660kB/s)(5264KiB/8172msec); 0 zone resets slat (usec): min=26, max=4342, avg=136.65, stdev=193.37 clat (msec): min=9, max=108, avg=49.78, stdev=10.95 lat (msec): min=9, max=108, avg=49.92, stdev=10.97 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 35], 10.00th=[ 37], 20.00th=[ 41], | 30.00th=[ 44], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 54], 80.00th=[ 58], 90.00th=[ 63], 95.00th=[ 68], | 99.00th=[ 88], 99.50th=[ 94], 99.90th=[ 105], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min= 512, max= 880, per=25.26%, avg=644.06, stdev=101.58, samples=16 iops : min= 128, max= 220, avg=160.94, stdev=25.42, samples=16 lat (msec) : 10=0.04%, 20=0.16%, 50=57.50%, 100=42.23%, 250=0.08% cpu : usr=0.55%, sys=4.38%, ctx=4642, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=192602: Tue Sep 8 14:30:56 2026 read: IOPS=152, BW=610KiB/s (625kB/s)(4984KiB/8168msec) slat (usec): min=1130, max=11296, avg=3338.54, stdev=1204.46 clat (msec): min=6, max=103, avg=48.07, stdev=10.69 lat (msec): min=12, max=109, avg=51.41, stdev=11.11 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 34], 10.00th=[ 37], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 56], 90.00th=[ 61], 95.00th=[ 67], | 99.00th=[ 85], 99.50th=[ 90], 99.90th=[ 97], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min= 448, max= 832, per=24.95%, avg=611.25, stdev=108.06, samples=16 iops : min= 112, max= 208, avg=152.75, stdev=27.02, samples=16 write: IOPS=160, BW=643KiB/s (659kB/s)(5256KiB/8168msec); 0 zone resets slat (usec): min=24, max=1502, avg=127.21, stdev=105.91 clat (usec): min=27226, max=96596, avg=49972.80, stdev=10085.01 lat (usec): min=27299, max=96654, avg=50101.35, stdev=10088.17 clat percentiles (usec): | 1.00th=[31589], 5.00th=[36439], 10.00th=[38536], 20.00th=[41681], | 30.00th=[44303], 40.00th=[46400], 50.00th=[49021], 60.00th=[51643], | 70.00th=[54264], 80.00th=[56886], 90.00th=[62653], 95.00th=[67634], | 99.00th=[85459], 99.50th=[90702], 99.90th=[95945], 99.95th=[96994], | 99.99th=[96994] bw ( KiB/s): min= 488, max= 742, per=25.16%, avg=641.62, stdev=68.72, samples=16 iops : min= 122, max= 185, avg=160.31, stdev=17.14, samples=16 lat (msec) : 10=0.04%, 20=0.08%, 50=58.32%, 100=41.52%, 250=0.04% cpu : usr=0.66%, sys=4.25%, ctx=4757, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=192603: Tue Sep 8 14:30:56 2026 read: IOPS=154, BW=618KiB/s (633kB/s)(5060KiB/8191msec) slat (usec): min=68, max=51502, avg=3383.61, stdev=2174.95 clat (msec): min=6, max=126, avg=48.47, stdev=11.87 lat (msec): min=9, max=130, avg=51.86, stdev=12.29 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 33], 10.00th=[ 36], 20.00th=[ 40], | 30.00th=[ 43], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 67], | 99.00th=[ 86], 99.50th=[ 113], 99.90th=[ 115], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 440, max= 784, per=25.04%, avg=613.50, stdev=79.73, samples=16 iops : min= 110, max= 196, avg=153.31, stdev=19.95, samples=16 write: IOPS=158, BW=632KiB/s (648kB/s)(5180KiB/8191msec); 0 zone resets slat (usec): min=35, max=2488, avg=131.11, stdev=126.71 clat (msec): min=10, max=120, avg=49.76, stdev=11.76 lat (msec): min=10, max=120, avg=49.89, stdev=11.77 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 54], 80.00th=[ 57], 90.00th=[ 63], 95.00th=[ 68], | 99.00th=[ 96], 99.50th=[ 111], 99.90th=[ 121], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 472, max= 856, per=24.58%, avg=626.81, stdev=96.83, samples=16 iops : min= 118, max= 214, avg=156.62, stdev=24.21, samples=16 lat (msec) : 10=0.08%, 20=0.74%, 50=56.60%, 100=41.80%, 250=0.78% cpu : usr=0.51%, sys=4.33%, ctx=4653, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2450KiB/s (2509kB/s), 609KiB/s-623KiB/s (624kB/s-638kB/s), io=19.6MiB (20.5MB), run=8106-8191msec WRITE: bw=2551KiB/s (2612kB/s), 632KiB/s-644KiB/s (648kB/s-660kB/s), io=20.4MiB (21.4MB), run=8106-8191msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=192608: Tue Sep 8 14:31:00 2026 read: IOPS=7, BW=303MiB/s (318MB/s)(40.0MiB/132msec) slat (nsec): min=10833k, max=10833k, avg=10833060.00, stdev= 0.00 clat (nsec): min=114507k, max=114507k, avg=114507476.00, stdev= 0.00 lat (nsec): min=125347k, max=125347k, avg=125346526.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 115], 10.00th=[ 115], 20.00th=[ 115], | 30.00th=[ 115], 40.00th=[ 115], 50.00th=[ 115], 60.00th=[ 115], | 70.00th=[ 115], 80.00th=[ 115], 90.00th=[ 115], 95.00th=[ 115], | 99.00th=[ 115], 99.50th=[ 115], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 115] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.63%, ctx=8, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=303MiB/s (318MB/s), 303MiB/s-303MiB/s (318MB/s-318MB/s), io=40.0MiB (41.9MB), run=132-132msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout