== sanity test 398c: run fio to test AIO ================= 17:44:44 (1786484684) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.164284 s, 255 MB/s osc.lustre-OST0000-osc-ffff8bf884bf3000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179840: Tue Aug 11 17:44:56 2026 write: IOPS=326, BW=1304KiB/s (1336kB/s)(10.0MiB/7851msec); 0 zone resets slat (usec): min=35, max=2390, avg=205.10, stdev=159.24 clat (usec): min=12021, max=77310, avg=48788.12, stdev=9063.83 lat (usec): min=12161, max=77436, avg=48994.70, stdev=9070.77 clat percentiles (usec): | 1.00th=[33162], 5.00th=[35390], 10.00th=[38011], 20.00th=[41157], | 30.00th=[43779], 40.00th=[45876], 50.00th=[47449], 60.00th=[50070], | 70.00th=[52691], 80.00th=[56361], 90.00th=[60556], 95.00th=[66323], | 99.00th=[73925], 99.50th=[74974], 99.90th=[76022], 99.95th=[76022], | 99.99th=[77071] bw ( KiB/s): min= 1048, max= 1520, per=24.94%, avg=1295.67, stdev=132.85, samples=15 iops : min= 262, max= 380, avg=323.87, stdev=33.20, samples=15 lat (msec) : 20=0.04%, 50=60.16%, 100=39.80% cpu : usr=0.39%, sys=4.69%, ctx=2532, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179841: Tue Aug 11 17:44:56 2026 write: IOPS=324, BW=1300KiB/s (1331kB/s)(10.0MiB/7879msec); 0 zone resets slat (usec): min=35, max=27357, avg=209.19, stdev=564.62 clat (usec): min=8974, max=76863, avg=48825.18, stdev=9047.97 lat (usec): min=9147, max=77899, avg=49035.77, stdev=9033.83 clat percentiles (usec): | 1.00th=[33162], 5.00th=[35914], 10.00th=[38536], 20.00th=[41157], | 30.00th=[43254], 40.00th=[45876], 50.00th=[47449], 60.00th=[50070], | 70.00th=[53216], 80.00th=[55313], 90.00th=[61080], 95.00th=[67634], | 99.00th=[72877], 99.50th=[74974], 99.90th=[76022], 99.95th=[76022], | 99.99th=[77071] bw ( KiB/s): min= 1040, max= 1528, per=24.89%, avg=1293.07, stdev=134.45, samples=15 iops : min= 260, max= 382, avg=323.20, stdev=33.63, samples=15 lat (msec) : 10=0.04%, 20=0.08%, 50=58.79%, 100=41.09% cpu : usr=0.50%, sys=4.54%, ctx=2697, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179842: Tue Aug 11 17:44:56 2026 write: IOPS=324, BW=1300KiB/s (1331kB/s)(10.0MiB/7878msec); 0 zone resets slat (usec): min=35, max=1734, avg=222.61, stdev=161.47 clat (usec): min=6496, max=79578, avg=48611.60, stdev=9160.28 lat (usec): min=6741, max=79649, avg=48835.84, stdev=9168.18 clat percentiles (usec): | 1.00th=[31589], 5.00th=[35390], 10.00th=[38011], 20.00th=[41157], | 30.00th=[43254], 40.00th=[45351], 50.00th=[46924], 60.00th=[49546], | 70.00th=[52691], 80.00th=[56361], 90.00th=[60556], 95.00th=[64750], | 99.00th=[74974], 99.50th=[77071], 99.90th=[79168], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min= 1072, max= 1528, per=24.82%, avg=1289.80, stdev=133.97, samples=15 iops : min= 268, max= 382, avg=322.40, stdev=33.50, samples=15 lat (msec) : 10=0.04%, 20=0.04%, 50=61.48%, 100=38.44% cpu : usr=0.39%, sys=5.31%, ctx=2648, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179843: Tue Aug 11 17:44:56 2026 write: IOPS=324, BW=1299KiB/s (1330kB/s)(10.0MiB/7882msec); 0 zone resets slat (usec): min=35, max=1708, avg=212.26, stdev=158.35 clat (usec): min=3375, max=81870, avg=48611.33, stdev=9132.50 lat (usec): min=4419, max=81939, avg=48824.80, stdev=9141.53 clat percentiles (usec): | 1.00th=[32113], 5.00th=[35390], 10.00th=[38011], 20.00th=[41157], | 30.00th=[43254], 40.00th=[45351], 50.00th=[47449], 60.00th=[49546], | 70.00th=[52167], 80.00th=[55837], 90.00th=[60556], 95.00th=[64750], | 99.00th=[73925], 99.50th=[74974], 99.90th=[81265], 99.95th=[81265], | 99.99th=[82314] bw ( KiB/s): min= 1088, max= 1496, per=24.76%, avg=1286.73, stdev=127.60, samples=15 iops : min= 272, max= 374, avg=321.67, stdev=31.89, samples=15 lat (msec) : 4=0.04%, 10=0.04%, 50=61.05%, 100=38.87% cpu : usr=0.39%, sys=4.73%, ctx=2616, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=5197KiB/s (5321kB/s), 1299KiB/s-1304KiB/s (1330kB/s-1336kB/s), io=40.0MiB (41.9MB), run=7851-7882msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179854: Tue Aug 11 17:45:08 2026 read: IOPS=146, BW=585KiB/s (599kB/s)(5048KiB/8624msec) slat (usec): min=1194, max=10155, avg=3558.02, stdev=1274.19 clat (usec): min=24398, max=95667, avg=51455.32, stdev=11352.63 lat (msec): min=27, max=101, avg=55.02, stdev=11.78 clat percentiles (usec): | 1.00th=[29754], 5.00th=[34866], 10.00th=[38536], 20.00th=[42206], | 30.00th=[45351], 40.00th=[47973], 50.00th=[49546], 60.00th=[52691], | 70.00th=[56361], 80.00th=[60031], 90.00th=[65799], 95.00th=[72877], | 99.00th=[86508], 99.50th=[89654], 99.90th=[94897], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min= 416, max= 680, per=24.95%, avg=579.44, stdev=73.75, samples=16 iops : min= 104, max= 170, avg=144.81, stdev=18.42, samples=16 write: IOPS=150, BW=602KiB/s (616kB/s)(5192KiB/8624msec); 0 zone resets slat (usec): min=52, max=1560, avg=211.28, stdev=166.81 clat (msec): min=17, max=107, avg=52.03, stdev=10.88 lat (msec): min=17, max=107, avg=52.24, stdev=10.89 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 38], 10.00th=[ 41], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 51], 60.00th=[ 54], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 66], 95.00th=[ 71], | 99.00th=[ 90], 99.50th=[ 93], 99.90th=[ 104], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min= 448, max= 752, per=24.82%, avg=600.06, stdev=80.03, samples=16 iops : min= 112, max= 188, avg=149.94, stdev=19.98, samples=16 lat (msec) : 20=0.04%, 50=49.49%, 100=50.35%, 250=0.12% cpu : usr=0.41%, sys=5.06%, ctx=4522, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179855: Tue Aug 11 17:45:08 2026 read: IOPS=143, BW=576KiB/s (590kB/s)(4976KiB/8639msec) slat (usec): min=1309, max=13636, avg=3524.17, stdev=1280.73 clat (msec): min=12, max=101, avg=51.27, stdev=11.72 lat (msec): min=14, max=105, avg=54.79, stdev=12.10 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 67], 95.00th=[ 74], | 99.00th=[ 91], 99.50th=[ 94], 99.90th=[ 97], 99.95th=[ 102], | 99.99th=[ 102] bw ( KiB/s): min= 440, max= 712, per=24.68%, avg=573.12, stdev=86.01, samples=17 iops : min= 110, max= 178, avg=143.24, stdev=21.52, samples=17 write: IOPS=152, BW=609KiB/s (624kB/s)(5264KiB/8639msec); 0 zone resets slat (usec): min=54, max=1511, avg=220.28, stdev=176.27 clat (usec): min=16464, max=94431, avg=52411.93, stdev=10526.61 lat (usec): min=16653, max=95002, avg=52633.45, stdev=10536.95 clat percentiles (usec): | 1.00th=[33817], 5.00th=[38011], 10.00th=[40633], 20.00th=[43779], | 30.00th=[46400], 40.00th=[49021], 50.00th=[51119], 60.00th=[53216], | 70.00th=[56361], 80.00th=[60031], 90.00th=[66323], 95.00th=[70779], | 99.00th=[86508], 99.50th=[92799], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] bw ( KiB/s): min= 496, max= 750, per=25.12%, avg=607.41, stdev=66.87, samples=17 iops : min= 124, max= 187, avg=151.82, stdev=16.65, samples=17 lat (msec) : 20=0.08%, 50=48.36%, 100=51.52%, 250=0.04% cpu : usr=0.58%, sys=4.87%, ctx=4496, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179856: Tue Aug 11 17:45:08 2026 read: IOPS=145, BW=583KiB/s (597kB/s)(4984KiB/8544msec) slat (usec): min=39, max=14194, avg=3508.93, stdev=1278.56 clat (msec): min=5, max=121, avg=50.95, stdev=12.26 lat (msec): min=6, max=124, avg=54.47, stdev=12.72 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 35], 10.00th=[ 38], 20.00th=[ 42], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 67], 95.00th=[ 73], | 99.00th=[ 88], 99.50th=[ 95], 99.90th=[ 108], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 408, max= 696, per=25.09%, avg=582.62, stdev=79.50, samples=16 iops : min= 102, max= 174, avg=145.63, stdev=19.85, samples=16 write: IOPS=153, BW=615KiB/s (630kB/s)(5256KiB/8544msec); 0 zone resets slat (usec): min=42, max=3551, avg=213.16, stdev=195.44 clat (msec): min=23, max=121, avg=51.98, stdev=11.14 lat (msec): min=23, max=121, avg=52.20, stdev=11.15 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 38], 10.00th=[ 41], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 49], 50.00th=[ 51], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 67], 95.00th=[ 71], | 99.00th=[ 88], 99.50th=[ 92], 99.90th=[ 116], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 400, max= 760, per=25.16%, avg=608.25, stdev=96.92, samples=16 iops : min= 100, max= 190, avg=152.00, stdev=24.19, samples=16 lat (msec) : 10=0.12%, 20=0.23%, 50=48.67%, 100=50.62%, 250=0.35% cpu : usr=0.42%, sys=5.17%, ctx=4482, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179857: Tue Aug 11 17:45:08 2026 read: IOPS=146, BW=586KiB/s (600kB/s)(5060KiB/8639msec) slat (usec): min=1432, max=9322, avg=3537.41, stdev=1243.04 clat (msec): min=20, max=105, avg=51.01, stdev=11.56 lat (msec): min=25, max=112, avg=54.55, stdev=11.95 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 42], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 59], 90.00th=[ 66], 95.00th=[ 73], | 99.00th=[ 88], 99.50th=[ 91], 99.90th=[ 103], 99.95th=[ 106], | 99.99th=[ 106] bw ( KiB/s): min= 456, max= 704, per=25.11%, avg=583.12, stdev=70.08, samples=17 iops : min= 114, max= 176, avg=145.71, stdev=17.49, samples=17 write: IOPS=149, BW=600KiB/s (614kB/s)(5180KiB/8639msec); 0 zone resets slat (usec): min=40, max=1756, avg=212.09, stdev=168.82 clat (msec): min=16, max=100, avg=52.65, stdev=10.24 lat (msec): min=16, max=100, avg=52.86, stdev=10.26 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 45], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 52], 60.00th=[ 54], | 70.00th=[ 57], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 72], | 99.00th=[ 82], 99.50th=[ 92], 99.90th=[ 99], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min= 408, max= 712, per=24.71%, avg=597.41, stdev=82.32, samples=17 iops : min= 102, max= 178, avg=149.29, stdev=20.58, samples=17 lat (msec) : 20=0.04%, 50=47.77%, 100=52.07%, 250=0.12% cpu : usr=0.53%, sys=4.89%, ctx=4532, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2323KiB/s (2379kB/s), 576KiB/s-586KiB/s (590kB/s-600kB/s), io=19.6MiB (20.5MB), run=8544-8639msec WRITE: bw=2418KiB/s (2476kB/s), 600KiB/s-615KiB/s (614kB/s-630kB/s), io=20.4MiB (21.4MB), run=8544-8639msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179861: Tue Aug 11 17:45:12 2026 read: IOPS=8, BW=328MiB/s (344MB/s)(40.0MiB/122msec) slat (nsec): min=13029k, max=13029k, avg=13028692.00, stdev= 0.00 clat (nsec): min=102197k, max=102197k, avg=102197357.00, stdev= 0.00 lat (nsec): min=115230k, max=115230k, avg=115230329.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 103], 10.00th=[ 103], 20.00th=[ 103], | 30.00th=[ 103], 40.00th=[ 103], 50.00th=[ 103], 60.00th=[ 103], | 70.00th=[ 103], 80.00th=[ 103], 90.00th=[ 103], 95.00th=[ 103], | 99.00th=[ 103], 99.50th=[ 103], 99.90th=[ 103], 99.95th=[ 103], | 99.99th=[ 103] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=9.92%, ctx=11, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=40.0MiB (41.9MB), run=122-122msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout