== sanity test 398c: run fio to test AIO ================= 10:51:07 (1782485467) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.555561 s, 75.5 MB/s osc.lustre-OST0000-osc-ffff9ce5997a5000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=183552: Fri Jun 26 10:51:38 2026 write: IOPS=111, BW=445KiB/s (455kB/s)(10.0MiB/23025msec); 0 zone resets slat (usec): min=37, max=13119, avg=250.96, stdev=615.74 clat (msec): min=16, max=305, avg=143.45, stdev=35.38 lat (msec): min=17, max=305, avg=143.70, stdev=35.36 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 99], 10.00th=[ 107], 20.00th=[ 117], | 30.00th=[ 125], 40.00th=[ 132], 50.00th=[ 138], 60.00th=[ 146], | 70.00th=[ 157], 80.00th=[ 169], 90.00th=[ 188], 95.00th=[ 209], | 99.00th=[ 255], 99.50th=[ 279], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 302, max= 638, per=25.20%, avg=445.07, stdev=74.82, samples=45 iops : min= 75, max= 159, avg=110.96, stdev=18.68, samples=45 lat (msec) : 20=0.12%, 50=0.31%, 100=5.47%, 250=93.01%, 500=1.09% cpu : usr=0.17%, sys=1.58%, ctx=2974, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183553: Fri Jun 26 10:51:38 2026 write: IOPS=110, BW=442KiB/s (452kB/s)(10.0MiB/23191msec); 0 zone resets slat (usec): min=36, max=12020, avg=244.19, stdev=614.90 clat (msec): min=11, max=290, avg=143.81, stdev=33.68 lat (msec): min=13, max=290, avg=144.05, stdev=33.70 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 117], | 30.00th=[ 124], 40.00th=[ 132], 50.00th=[ 140], 60.00th=[ 148], | 70.00th=[ 157], 80.00th=[ 169], 90.00th=[ 186], 95.00th=[ 201], | 99.00th=[ 259], 99.50th=[ 292], 99.90th=[ 292], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 254, max= 638, per=24.87%, avg=439.18, stdev=77.50, samples=45 iops : min= 63, max= 159, avg=109.47, stdev=19.34, samples=45 lat (msec) : 20=0.12%, 50=0.23%, 100=4.77%, 250=93.79%, 500=1.09% cpu : usr=0.21%, sys=1.48%, ctx=2639, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183554: Fri Jun 26 10:51:38 2026 write: IOPS=110, BW=442KiB/s (453kB/s)(10.0MiB/23142msec); 0 zone resets slat (usec): min=36, max=10225, avg=253.38, stdev=629.90 clat (msec): min=17, max=303, avg=143.61, stdev=34.18 lat (msec): min=18, max=303, avg=143.87, stdev=34.18 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 101], 10.00th=[ 108], 20.00th=[ 117], | 30.00th=[ 125], 40.00th=[ 132], 50.00th=[ 140], 60.00th=[ 146], | 70.00th=[ 155], 80.00th=[ 169], 90.00th=[ 188], 95.00th=[ 205], | 99.00th=[ 257], 99.50th=[ 284], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 254, max= 638, per=24.97%, avg=440.96, stdev=86.30, samples=45 iops : min= 63, max= 159, avg=109.93, stdev=21.53, samples=45 lat (msec) : 20=0.04%, 50=0.59%, 100=4.10%, 250=93.95%, 500=1.33% cpu : usr=0.20%, sys=1.46%, ctx=2786, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183555: Fri Jun 26 10:51:38 2026 write: IOPS=110, BW=442KiB/s (453kB/s)(10.0MiB/23157msec); 0 zone resets slat (usec): min=36, max=12035, avg=236.95, stdev=603.78 clat (msec): min=22, max=289, avg=143.67, stdev=33.62 lat (msec): min=22, max=289, avg=143.91, stdev=33.66 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 97], 10.00th=[ 110], 20.00th=[ 118], | 30.00th=[ 125], 40.00th=[ 132], 50.00th=[ 138], 60.00th=[ 144], | 70.00th=[ 157], 80.00th=[ 169], 90.00th=[ 186], 95.00th=[ 203], | 99.00th=[ 251], 99.50th=[ 271], 99.90th=[ 288], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 256, max= 638, per=24.97%, avg=440.98, stdev=82.25, samples=45 iops : min= 64, max= 159, avg=109.93, stdev=20.54, samples=45 lat (msec) : 50=0.12%, 100=5.90%, 250=92.89%, 500=1.09% cpu : usr=0.22%, sys=1.34%, ctx=2586, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1766KiB/s (1809kB/s), 442KiB/s-445KiB/s (452kB/s-455kB/s), io=40.0MiB (41.9MB), run=23025-23191msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=183569: Fri Jun 26 10:52:08 2026 read: IOPS=55, BW=220KiB/s (225kB/s)(5048KiB/22931msec) slat (usec): min=57, max=37015, avg=9970.02, stdev=5432.39 clat (msec): min=46, max=275, avg=136.16, stdev=37.11 lat (msec): min=58, max=292, avg=146.14, stdev=38.83 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 86], 10.00th=[ 92], 20.00th=[ 104], | 30.00th=[ 112], 40.00th=[ 123], 50.00th=[ 133], 60.00th=[ 144], | 70.00th=[ 155], 80.00th=[ 167], 90.00th=[ 186], 95.00th=[ 201], | 99.00th=[ 247], 99.50th=[ 257], 99.90th=[ 271], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 119, max= 320, per=25.32%, avg=220.02, stdev=47.53, samples=45 iops : min= 29, max= 80, avg=54.64, stdev=12.04, samples=45 write: IOPS=56, BW=226KiB/s (232kB/s)(5192KiB/22931msec); 0 zone resets slat (usec): min=39, max=10913, avg=272.05, stdev=627.09 clat (msec): min=14, max=267, avg=139.86, stdev=35.82 lat (msec): min=14, max=267, avg=140.14, stdev=35.84 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 89], 10.00th=[ 95], 20.00th=[ 106], | 30.00th=[ 116], 40.00th=[ 128], 50.00th=[ 138], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 171], 90.00th=[ 186], 95.00th=[ 203], | 99.00th=[ 230], 99.50th=[ 236], 99.90th=[ 257], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 136, max= 352, per=25.07%, avg=226.89, stdev=57.98, samples=45 iops : min= 34, max= 88, avg=56.36, stdev=14.57, samples=45 lat (msec) : 20=0.04%, 50=0.12%, 100=15.86%, 250=83.55%, 500=0.43% cpu : usr=0.29%, sys=2.28%, ctx=4584, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183570: Fri Jun 26 10:52:08 2026 read: IOPS=54, BW=217KiB/s (222kB/s)(4976KiB/22979msec) slat (usec): min=2322, max=40263, avg=10232.47, stdev=5413.87 clat (msec): min=44, max=262, avg=136.81, stdev=34.63 lat (msec): min=50, max=269, avg=147.05, stdev=36.29 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 87], 10.00th=[ 93], 20.00th=[ 105], | 30.00th=[ 114], 40.00th=[ 125], 50.00th=[ 138], 60.00th=[ 146], | 70.00th=[ 155], 80.00th=[ 169], 90.00th=[ 184], 95.00th=[ 197], | 99.00th=[ 222], 99.50th=[ 226], 99.90th=[ 241], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 120, max= 311, per=24.85%, avg=215.93, stdev=50.04, samples=45 iops : min= 30, max= 77, avg=53.60, stdev=12.58, samples=45 write: IOPS=57, BW=229KiB/s (235kB/s)(5264KiB/22979msec); 0 zone resets slat (usec): min=39, max=22893, avg=259.11, stdev=800.94 clat (msec): min=55, max=268, avg=138.91, stdev=33.60 lat (msec): min=55, max=268, avg=139.17, stdev=33.62 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 90], 10.00th=[ 96], 20.00th=[ 107], | 30.00th=[ 118], 40.00th=[ 127], 50.00th=[ 138], 60.00th=[ 148], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 184], 95.00th=[ 197], | 99.00th=[ 220], 99.50th=[ 224], 99.90th=[ 253], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 127, max= 373, per=25.37%, avg=229.56, stdev=57.60, samples=45 iops : min= 31, max= 93, avg=57.00, stdev=14.56, samples=45 lat (msec) : 50=0.04%, 100=14.30%, 250=85.55%, 500=0.12% cpu : usr=0.32%, sys=2.42%, ctx=4618, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183571: Fri Jun 26 10:52:08 2026 read: IOPS=54, BW=216KiB/s (221kB/s)(4984KiB/23072msec) slat (usec): min=2340, max=38157, avg=10160.74, stdev=5448.95 clat (msec): min=11, max=243, avg=135.13, stdev=34.29 lat (msec): min=17, max=254, avg=145.29, stdev=35.80 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 86], 10.00th=[ 93], 20.00th=[ 104], | 30.00th=[ 113], 40.00th=[ 122], 50.00th=[ 134], 60.00th=[ 144], | 70.00th=[ 153], 80.00th=[ 165], 90.00th=[ 184], 95.00th=[ 197], | 99.00th=[ 215], 99.50th=[ 224], 99.90th=[ 236], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 119, max= 320, per=24.80%, avg=215.49, stdev=53.17, samples=45 iops : min= 29, max= 80, avg=53.51, stdev=13.39, samples=45 write: IOPS=56, BW=228KiB/s (233kB/s)(5256KiB/23072msec); 0 zone resets slat (usec): min=53, max=12926, avg=269.88, stdev=708.12 clat (msec): min=48, max=238, avg=141.43, stdev=34.01 lat (msec): min=48, max=238, avg=141.71, stdev=34.03 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 92], 10.00th=[ 100], 20.00th=[ 109], | 30.00th=[ 120], 40.00th=[ 130], 50.00th=[ 140], 60.00th=[ 150], | 70.00th=[ 161], 80.00th=[ 171], 90.00th=[ 188], 95.00th=[ 203], | 99.00th=[ 222], 99.50th=[ 228], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 135, max= 343, per=25.01%, avg=226.38, stdev=51.30, samples=45 iops : min= 33, max= 85, avg=56.20, stdev=12.93, samples=45 lat (msec) : 20=0.08%, 50=0.12%, 100=13.16%, 250=86.64% cpu : usr=0.30%, sys=2.47%, ctx=4736, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183572: Fri Jun 26 10:52:08 2026 read: IOPS=54, BW=219KiB/s (224kB/s)(5060KiB/23085msec) slat (usec): min=2231, max=46258, avg=10349.00, stdev=5709.75 clat (msec): min=36, max=247, avg=135.90, stdev=35.60 lat (msec): min=47, max=260, avg=146.25, stdev=37.35 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 86], 10.00th=[ 94], 20.00th=[ 104], | 30.00th=[ 112], 40.00th=[ 122], 50.00th=[ 132], 60.00th=[ 144], | 70.00th=[ 155], 80.00th=[ 167], 90.00th=[ 186], 95.00th=[ 201], | 99.00th=[ 226], 99.50th=[ 230], 99.90th=[ 241], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 127, max= 373, per=25.15%, avg=218.56, stdev=54.85, samples=45 iops : min= 31, max= 93, avg=54.27, stdev=13.88, samples=45 write: IOPS=56, BW=224KiB/s (230kB/s)(5180KiB/23085msec); 0 zone resets slat (usec): min=38, max=9648, avg=278.96, stdev=601.02 clat (msec): min=59, max=246, avg=140.93, stdev=34.70 lat (msec): min=59, max=246, avg=141.22, stdev=34.71 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 91], 10.00th=[ 97], 20.00th=[ 110], | 30.00th=[ 121], 40.00th=[ 129], 50.00th=[ 138], 60.00th=[ 148], | 70.00th=[ 159], 80.00th=[ 174], 90.00th=[ 190], 95.00th=[ 203], | 99.00th=[ 226], 99.50th=[ 230], 99.90th=[ 247], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 143, max= 333, per=24.67%, avg=223.29, stdev=50.28, samples=45 iops : min= 35, max= 83, avg=55.47, stdev=12.63, samples=45 lat (msec) : 50=0.08%, 100=13.87%, 250=86.05% cpu : usr=0.29%, sys=2.43%, ctx=4651, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=869KiB/s (890kB/s), 216KiB/s-220KiB/s (221kB/s-225kB/s), io=19.6MiB (20.5MB), run=22931-23085msec WRITE: bw=905KiB/s (927kB/s), 224KiB/s-229KiB/s (230kB/s-235kB/s), io=20.4MiB (21.4MB), run=22931-23085msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=183580: Fri Jun 26 10:52:17 2026 read: IOPS=2, BW=85.5MiB/s (89.6MB/s)(40.0MiB/468msec) slat (nsec): min=15400k, max=15400k, avg=15400330.00, stdev= 0.00 clat (nsec): min=436134k, max=436134k, avg=436133858.00, stdev= 0.00 lat (nsec): min=451539k, max=451539k, avg=451538895.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 435], 10.00th=[ 435], 20.00th=[ 435], | 30.00th=[ 435], 40.00th=[ 435], 50.00th=[ 435], 60.00th=[ 435], | 70.00th=[ 435], 80.00th=[ 435], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] lat (msec) : 500=100.00% cpu : usr=0.21%, sys=2.14%, ctx=10, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=85.5MiB/s (89.6MB/s), 85.5MiB/s-85.5MiB/s (89.6MB/s-89.6MB/s), io=40.0MiB (41.9MB), run=468-468msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout