== sanity test 398c: run fio to test AIO ================= 10:22:13 (1782483733) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.180849 s, 232 MB/s osc.lustre-OST0000-osc-ffff9c30ca9c7000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147166: Fri Jun 26 10:22:56 2026 write: IOPS=65, BW=262KiB/s (269kB/s)(10.0MiB/39018msec); 0 zone resets slat (usec): min=47, max=3925, avg=234.04, stdev=164.06 clat (msec): min=56, max=477, avg=243.27, stdev=62.04 lat (msec): min=56, max=477, avg=243.51, stdev=62.05 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 148], 10.00th=[ 171], 20.00th=[ 194], | 30.00th=[ 211], 40.00th=[ 228], 50.00th=[ 243], 60.00th=[ 257], | 70.00th=[ 279], 80.00th=[ 296], 90.00th=[ 321], 95.00th=[ 342], | 99.00th=[ 393], 99.50th=[ 430], 99.90th=[ 464], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 168, max= 535, per=24.96%, avg=261.81, stdev=54.70, samples=77 iops : min= 42, max= 133, avg=65.32, stdev=13.62, samples=77 lat (msec) : 100=2.03%, 250=52.77%, 500=45.20% cpu : usr=0.14%, sys=1.24%, ctx=2758, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147167: Fri Jun 26 10:22:56 2026 write: IOPS=65, BW=263KiB/s (269kB/s)(10.0MiB/38994msec); 0 zone resets slat (usec): min=36, max=5321, avg=226.77, stdev=210.05 clat (msec): min=53, max=482, avg=243.33, stdev=63.61 lat (msec): min=53, max=482, avg=243.55, stdev=63.63 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 148], 10.00th=[ 171], 20.00th=[ 192], | 30.00th=[ 209], 40.00th=[ 226], 50.00th=[ 243], 60.00th=[ 257], | 70.00th=[ 279], 80.00th=[ 296], 90.00th=[ 326], 95.00th=[ 347], | 99.00th=[ 384], 99.50th=[ 397], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 485] bw ( KiB/s): min= 136, max= 619, per=24.99%, avg=262.18, stdev=64.16, samples=77 iops : min= 34, max= 154, avg=65.42, stdev=15.96, samples=77 lat (msec) : 100=3.12%, 250=52.03%, 500=44.84% cpu : usr=0.13%, sys=1.21%, ctx=2711, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147168: Fri Jun 26 10:22:56 2026 write: IOPS=65, BW=262KiB/s (269kB/s)(10.0MiB/39018msec); 0 zone resets slat (usec): min=35, max=4360, avg=235.20, stdev=189.64 clat (msec): min=77, max=448, avg=243.34, stdev=61.96 lat (msec): min=77, max=449, avg=243.58, stdev=61.96 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 148], 10.00th=[ 171], 20.00th=[ 192], | 30.00th=[ 209], 40.00th=[ 228], 50.00th=[ 243], 60.00th=[ 259], | 70.00th=[ 279], 80.00th=[ 296], 90.00th=[ 326], 95.00th=[ 342], | 99.00th=[ 384], 99.50th=[ 397], 99.90th=[ 430], 99.95th=[ 447], | 99.99th=[ 451] bw ( KiB/s): min= 176, max= 535, per=24.95%, avg=261.71, stdev=56.43, samples=77 iops : min= 44, max= 133, avg=65.30, stdev=14.07, samples=77 lat (msec) : 100=1.99%, 250=53.05%, 500=44.96% cpu : usr=0.15%, sys=1.23%, ctx=2743, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147169: Fri Jun 26 10:22:56 2026 write: IOPS=65, BW=263KiB/s (269kB/s)(10.0MiB/38926msec); 0 zone resets slat (usec): min=46, max=5847, avg=243.10, stdev=211.47 clat (msec): min=43, max=445, avg=242.78, stdev=62.42 lat (msec): min=43, max=446, avg=243.02, stdev=62.43 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 144], 10.00th=[ 169], 20.00th=[ 192], | 30.00th=[ 211], 40.00th=[ 228], 50.00th=[ 243], 60.00th=[ 259], | 70.00th=[ 275], 80.00th=[ 296], 90.00th=[ 321], 95.00th=[ 342], | 99.00th=[ 384], 99.50th=[ 397], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 447] bw ( KiB/s): min= 152, max= 600, per=25.01%, avg=262.35, stdev=60.67, samples=77 iops : min= 38, max= 150, avg=65.47, stdev=15.18, samples=77 lat (msec) : 50=0.08%, 100=2.23%, 250=52.30%, 500=45.39% cpu : usr=0.15%, sys=1.27%, ctx=2817, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1050KiB/s (1075kB/s), 262KiB/s-263KiB/s (269kB/s-269kB/s), io=40.0MiB (41.9MB), run=38926-39018msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147184: Fri Jun 26 10:23:33 2026 read: IOPS=37, BW=149KiB/s (152kB/s)(5048KiB/33920msec) slat (usec): min=1326, max=17102, avg=4509.36, stdev=2123.70 clat (msec): min=7, max=273, avg=131.79, stdev=37.65 lat (msec): min=10, max=280, avg=136.30, stdev=38.31 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 83], 10.00th=[ 92], 20.00th=[ 104], | 30.00th=[ 111], 40.00th=[ 121], 50.00th=[ 128], 60.00th=[ 136], | 70.00th=[ 146], 80.00th=[ 159], 90.00th=[ 182], 95.00th=[ 201], | 99.00th=[ 247], 99.50th=[ 259], 99.90th=[ 264], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 47, max= 248, per=25.15%, avg=148.63, stdev=46.88, samples=67 iops : min= 11, max= 62, avg=36.99, stdev=11.79, samples=67 write: IOPS=38, BW=153KiB/s (157kB/s)(5192KiB/33920msec); 0 zone resets slat (usec): min=61, max=3279, avg=232.91, stdev=184.34 clat (msec): min=87, max=611, avg=284.67, stdev=88.28 lat (msec): min=88, max=611, avg=284.90, stdev=88.29 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 155], 10.00th=[ 180], 20.00th=[ 207], | 30.00th=[ 232], 40.00th=[ 253], 50.00th=[ 279], 60.00th=[ 305], | 70.00th=[ 330], 80.00th=[ 351], 90.00th=[ 405], 95.00th=[ 443], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 609], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 79, max= 240, per=24.76%, avg=152.30, stdev=38.70, samples=67 iops : min= 19, max= 60, avg=37.90, stdev= 9.70, samples=67 lat (msec) : 10=0.04%, 20=0.20%, 50=0.51%, 100=7.42%, 250=60.51% lat (msec) : 500=30.62%, 750=0.70% cpu : usr=0.18%, sys=1.59%, ctx=5086, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147185: Fri Jun 26 10:23:33 2026 read: IOPS=36, BW=147KiB/s (150kB/s)(4976KiB/33950msec) slat (usec): min=33, max=20589, avg=4435.44, stdev=2120.06 clat (msec): min=2, max=289, avg=132.10, stdev=38.05 lat (msec): min=6, max=296, avg=136.54, stdev=38.67 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 83], 10.00th=[ 91], 20.00th=[ 104], | 30.00th=[ 112], 40.00th=[ 122], 50.00th=[ 130], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 159], 90.00th=[ 180], 95.00th=[ 205], | 99.00th=[ 241], 99.50th=[ 251], 99.90th=[ 279], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 63, max= 288, per=24.73%, avg=146.13, stdev=42.14, samples=67 iops : min= 15, max= 72, avg=36.34, stdev=10.53, samples=67 write: IOPS=38, BW=155KiB/s (159kB/s)(5264KiB/33950msec); 0 zone resets slat (usec): min=39, max=3274, avg=236.25, stdev=219.64 clat (msec): min=45, max=605, avg=283.16, stdev=88.87 lat (msec): min=45, max=605, avg=283.40, stdev=88.88 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 153], 10.00th=[ 178], 20.00th=[ 209], | 30.00th=[ 228], 40.00th=[ 253], 50.00th=[ 275], 60.00th=[ 300], | 70.00th=[ 330], 80.00th=[ 355], 90.00th=[ 401], 95.00th=[ 439], | 99.00th=[ 506], 99.50th=[ 535], 99.90th=[ 600], 99.95th=[ 609], | 99.99th=[ 609] bw ( KiB/s): min= 87, max= 248, per=25.10%, avg=154.37, stdev=38.52, samples=67 iops : min= 21, max= 62, avg=38.40, stdev= 9.71, samples=67 lat (msec) : 4=0.04%, 10=0.20%, 20=0.20%, 50=0.39%, 100=8.05% lat (msec) : 250=59.65%, 500=30.86%, 750=0.62% cpu : usr=0.17%, sys=1.67%, ctx=4991, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147186: Fri Jun 26 10:23:33 2026 read: IOPS=36, BW=147KiB/s (150kB/s)(4984KiB/33949msec) slat (usec): min=1277, max=27378, avg=4465.66, stdev=2216.97 clat (msec): min=7, max=273, avg=132.04, stdev=35.65 lat (msec): min=9, max=280, avg=136.51, stdev=36.29 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 86], 10.00th=[ 94], 20.00th=[ 105], | 30.00th=[ 112], 40.00th=[ 120], 50.00th=[ 128], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 159], 90.00th=[ 180], 95.00th=[ 199], | 99.00th=[ 239], 99.50th=[ 249], 99.90th=[ 259], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 55, max= 264, per=24.74%, avg=146.22, stdev=45.93, samples=67 iops : min= 13, max= 66, avg=36.36, stdev=11.55, samples=67 write: IOPS=38, BW=155KiB/s (159kB/s)(5256KiB/33949msec); 0 zone resets slat (usec): min=55, max=3442, avg=234.64, stdev=211.05 clat (msec): min=35, max=602, avg=283.07, stdev=90.05 lat (msec): min=36, max=602, avg=283.31, stdev=90.06 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 148], 10.00th=[ 171], 20.00th=[ 207], | 30.00th=[ 228], 40.00th=[ 253], 50.00th=[ 275], 60.00th=[ 305], | 70.00th=[ 330], 80.00th=[ 355], 90.00th=[ 409], 95.00th=[ 443], | 99.00th=[ 510], 99.50th=[ 550], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 72, max= 256, per=24.82%, avg=152.63, stdev=40.93, samples=67 iops : min= 18, max= 64, avg=37.97, stdev=10.28, samples=67 lat (msec) : 10=0.04%, 20=0.16%, 50=0.31%, 100=7.38%, 250=60.23% lat (msec) : 500=31.13%, 750=0.74% cpu : usr=0.16%, sys=1.66%, ctx=5028, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147187: Fri Jun 26 10:23:33 2026 read: IOPS=37, BW=150KiB/s (153kB/s)(5060KiB/33806msec) slat (usec): min=1292, max=16399, avg=4473.24, stdev=2127.36 clat (msec): min=4, max=282, avg=133.12, stdev=36.59 lat (msec): min=9, max=289, avg=137.60, stdev=37.18 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 84], 10.00th=[ 92], 20.00th=[ 105], | 30.00th=[ 112], 40.00th=[ 122], 50.00th=[ 131], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 159], 90.00th=[ 180], 95.00th=[ 201], | 99.00th=[ 245], 99.50th=[ 253], 99.90th=[ 268], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 56, max= 248, per=25.32%, avg=149.63, stdev=45.29, samples=67 iops : min= 14, max= 62, avg=37.24, stdev=11.36, samples=67 write: IOPS=38, BW=153KiB/s (157kB/s)(5180KiB/33806msec); 0 zone resets slat (usec): min=59, max=5621, avg=238.44, stdev=251.70 clat (msec): min=37, max=590, avg=282.28, stdev=87.04 lat (msec): min=37, max=590, avg=282.52, stdev=87.06 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 157], 10.00th=[ 180], 20.00th=[ 207], | 30.00th=[ 226], 40.00th=[ 251], 50.00th=[ 275], 60.00th=[ 300], | 70.00th=[ 326], 80.00th=[ 355], 90.00th=[ 401], 95.00th=[ 435], | 99.00th=[ 518], 99.50th=[ 542], 99.90th=[ 584], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 88, max= 240, per=24.78%, avg=152.37, stdev=37.83, samples=67 iops : min= 22, max= 60, avg=37.91, stdev= 9.51, samples=67 lat (msec) : 10=0.08%, 20=0.12%, 50=0.27%, 100=7.70%, 250=61.02% lat (msec) : 500=30.00%, 750=0.82% cpu : usr=0.17%, sys=1.68%, ctx=5108, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=591KiB/s (605kB/s), 147KiB/s-150KiB/s (150kB/s-153kB/s), io=19.6MiB (20.5MB), run=33806-33950msec WRITE: bw=615KiB/s (630kB/s), 153KiB/s-155KiB/s (157kB/s-159kB/s), io=20.4MiB (21.4MB), run=33806-33950msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147196: Fri Jun 26 10:23:38 2026 read: IOPS=5, BW=233MiB/s (244MB/s)(40.0MiB/172msec) slat (nsec): min=15343k, max=15343k, avg=15342527.00, stdev= 0.00 clat (nsec): min=148993k, max=148993k, avg=148992871.00, stdev= 0.00 lat (nsec): min=164341k, max=164341k, avg=164341070.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 150], | 30.00th=[ 150], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 150], | 70.00th=[ 150], 80.00th=[ 150], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.02%, ctx=43, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=233MiB/s (244MB/s), 233MiB/s-233MiB/s (244MB/s-244MB/s), io=40.0MiB (41.9MB), run=172-172msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout