== sanity test 398c: run fio to test AIO ================= 22:51:36 (1786762296) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.447792 s, 93.7 MB/s osc.lustre-OST0000-osc-ffff96fcebe33000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180252: Fri Aug 14 22:52:07 2026 write: IOPS=113, BW=453KiB/s (464kB/s)(10.0MiB/22618msec); 0 zone resets slat (usec): min=50, max=8417, avg=295.16, stdev=663.15 clat (msec): min=23, max=218, avg=140.33, stdev=27.68 lat (msec): min=23, max=218, avg=140.62, stdev=27.65 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 100], 10.00th=[ 106], 20.00th=[ 117], | 30.00th=[ 125], 40.00th=[ 130], 50.00th=[ 140], 60.00th=[ 146], | 70.00th=[ 155], 80.00th=[ 165], 90.00th=[ 182], 95.00th=[ 188], | 99.00th=[ 205], 99.50th=[ 213], 99.90th=[ 220], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 328, max= 579, per=24.86%, avg=450.02, stdev=58.85, samples=44 iops : min= 82, max= 144, avg=112.18, stdev=14.72, samples=44 lat (msec) : 50=0.12%, 100=5.78%, 250=94.10% cpu : usr=0.16%, sys=1.25%, ctx=2807, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180253: Fri Aug 14 22:52:07 2026 write: IOPS=113, BW=453KiB/s (464kB/s)(10.0MiB/22599msec); 0 zone resets slat (usec): min=51, max=9685, avg=279.63, stdev=612.54 clat (msec): min=34, max=225, avg=140.31, stdev=27.45 lat (msec): min=34, max=226, avg=140.59, stdev=27.45 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 100], 10.00th=[ 107], 20.00th=[ 120], | 30.00th=[ 125], 40.00th=[ 131], 50.00th=[ 138], 60.00th=[ 144], | 70.00th=[ 155], 80.00th=[ 165], 90.00th=[ 178], 95.00th=[ 190], | 99.00th=[ 207], 99.50th=[ 213], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 360, max= 611, per=24.92%, avg=451.11, stdev=57.78, samples=44 iops : min= 90, max= 152, avg=112.43, stdev=14.42, samples=44 lat (msec) : 50=0.12%, 100=5.59%, 250=94.30% cpu : usr=0.19%, sys=1.25%, ctx=2657, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180254: Fri Aug 14 22:52:07 2026 write: IOPS=113, BW=456KiB/s (467kB/s)(10.0MiB/22459msec); 0 zone resets slat (usec): min=46, max=20020, avg=299.50, stdev=780.34 clat (msec): min=16, max=219, avg=139.94, stdev=27.84 lat (msec): min=17, max=219, avg=140.25, stdev=27.85 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 99], 10.00th=[ 107], 20.00th=[ 117], | 30.00th=[ 125], 40.00th=[ 132], 50.00th=[ 138], 60.00th=[ 146], | 70.00th=[ 155], 80.00th=[ 165], 90.00th=[ 180], 95.00th=[ 188], | 99.00th=[ 203], 99.50th=[ 211], 99.90th=[ 215], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 303, max= 641, per=25.07%, avg=453.75, stdev=66.68, samples=44 iops : min= 75, max= 160, avg=113.11, stdev=16.72, samples=44 lat (msec) : 20=0.12%, 50=0.16%, 100=5.74%, 250=93.98% cpu : usr=0.14%, sys=1.59%, ctx=2827, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180255: Fri Aug 14 22:52:07 2026 write: IOPS=113, BW=453KiB/s (464kB/s)(10.0MiB/22613msec); 0 zone resets slat (usec): min=50, max=9924, avg=268.74, stdev=597.83 clat (msec): min=21, max=227, avg=140.44, stdev=28.21 lat (msec): min=22, max=228, avg=140.71, stdev=28.21 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 99], 10.00th=[ 107], 20.00th=[ 120], | 30.00th=[ 126], 40.00th=[ 132], 50.00th=[ 140], 60.00th=[ 146], | 70.00th=[ 155], 80.00th=[ 161], 90.00th=[ 176], 95.00th=[ 194], | 99.00th=[ 209], 99.50th=[ 213], 99.90th=[ 228], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 382, max= 592, per=24.97%, avg=451.91, stdev=61.13, samples=44 iops : min= 95, max= 148, avg=112.64, stdev=15.29, samples=44 lat (msec) : 50=0.23%, 100=5.62%, 250=94.14% cpu : usr=0.15%, sys=1.27%, ctx=2742, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1811KiB/s (1854kB/s), 453KiB/s-456KiB/s (464kB/s-467kB/s), io=40.0MiB (41.9MB), run=22459-22618msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180268: Fri Aug 14 22:52:38 2026 read: IOPS=52, BW=210KiB/s (215kB/s)(5048KiB/24021msec) slat (usec): min=88, max=37058, avg=10233.91, stdev=5026.48 clat (msec): min=40, max=322, avg=142.90, stdev=32.07 lat (msec): min=44, max=344, avg=153.14, stdev=32.97 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 100], 10.00th=[ 106], 20.00th=[ 116], | 30.00th=[ 125], 40.00th=[ 132], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 157], 80.00th=[ 167], 90.00th=[ 182], 95.00th=[ 197], | 99.00th=[ 236], 99.50th=[ 279], 99.90th=[ 305], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 120, max= 310, per=25.14%, avg=209.94, stdev=40.39, samples=47 iops : min= 30, max= 77, avg=52.09, stdev=10.07, samples=47 write: IOPS=54, BW=216KiB/s (221kB/s)(5192KiB/24021msec); 0 zone resets slat (usec): min=59, max=13832, avg=332.75, stdev=846.44 clat (msec): min=39, max=285, avg=145.11, stdev=29.70 lat (msec): min=39, max=285, avg=145.45, stdev=29.67 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 103], 10.00th=[ 110], 20.00th=[ 121], | 30.00th=[ 130], 40.00th=[ 138], 50.00th=[ 144], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 167], 90.00th=[ 180], 95.00th=[ 192], | 99.00th=[ 253], 99.50th=[ 266], 99.90th=[ 284], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 152, max= 351, per=24.93%, avg=216.68, stdev=43.62, samples=47 iops : min= 38, max= 87, avg=53.77, stdev=10.90, samples=47 lat (msec) : 50=0.20%, 100=4.41%, 250=94.45%, 500=0.94% cpu : usr=0.27%, sys=2.19%, ctx=4774, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180269: Fri Aug 14 22:52:38 2026 read: IOPS=51, BW=208KiB/s (213kB/s)(4976KiB/23975msec) slat (usec): min=2791, max=64188, avg=10552.75, stdev=5323.87 clat (msec): min=77, max=308, avg=141.86, stdev=30.41 lat (msec): min=82, max=333, avg=152.42, stdev=31.36 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 99], 10.00th=[ 106], 20.00th=[ 116], | 30.00th=[ 125], 40.00th=[ 133], 50.00th=[ 140], 60.00th=[ 146], | 70.00th=[ 155], 80.00th=[ 165], 90.00th=[ 180], 95.00th=[ 192], | 99.00th=[ 230], 99.50th=[ 251], 99.90th=[ 292], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 106, max= 292, per=24.84%, avg=207.45, stdev=38.61, samples=47 iops : min= 26, max= 73, avg=51.45, stdev= 9.69, samples=47 write: IOPS=54, BW=220KiB/s (225kB/s)(5264KiB/23975msec); 0 zone resets slat (usec): min=63, max=18284, avg=341.42, stdev=839.14 clat (msec): min=74, max=308, avg=145.13, stdev=28.13 lat (msec): min=74, max=308, avg=145.48, stdev=28.14 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 105], 10.00th=[ 112], 20.00th=[ 122], | 30.00th=[ 130], 40.00th=[ 136], 50.00th=[ 144], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 167], 90.00th=[ 182], 95.00th=[ 194], | 99.00th=[ 220], 99.50th=[ 228], 99.90th=[ 284], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 135, max= 317, per=25.37%, avg=220.49, stdev=43.47, samples=47 iops : min= 33, max= 79, avg=54.72, stdev=10.84, samples=47 lat (msec) : 100=4.61%, 250=94.92%, 500=0.47% cpu : usr=0.23%, sys=2.17%, ctx=4714, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180270: Fri Aug 14 22:52:38 2026 read: IOPS=51, BW=208KiB/s (213kB/s)(4984KiB/24005msec) slat (usec): min=2583, max=45800, avg=10830.45, stdev=5497.73 clat (msec): min=35, max=296, avg=141.89, stdev=31.45 lat (msec): min=45, max=317, avg=152.73, stdev=32.66 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 96], 10.00th=[ 105], 20.00th=[ 115], | 30.00th=[ 126], 40.00th=[ 132], 50.00th=[ 140], 60.00th=[ 148], | 70.00th=[ 157], 80.00th=[ 167], 90.00th=[ 184], 95.00th=[ 199], | 99.00th=[ 226], 99.50th=[ 236], 99.90th=[ 288], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 99, max= 302, per=24.88%, avg=207.79, stdev=44.09, samples=47 iops : min= 24, max= 75, avg=51.53, stdev=11.00, samples=47 write: IOPS=54, BW=219KiB/s (224kB/s)(5256KiB/24005msec); 0 zone resets slat (usec): min=55, max=13618, avg=323.01, stdev=703.45 clat (msec): min=26, max=263, avg=144.93, stdev=28.04 lat (msec): min=27, max=263, avg=145.26, stdev=28.05 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 103], 10.00th=[ 110], 20.00th=[ 121], | 30.00th=[ 129], 40.00th=[ 138], 50.00th=[ 144], 60.00th=[ 153], | 70.00th=[ 159], 80.00th=[ 167], 90.00th=[ 182], 95.00th=[ 194], | 99.00th=[ 215], 99.50th=[ 220], 99.90th=[ 245], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 111, max= 303, per=25.16%, avg=218.62, stdev=45.38, samples=47 iops : min= 27, max= 75, avg=54.23, stdev=11.36, samples=47 lat (msec) : 50=0.08%, 100=5.20%, 250=94.53%, 500=0.20% cpu : usr=0.21%, sys=2.15%, ctx=4556, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180271: Fri Aug 14 22:52:38 2026 read: IOPS=53, BW=212KiB/s (217kB/s)(5060KiB/23860msec) slat (usec): min=58, max=46307, avg=10427.34, stdev=5151.86 clat (msec): min=41, max=275, avg=141.57, stdev=31.20 lat (msec): min=41, max=293, avg=152.00, stdev=32.35 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 96], 10.00th=[ 104], 20.00th=[ 115], | 30.00th=[ 125], 40.00th=[ 132], 50.00th=[ 140], 60.00th=[ 148], | 70.00th=[ 155], 80.00th=[ 165], 90.00th=[ 184], 95.00th=[ 197], | 99.00th=[ 230], 99.50th=[ 239], 99.90th=[ 264], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 136, max= 343, per=25.40%, avg=212.09, stdev=38.47, samples=47 iops : min= 34, max= 85, avg=52.62, stdev= 9.60, samples=47 write: IOPS=54, BW=217KiB/s (222kB/s)(5180KiB/23860msec); 0 zone resets slat (usec): min=57, max=12991, avg=332.18, stdev=783.34 clat (msec): min=32, max=325, avg=145.39, stdev=28.30 lat (msec): min=33, max=325, avg=145.73, stdev=28.33 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 105], 10.00th=[ 111], 20.00th=[ 122], | 30.00th=[ 131], 40.00th=[ 138], 50.00th=[ 144], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 167], 90.00th=[ 182], 95.00th=[ 192], | 99.00th=[ 224], 99.50th=[ 228], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 119, max= 310, per=24.99%, avg=217.19, stdev=39.03, samples=47 iops : min= 29, max= 77, avg=53.89, stdev= 9.78, samples=47 lat (msec) : 50=0.16%, 100=5.35%, 250=94.22%, 500=0.27% cpu : usr=0.23%, sys=2.18%, ctx=4566, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=835KiB/s (855kB/s), 208KiB/s-212KiB/s (213kB/s-217kB/s), io=19.6MiB (20.5MB), run=23860-24021msec WRITE: bw=870KiB/s (891kB/s), 216KiB/s-220KiB/s (221kB/s-225kB/s), io=20.4MiB (21.4MB), run=23860-24021msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180279: Fri Aug 14 22:52:48 2026 read: IOPS=1, BW=76.5MiB/s (80.2MB/s)(40.0MiB/523msec) slat (nsec): min=34251k, max=34251k, avg=34250871.00, stdev= 0.00 clat (nsec): min=456837k, max=456837k, avg=456836649.00, stdev= 0.00 lat (nsec): min=491092k, max=491092k, avg=491092007.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 456], 10.00th=[ 456], 20.00th=[ 456], | 30.00th=[ 456], 40.00th=[ 456], 50.00th=[ 456], 60.00th=[ 456], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.92%, ctx=11, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=76.5MiB/s (80.2MB/s), 76.5MiB/s-76.5MiB/s (80.2MB/s-80.2MB/s), io=40.0MiB (41.9MB), run=523-523msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout