== sanity test 398c: run fio to test AIO ================= 16:27:19 (1786825639) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.629433 s, 66.6 MB/s osc.lustre-OST0000-osc-ffff8efb603bc000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179988: Sat Aug 15 16:27:49 2026 write: IOPS=116, BW=466KiB/s (478kB/s)(10.0MiB/21952msec); 0 zone resets slat (usec): min=34, max=14122, avg=279.20, stdev=775.60 clat (msec): min=30, max=294, avg=136.86, stdev=40.30 lat (msec): min=30, max=295, avg=137.14, stdev=40.34 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 71], 10.00th=[ 84], 20.00th=[ 103], | 30.00th=[ 116], 40.00th=[ 126], 50.00th=[ 134], 60.00th=[ 146], | 70.00th=[ 155], 80.00th=[ 169], 90.00th=[ 190], 95.00th=[ 207], | 99.00th=[ 249], 99.50th=[ 257], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 296] bw ( KiB/s): min= 262, max= 768, per=24.92%, avg=464.51, stdev=109.98, samples=43 iops : min= 65, max= 192, avg=115.86, stdev=27.56, samples=43 lat (msec) : 50=0.16%, 100=18.28%, 250=80.90%, 500=0.66% cpu : usr=0.16%, sys=1.55%, ctx=2750, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179989: Sat Aug 15 16:27:49 2026 write: IOPS=116, BW=466KiB/s (477kB/s)(10.0MiB/21971msec); 0 zone resets slat (usec): min=36, max=18862, avg=297.24, stdev=864.99 clat (msec): min=16, max=283, avg=136.03, stdev=40.89 lat (msec): min=17, max=283, avg=136.33, stdev=40.93 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 72], 10.00th=[ 84], 20.00th=[ 104], | 30.00th=[ 113], 40.00th=[ 125], 50.00th=[ 132], 60.00th=[ 142], | 70.00th=[ 153], 80.00th=[ 167], 90.00th=[ 190], 95.00th=[ 211], | 99.00th=[ 245], 99.50th=[ 266], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 278, max= 784, per=24.92%, avg=464.53, stdev=112.70, samples=43 iops : min= 69, max= 196, avg=115.81, stdev=28.26, samples=43 lat (msec) : 20=0.04%, 50=0.04%, 100=17.54%, 250=81.72%, 500=0.66% cpu : usr=0.23%, sys=1.50%, ctx=2857, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179990: Sat Aug 15 16:27:49 2026 write: IOPS=116, BW=467KiB/s (478kB/s)(10.0MiB/21916msec); 0 zone resets slat (usec): min=36, max=37211, avg=307.87, stdev=1108.49 clat (msec): min=18, max=284, avg=135.80, stdev=41.38 lat (msec): min=20, max=284, avg=136.11, stdev=41.40 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 70], 10.00th=[ 83], 20.00th=[ 103], | 30.00th=[ 114], 40.00th=[ 124], 50.00th=[ 133], 60.00th=[ 142], | 70.00th=[ 153], 80.00th=[ 169], 90.00th=[ 190], 95.00th=[ 211], | 99.00th=[ 255], 99.50th=[ 262], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 262, max= 768, per=24.95%, avg=465.12, stdev=113.15, samples=43 iops : min= 65, max= 192, avg=115.95, stdev=28.37, samples=43 lat (msec) : 20=0.04%, 50=0.39%, 100=17.54%, 250=80.47%, 500=1.56% cpu : usr=0.21%, sys=1.55%, ctx=2802, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179991: Sat Aug 15 16:27:49 2026 write: IOPS=116, BW=467KiB/s (478kB/s)(10.0MiB/21940msec); 0 zone resets slat (usec): min=36, max=25789, avg=318.74, stdev=1013.47 clat (msec): min=20, max=283, avg=136.67, stdev=41.01 lat (msec): min=20, max=283, avg=136.99, stdev=41.05 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 73], 10.00th=[ 85], 20.00th=[ 103], | 30.00th=[ 113], 40.00th=[ 125], 50.00th=[ 134], 60.00th=[ 144], | 70.00th=[ 155], 80.00th=[ 169], 90.00th=[ 190], 95.00th=[ 211], | 99.00th=[ 249], 99.50th=[ 275], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 262, max= 768, per=24.98%, avg=465.65, stdev=107.11, samples=43 iops : min= 65, max= 192, avg=116.09, stdev=26.82, samples=43 lat (msec) : 50=0.20%, 100=18.12%, 250=80.86%, 500=0.82% cpu : usr=0.21%, sys=1.65%, ctx=2802, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1864KiB/s (1909kB/s), 466KiB/s-467KiB/s (477kB/s-478kB/s), io=40.0MiB (41.9MB), run=21916-21971msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180005: Sat Aug 15 16:28:27 2026 read: IOPS=43, BW=175KiB/s (179kB/s)(5048KiB/28848msec) slat (usec): min=84, max=67661, avg=13135.02, stdev=8233.78 clat (msec): min=17, max=415, avg=170.07, stdev=54.36 lat (msec): min=17, max=447, avg=183.21, stdev=56.71 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 104], 10.00th=[ 112], 20.00th=[ 126], | 30.00th=[ 140], 40.00th=[ 150], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 188], 80.00th=[ 207], 90.00th=[ 234], 95.00th=[ 271], | 99.00th=[ 355], 99.50th=[ 372], 99.90th=[ 414], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 79, max= 303, per=25.21%, avg=174.21, stdev=49.32, samples=57 iops : min= 19, max= 75, avg=43.11, stdev=12.39, samples=57 write: IOPS=44, BW=180KiB/s (184kB/s)(5192KiB/28848msec); 0 zone resets slat (usec): min=35, max=10687, avg=327.97, stdev=824.89 clat (msec): min=40, max=395, avg=176.52, stdev=50.31 lat (msec): min=41, max=395, avg=176.85, stdev=50.33 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 110], 10.00th=[ 120], 20.00th=[ 134], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 213], 90.00th=[ 239], 95.00th=[ 279], | 99.00th=[ 326], 99.50th=[ 351], 99.90th=[ 388], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 78, max= 287, per=24.98%, avg=179.63, stdev=47.99, samples=57 iops : min= 19, max= 71, avg=44.46, stdev=11.99, samples=57 lat (msec) : 20=0.08%, 50=0.12%, 100=3.09%, 250=89.34%, 500=7.38% cpu : usr=0.29%, sys=2.26%, ctx=4535, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180006: Sat Aug 15 16:28:27 2026 read: IOPS=42, BW=171KiB/s (175kB/s)(4976KiB/29036msec) slat (usec): min=2571, max=59369, avg=13180.58, stdev=8043.10 clat (msec): min=63, max=409, avg=170.70, stdev=54.82 lat (msec): min=87, max=425, avg=183.89, stdev=57.13 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 102], 10.00th=[ 111], 20.00th=[ 127], | 30.00th=[ 140], 40.00th=[ 150], 50.00th=[ 161], 60.00th=[ 176], | 70.00th=[ 190], 80.00th=[ 207], 90.00th=[ 239], 95.00th=[ 279], | 99.00th=[ 359], 99.50th=[ 372], 99.90th=[ 405], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 72, max= 303, per=24.69%, avg=170.60, stdev=49.87, samples=57 iops : min= 18, max= 75, avg=42.18, stdev=12.51, samples=57 write: IOPS=45, BW=181KiB/s (186kB/s)(5264KiB/29036msec); 0 zone resets slat (usec): min=42, max=8192, avg=327.45, stdev=745.23 clat (msec): min=72, max=399, avg=176.65, stdev=48.39 lat (msec): min=73, max=399, avg=176.99, stdev=48.40 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 110], 10.00th=[ 122], 20.00th=[ 136], | 30.00th=[ 150], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 209], 90.00th=[ 239], 95.00th=[ 271], | 99.00th=[ 334], 99.50th=[ 355], 99.90th=[ 388], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 63, max= 255, per=25.23%, avg=181.37, stdev=46.18, samples=57 iops : min= 15, max= 63, avg=44.89, stdev=11.57, samples=57 lat (msec) : 100=3.28%, 250=88.95%, 500=7.77% cpu : usr=0.32%, sys=2.11%, ctx=4778, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180007: Sat Aug 15 16:28:27 2026 read: IOPS=43, BW=173KiB/s (177kB/s)(4984KiB/28890msec) slat (usec): min=2260, max=53987, avg=12990.98, stdev=7826.26 clat (msec): min=65, max=408, avg=169.99, stdev=52.47 lat (msec): min=91, max=417, avg=182.98, stdev=55.06 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 103], 10.00th=[ 111], 20.00th=[ 127], | 30.00th=[ 140], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 176], | 70.00th=[ 186], 80.00th=[ 205], 90.00th=[ 232], 95.00th=[ 271], | 99.00th=[ 351], 99.50th=[ 368], 99.90th=[ 388], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 55, max= 295, per=24.98%, avg=172.61, stdev=47.07, samples=57 iops : min= 13, max= 73, avg=42.68, stdev=11.84, samples=57 write: IOPS=45, BW=182KiB/s (186kB/s)(5256KiB/28890msec); 0 zone resets slat (usec): min=36, max=13464, avg=319.32, stdev=945.19 clat (msec): min=62, max=372, avg=175.48, stdev=48.79 lat (msec): min=62, max=372, avg=175.81, stdev=48.79 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 111], 10.00th=[ 122], 20.00th=[ 138], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 192], 80.00th=[ 209], 90.00th=[ 236], 95.00th=[ 271], | 99.00th=[ 338], 99.50th=[ 351], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 80, max= 316, per=25.24%, avg=181.46, stdev=54.04, samples=57 iops : min= 20, max= 79, avg=44.91, stdev=13.53, samples=57 lat (msec) : 100=2.77%, 250=89.84%, 500=7.38% cpu : usr=0.26%, sys=2.27%, ctx=4479, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180008: Sat Aug 15 16:28:27 2026 read: IOPS=43, BW=174KiB/s (178kB/s)(5060KiB/29041msec) slat (usec): min=2398, max=68251, avg=13128.53, stdev=8006.54 clat (msec): min=17, max=401, avg=170.74, stdev=52.71 lat (msec): min=37, max=426, avg=183.88, stdev=55.13 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 101], 10.00th=[ 110], 20.00th=[ 128], | 30.00th=[ 140], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 178], | 70.00th=[ 192], 80.00th=[ 209], 90.00th=[ 232], 95.00th=[ 271], | 99.00th=[ 347], 99.50th=[ 359], 99.90th=[ 384], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 58, max= 255, per=25.10%, avg=173.42, stdev=46.91, samples=57 iops : min= 14, max= 63, avg=42.91, stdev=11.76, samples=57 write: IOPS=44, BW=178KiB/s (183kB/s)(5180KiB/29041msec); 0 zone resets slat (usec): min=38, max=13801, avg=347.03, stdev=1077.11 clat (msec): min=37, max=350, avg=175.58, stdev=48.42 lat (msec): min=38, max=350, avg=175.93, stdev=48.46 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 113], 10.00th=[ 121], 20.00th=[ 136], | 30.00th=[ 146], 40.00th=[ 157], 50.00th=[ 169], 60.00th=[ 184], | 70.00th=[ 194], 80.00th=[ 211], 90.00th=[ 236], 95.00th=[ 271], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 347], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 79, max= 293, per=24.63%, avg=177.11, stdev=47.48, samples=57 iops : min= 19, max= 73, avg=43.84, stdev=11.88, samples=57 lat (msec) : 20=0.04%, 50=0.20%, 100=3.05%, 250=89.38%, 500=7.34% cpu : usr=0.24%, sys=2.26%, ctx=4569, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=691KiB/s (708kB/s), 171KiB/s-175KiB/s (175kB/s-179kB/s), io=19.6MiB (20.5MB), run=28848-29041msec WRITE: bw=719KiB/s (737kB/s), 178KiB/s-182KiB/s (183kB/s-186kB/s), io=20.4MiB (21.4MB), run=28848-29041msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180017: Sat Aug 15 16:28:40 2026 read: IOPS=1, BW=74.5MiB/s (78.1MB/s)(40.0MiB/537msec) slat (nsec): min=54481k, max=54481k, avg=54481204.00, stdev= 0.00 clat (nsec): min=457052k, max=457052k, avg=457052297.00, stdev= 0.00 lat (nsec): min=511538k, max=511538k, avg=511537768.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 456], 10.00th=[ 456], 20.00th=[ 456], | 30.00th=[ 456], 40.00th=[ 456], 50.00th=[ 456], 60.00th=[ 456], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] lat (msec) : 500=100.00% cpu : usr=0.56%, sys=3.54%, ctx=14, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=74.5MiB/s (78.1MB/s), 74.5MiB/s-74.5MiB/s (78.1MB/s-78.1MB/s), io=40.0MiB (41.9MB), run=537-537msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout