== sanity test 398c: run fio to test AIO ================= 18:07:39 (1790978859) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.718898 s, 58.3 MB/s osc.lustre-OST0000-osc-ffff96030735e800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180690: Fri Oct 2 18:08:17 2026 write: IOPS=99, BW=397KiB/s (406kB/s)(10.0MiB/25803msec); 0 zone resets slat (usec): min=22, max=23696, avg=175.91, stdev=715.45 clat (msec): min=17, max=328, avg=160.32, stdev=46.60 lat (msec): min=17, max=328, avg=160.49, stdev=46.55 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 89], 10.00th=[ 105], 20.00th=[ 123], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 155], 60.00th=[ 165], | 70.00th=[ 182], 80.00th=[ 199], 90.00th=[ 224], 95.00th=[ 239], | 99.00th=[ 292], 99.50th=[ 309], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 253, max= 742, per=24.87%, avg=394.40, stdev=96.06, samples=50 iops : min= 63, max= 185, avg=98.14, stdev=24.05, samples=50 lat (msec) : 20=0.08%, 50=0.04%, 100=7.66%, 250=88.44%, 500=3.79% cpu : usr=0.14%, sys=0.96%, ctx=2424, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180691: Fri Oct 2 18:08:17 2026 write: IOPS=99, BW=400KiB/s (409kB/s)(10.0MiB/25618msec); 0 zone resets slat (usec): min=21, max=20748, avg=173.47, stdev=684.13 clat (msec): min=17, max=339, avg=159.79, stdev=47.62 lat (msec): min=17, max=339, avg=159.96, stdev=47.64 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 82], 10.00th=[ 105], 20.00th=[ 125], | 30.00th=[ 134], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 165], | 70.00th=[ 182], 80.00th=[ 201], 90.00th=[ 222], 95.00th=[ 247], | 99.00th=[ 292], 99.50th=[ 305], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 253, max= 735, per=25.12%, avg=398.40, stdev=97.23, samples=50 iops : min= 63, max= 183, avg=99.14, stdev=24.31, samples=50 lat (msec) : 20=0.04%, 50=0.16%, 100=8.87%, 250=86.88%, 500=4.06% cpu : usr=0.21%, sys=0.94%, ctx=2407, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180692: Fri Oct 2 18:08:17 2026 write: IOPS=99, BW=397KiB/s (407kB/s)(10.0MiB/25783msec); 0 zone resets slat (usec): min=22, max=15631, avg=166.95, stdev=638.37 clat (msec): min=10, max=339, avg=160.35, stdev=46.27 lat (msec): min=10, max=339, avg=160.52, stdev=46.27 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 90], 10.00th=[ 105], 20.00th=[ 125], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 165], | 70.00th=[ 184], 80.00th=[ 199], 90.00th=[ 222], 95.00th=[ 243], | 99.00th=[ 284], 99.50th=[ 305], 99.90th=[ 334], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 253, max= 727, per=24.85%, avg=394.18, stdev=97.40, samples=50 iops : min= 63, max= 181, avg=98.10, stdev=24.38, samples=50 lat (msec) : 20=0.04%, 50=0.08%, 100=7.93%, 250=88.09%, 500=3.87% cpu : usr=0.18%, sys=1.04%, ctx=2268, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180693: Fri Oct 2 18:08:17 2026 write: IOPS=99, BW=397KiB/s (406kB/s)(10.0MiB/25817msec); 0 zone resets slat (usec): min=22, max=11852, avg=152.71, stdev=522.22 clat (msec): min=25, max=327, avg=160.29, stdev=46.51 lat (msec): min=25, max=327, avg=160.44, stdev=46.52 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 89], 10.00th=[ 106], 20.00th=[ 125], | 30.00th=[ 134], 40.00th=[ 144], 50.00th=[ 155], 60.00th=[ 169], | 70.00th=[ 182], 80.00th=[ 199], 90.00th=[ 220], 95.00th=[ 243], | 99.00th=[ 292], 99.50th=[ 321], 99.90th=[ 321], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 245, max= 735, per=24.81%, avg=393.50, stdev=100.01, samples=50 iops : min= 61, max= 183, avg=97.88, stdev=25.00, samples=50 lat (msec) : 50=0.04%, 100=7.89%, 250=88.32%, 500=3.75% cpu : usr=0.15%, sys=0.80%, ctx=1935, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1587KiB/s (1625kB/s), 397KiB/s-400KiB/s (406kB/s-409kB/s), io=40.0MiB (41.9MB), run=25618-25817msec 0% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180707: Fri Oct 2 18:09:01 2026 read: IOPS=36, BW=146KiB/s (149kB/s)(5048KiB/34645msec) slat (usec): min=2537, max=70018, avg=15060.05, stdev=9136.49 clat (msec): min=80, max=441, avg=204.97, stdev=54.53 lat (msec): min=84, max=450, avg=220.03, stdev=56.80 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 129], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 247], 90.00th=[ 279], 95.00th=[ 313], | 99.00th=[ 351], 99.50th=[ 384], 99.90th=[ 426], 99.95th=[ 443], | 99.99th=[ 443] bw ( KiB/s): min= 55, max= 271, per=25.22%, avg=145.54, stdev=39.52, samples=68 iops : min= 13, max= 67, avg=35.91, stdev= 9.88, samples=68 write: IOPS=37, BW=150KiB/s (153kB/s)(5192KiB/34645msec); 0 zone resets slat (usec): min=27, max=7772, avg=179.32, stdev=487.47 clat (msec): min=26, max=451, avg=209.48, stdev=54.60 lat (msec): min=26, max=452, avg=209.66, stdev=54.58 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 130], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 249], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 368], 99.50th=[ 376], 99.90th=[ 451], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 55, max= 263, per=24.99%, avg=150.21, stdev=40.69, samples=68 iops : min= 13, max= 65, avg=37.06, stdev=10.15, samples=68 lat (msec) : 50=0.04%, 100=0.78%, 250=80.00%, 500=19.18% cpu : usr=0.27%, sys=1.47%, ctx=4007, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180708: Fri Oct 2 18:09:01 2026 read: IOPS=36, BW=144KiB/s (148kB/s)(4976KiB/34525msec) slat (usec): min=41, max=63180, avg=15450.15, stdev=9239.67 clat (msec): min=13, max=473, avg=205.11, stdev=56.75 lat (msec): min=13, max=498, avg=220.57, stdev=58.88 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 127], 10.00th=[ 144], 20.00th=[ 163], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 197], 60.00th=[ 209], | 70.00th=[ 226], 80.00th=[ 249], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 359], 99.50th=[ 414], 99.90th=[ 464], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 47, max= 240, per=24.86%, avg=143.47, stdev=37.84, samples=68 iops : min= 11, max= 60, avg=35.38, stdev= 9.47, samples=68 write: IOPS=38, BW=152KiB/s (156kB/s)(5264KiB/34525msec); 0 zone resets slat (usec): min=23, max=15421, avg=204.00, stdev=724.25 clat (msec): min=49, max=473, avg=210.35, stdev=55.02 lat (msec): min=49, max=473, avg=210.56, stdev=55.05 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 130], 10.00th=[ 150], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 218], | 70.00th=[ 232], 80.00th=[ 253], 90.00th=[ 284], 95.00th=[ 313], | 99.00th=[ 368], 99.50th=[ 380], 99.90th=[ 422], 99.95th=[ 472], | 99.99th=[ 472] bw ( KiB/s): min= 71, max= 239, per=25.35%, avg=152.37, stdev=40.19, samples=68 iops : min= 17, max= 59, avg=37.60, stdev=10.05, samples=68 lat (msec) : 20=0.04%, 50=0.23%, 100=0.82%, 250=78.48%, 500=20.43% cpu : usr=0.29%, sys=1.66%, ctx=4033, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180709: Fri Oct 2 18:09:01 2026 read: IOPS=35, BW=144KiB/s (147kB/s)(4984KiB/34644msec) slat (usec): min=2638, max=62174, avg=15571.86, stdev=9307.05 clat (msec): min=12, max=412, avg=204.54, stdev=55.59 lat (msec): min=23, max=427, avg=220.12, stdev=57.74 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 125], 10.00th=[ 144], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 197], 60.00th=[ 211], | 70.00th=[ 228], 80.00th=[ 251], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 351], 99.50th=[ 368], 99.90th=[ 397], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 55, max= 223, per=24.93%, avg=143.84, stdev=39.85, samples=68 iops : min= 13, max= 55, avg=35.49, stdev=10.03, samples=68 write: IOPS=37, BW=152KiB/s (155kB/s)(5256KiB/34644msec); 0 zone resets slat (usec): min=30, max=9687, avg=213.38, stdev=698.29 clat (msec): min=93, max=404, avg=209.18, stdev=52.34 lat (msec): min=93, max=404, avg=209.40, stdev=52.35 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 133], 10.00th=[ 150], 20.00th=[ 169], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 232], 80.00th=[ 253], 90.00th=[ 279], 95.00th=[ 309], | 99.00th=[ 359], 99.50th=[ 376], 99.90th=[ 393], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 55, max= 287, per=25.21%, avg=151.50, stdev=46.78, samples=68 iops : min= 13, max= 71, avg=37.40, stdev=11.64, samples=68 lat (msec) : 20=0.04%, 100=1.13%, 250=78.36%, 500=20.47% cpu : usr=0.20%, sys=1.56%, ctx=4030, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180710: Fri Oct 2 18:09:01 2026 read: IOPS=36, BW=146KiB/s (149kB/s)(5060KiB/34760msec) slat (usec): min=71, max=79227, avg=15629.78, stdev=9373.15 clat (msec): min=17, max=406, avg=204.82, stdev=56.34 lat (msec): min=24, max=420, avg=220.46, stdev=58.53 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 120], 10.00th=[ 142], 20.00th=[ 161], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 211], | 70.00th=[ 228], 80.00th=[ 249], 90.00th=[ 279], 95.00th=[ 309], | 99.00th=[ 359], 99.50th=[ 376], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 49, max= 224, per=25.04%, avg=144.50, stdev=36.63, samples=68 iops : min= 12, max= 56, avg=35.63, stdev= 9.17, samples=68 write: IOPS=37, BW=149KiB/s (153kB/s)(5180KiB/34760msec); 0 zone resets slat (usec): min=24, max=9880, avg=196.96, stdev=669.31 clat (msec): min=19, max=410, avg=210.52, stdev=54.81 lat (msec): min=19, max=410, avg=210.72, stdev=54.78 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 126], 10.00th=[ 150], 20.00th=[ 169], | 30.00th=[ 182], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 215], | 70.00th=[ 232], 80.00th=[ 253], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 359], 99.50th=[ 380], 99.90th=[ 409], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 62, max= 279, per=24.62%, avg=147.94, stdev=42.97, samples=68 iops : min= 15, max= 69, avg=36.51, stdev=10.73, samples=68 lat (msec) : 20=0.08%, 50=0.35%, 100=1.09%, 250=78.09%, 500=20.39% cpu : usr=0.27%, sys=1.55%, ctx=3916, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=577KiB/s (591kB/s), 144KiB/s-146KiB/s (147kB/s-149kB/s), io=19.6MiB (20.5MB), run=34525-34760msec WRITE: bw=601KiB/s (615kB/s), 149KiB/s-152KiB/s (153kB/s-156kB/s), io=20.4MiB (21.4MB), run=34525-34760msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180719: Fri Oct 2 18:09:14 2026 read: IOPS=2, BW=81.6MiB/s (85.6MB/s)(40.0MiB/490msec) slat (nsec): min=19162k, max=19162k, avg=19162351.00, stdev= 0.00 clat (nsec): min=448361k, max=448361k, avg=448360666.00, stdev= 0.00 lat (nsec): min=467528k, max=467528k, avg=467527885.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 447], 10.00th=[ 447], 20.00th=[ 447], | 30.00th=[ 447], 40.00th=[ 447], 50.00th=[ 447], 60.00th=[ 447], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.45%, ctx=5, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=81.6MiB/s (85.6MB/s), 81.6MiB/s-81.6MiB/s (85.6MB/s-85.6MB/s), io=40.0MiB (41.9MB), run=490-490msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout