== sanity test 398c: run fio to test AIO ================= 22:58:44 (1788922724) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.390183 s, 107 MB/s osc.lustre-OST0000-osc-ffff9662096ab800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=150303: Tue Sep 8 22:59:31 2026 write: IOPS=70, BW=280KiB/s (287kB/s)(10.0MiB/36553msec); 0 zone resets slat (usec): min=28, max=15623, avg=268.10, stdev=955.96 clat (msec): min=15, max=628, avg=227.39, stdev=73.58 lat (msec): min=16, max=628, avg=227.66, stdev=73.58 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 169], | 30.00th=[ 182], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 226], | 70.00th=[ 253], 80.00th=[ 288], 90.00th=[ 330], 95.00th=[ 372], | 99.00th=[ 439], 99.50th=[ 506], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 127, max= 400, per=25.13%, avg=280.50, stdev=73.56, samples=72 iops : min= 31, max= 100, avg=69.74, stdev=18.38, samples=72 lat (msec) : 20=0.04%, 50=0.12%, 100=0.23%, 250=68.63%, 500=30.43% lat (msec) : 750=0.55% cpu : usr=0.20%, sys=0.85%, ctx=2388, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150304: Tue Sep 8 22:59:31 2026 write: IOPS=70, BW=280KiB/s (287kB/s)(10.0MiB/36518msec); 0 zone resets slat (usec): min=30, max=18071, avg=328.60, stdev=1246.83 clat (msec): min=13, max=592, avg=227.65, stdev=72.80 lat (msec): min=14, max=592, avg=227.98, stdev=72.81 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 140], 10.00th=[ 155], 20.00th=[ 169], | 30.00th=[ 180], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 228], | 70.00th=[ 253], 80.00th=[ 288], 90.00th=[ 330], 95.00th=[ 368], | 99.00th=[ 443], 99.50th=[ 489], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 126, max= 487, per=25.13%, avg=280.43, stdev=76.09, samples=72 iops : min= 31, max= 121, avg=69.71, stdev=19.03, samples=72 lat (msec) : 20=0.04%, 50=0.08%, 100=0.51%, 250=68.40%, 500=30.66% lat (msec) : 750=0.31% cpu : usr=0.20%, sys=0.94%, ctx=2780, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150305: Tue Sep 8 22:59:31 2026 write: IOPS=69, BW=280KiB/s (286kB/s)(10.0MiB/36625msec); 0 zone resets slat (usec): min=25, max=21804, avg=218.88, stdev=872.50 clat (msec): min=74, max=502, avg=227.90, stdev=69.43 lat (msec): min=74, max=502, avg=228.12, stdev=69.48 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 144], 10.00th=[ 155], 20.00th=[ 167], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 232], | 70.00th=[ 253], 80.00th=[ 288], 90.00th=[ 330], 95.00th=[ 372], | 99.00th=[ 422], 99.50th=[ 451], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 128, max= 447, per=24.97%, avg=278.69, stdev=79.34, samples=72 iops : min= 32, max= 111, avg=69.29, stdev=19.83, samples=72 lat (msec) : 100=0.16%, 250=68.83%, 500=30.90%, 750=0.12% cpu : usr=0.16%, sys=0.76%, ctx=1914, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150306: Tue Sep 8 22:59:31 2026 write: IOPS=69, BW=279KiB/s (286kB/s)(10.0MiB/36679msec); 0 zone resets slat (usec): min=27, max=23983, avg=272.66, stdev=1113.28 clat (msec): min=46, max=505, avg=228.16, stdev=70.31 lat (msec): min=46, max=505, avg=228.44, stdev=70.33 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 142], 10.00th=[ 155], 20.00th=[ 171], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 213], 60.00th=[ 230], | 70.00th=[ 251], 80.00th=[ 288], 90.00th=[ 330], 95.00th=[ 363], | 99.00th=[ 430], 99.50th=[ 489], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 128, max= 391, per=24.97%, avg=278.69, stdev=79.26, samples=72 iops : min= 32, max= 97, avg=69.29, stdev=19.85, samples=72 lat (msec) : 50=0.04%, 100=0.08%, 250=68.98%, 500=30.51%, 750=0.39% cpu : usr=0.17%, sys=0.77%, ctx=2084, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1117KiB/s (1144kB/s), 279KiB/s-280KiB/s (286kB/s-287kB/s), io=40.0MiB (41.9MB), run=36518-36679msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=150322: Tue Sep 8 23:00:18 2026 read: IOPS=34, BW=139KiB/s (142kB/s)(5048KiB/36405msec) slat (usec): min=2620, max=87354, avg=16054.21, stdev=9690.36 clat (msec): min=35, max=555, avg=213.05, stdev=67.55 lat (msec): min=42, max=571, avg=229.11, stdev=71.08 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 129], 10.00th=[ 142], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 234], 80.00th=[ 262], 90.00th=[ 300], 95.00th=[ 342], | 99.00th=[ 460], 99.50th=[ 493], 99.90th=[ 550], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 27, max= 255, per=25.39%, avg=139.39, stdev=44.16, samples=71 iops : min= 6, max= 63, avg=34.39, stdev=11.07, samples=71 write: IOPS=35, BW=143KiB/s (146kB/s)(5192KiB/36405msec); 0 zone resets slat (usec): min=32, max=19662, avg=280.53, stdev=920.87 clat (msec): min=102, max=522, avg=221.82, stdev=66.03 lat (msec): min=102, max=523, avg=222.10, stdev=66.07 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 136], 10.00th=[ 148], 20.00th=[ 167], | 30.00th=[ 182], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 228], | 70.00th=[ 251], 80.00th=[ 271], 90.00th=[ 305], 95.00th=[ 342], | 99.00th=[ 435], 99.50th=[ 472], 99.90th=[ 518], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 31, max= 229, per=25.14%, avg=143.55, stdev=41.51, samples=71 iops : min= 7, max= 57, avg=35.46, stdev=10.41, samples=71 lat (msec) : 50=0.04%, 100=0.27%, 250=72.85%, 500=26.52%, 750=0.31% cpu : usr=0.20%, sys=1.69%, ctx=4428, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150323: Tue Sep 8 23:00:18 2026 read: IOPS=34, BW=136KiB/s (139kB/s)(4976KiB/36536msec) slat (usec): min=2730, max=89079, avg=16402.91, stdev=9840.07 clat (msec): min=21, max=491, avg=215.39, stdev=67.83 lat (msec): min=40, max=508, avg=231.80, stdev=71.58 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 127], 10.00th=[ 142], 20.00th=[ 159], | 30.00th=[ 176], 40.00th=[ 188], 50.00th=[ 203], 60.00th=[ 220], | 70.00th=[ 241], 80.00th=[ 271], 90.00th=[ 305], 95.00th=[ 342], | 99.00th=[ 430], 99.50th=[ 451], 99.90th=[ 485], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 27, max= 250, per=24.76%, avg=135.92, stdev=42.26, samples=72 iops : min= 6, max= 62, avg=33.53, stdev=10.59, samples=72 write: IOPS=36, BW=144KiB/s (148kB/s)(5264KiB/36536msec); 0 zone resets slat (usec): min=32, max=19669, avg=226.02, stdev=814.75 clat (msec): min=100, max=457, avg=221.51, stdev=64.93 lat (msec): min=101, max=458, avg=221.74, stdev=64.88 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 133], 10.00th=[ 146], 20.00th=[ 167], | 30.00th=[ 182], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 228], | 70.00th=[ 249], 80.00th=[ 271], 90.00th=[ 305], 95.00th=[ 347], | 99.00th=[ 422], 99.50th=[ 435], 99.90th=[ 456], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 39, max= 237, per=25.21%, avg=143.94, stdev=44.50, samples=72 iops : min= 9, max= 59, avg=35.57, stdev=11.21, samples=72 lat (msec) : 50=0.04%, 100=0.35%, 250=71.56%, 500=28.05% cpu : usr=0.20%, sys=1.54%, ctx=4311, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150324: Tue Sep 8 23:00:18 2026 read: IOPS=34, BW=136KiB/s (140kB/s)(4984KiB/36526msec) slat (usec): min=2225, max=76762, avg=16097.70, stdev=9917.61 clat (msec): min=18, max=503, avg=214.37, stdev=66.61 lat (msec): min=36, max=573, avg=230.48, stdev=70.51 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 129], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 174], 40.00th=[ 192], 50.00th=[ 203], 60.00th=[ 220], | 70.00th=[ 241], 80.00th=[ 271], 90.00th=[ 305], 95.00th=[ 338], | 99.00th=[ 409], 99.50th=[ 430], 99.90th=[ 498], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 33, max= 248, per=24.87%, avg=136.51, stdev=44.75, samples=72 iops : min= 8, max= 62, avg=33.69, stdev=11.21, samples=72 write: IOPS=35, BW=144KiB/s (147kB/s)(5256KiB/36526msec); 0 zone resets slat (usec): min=34, max=28279, avg=290.78, stdev=1332.71 clat (msec): min=94, max=504, avg=222.71, stdev=63.20 lat (msec): min=94, max=504, avg=223.00, stdev=63.28 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 169], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 211], 60.00th=[ 230], | 70.00th=[ 251], 80.00th=[ 271], 90.00th=[ 305], 95.00th=[ 342], | 99.00th=[ 405], 99.50th=[ 435], 99.90th=[ 489], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 44, max= 248, per=25.12%, avg=143.43, stdev=43.55, samples=72 iops : min= 11, max= 62, avg=35.43, stdev=10.91, samples=72 lat (msec) : 20=0.04%, 100=0.39%, 250=71.21%, 500=28.28%, 750=0.08% cpu : usr=0.32%, sys=1.67%, ctx=4512, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150325: Tue Sep 8 23:00:18 2026 read: IOPS=34, BW=139KiB/s (142kB/s)(5060KiB/36401msec) slat (usec): min=41, max=75052, avg=15902.87, stdev=10034.01 clat (msec): min=35, max=539, avg=214.49, stdev=67.41 lat (msec): min=35, max=577, avg=230.40, stdev=71.29 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 130], 10.00th=[ 142], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 188], 50.00th=[ 203], 60.00th=[ 220], | 70.00th=[ 241], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 342], | 99.00th=[ 430], 99.50th=[ 460], 99.90th=[ 481], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 54, max= 264, per=25.45%, avg=139.75, stdev=40.64, samples=71 iops : min= 13, max= 66, avg=34.48, stdev=10.17, samples=71 write: IOPS=35, BW=142KiB/s (146kB/s)(5180KiB/36401msec); 0 zone resets slat (usec): min=36, max=16093, avg=249.23, stdev=911.07 clat (msec): min=73, max=619, avg=223.60, stdev=66.21 lat (msec): min=73, max=619, avg=223.85, stdev=66.22 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 136], 10.00th=[ 150], 20.00th=[ 171], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 213], 60.00th=[ 228], | 70.00th=[ 249], 80.00th=[ 271], 90.00th=[ 309], 95.00th=[ 342], | 99.00th=[ 430], 99.50th=[ 477], 99.90th=[ 575], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 39, max= 255, per=24.94%, avg=142.39, stdev=39.89, samples=71 iops : min= 9, max= 63, avg=35.17, stdev=10.00, samples=71 lat (msec) : 50=0.16%, 100=0.20%, 250=72.46%, 500=26.91%, 750=0.27% cpu : usr=0.24%, sys=1.70%, ctx=4263, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=549KiB/s (562kB/s), 136KiB/s-139KiB/s (139kB/s-142kB/s), io=19.6MiB (20.5MB), run=36401-36536msec WRITE: bw=572KiB/s (586kB/s), 142KiB/s-144KiB/s (146kB/s-148kB/s), io=20.4MiB (21.4MB), run=36401-36536msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=150334: Tue Sep 8 23:00:29 2026 read: IOPS=1, BW=52.2MiB/s (54.7MB/s)(40.0MiB/767msec) slat (nsec): min=55659k, max=55659k, avg=55658644.00, stdev= 0.00 clat (nsec): min=694200k, max=694200k, avg=694200054.00, stdev= 0.00 lat (nsec): min=749863k, max=749863k, avg=749863085.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 693], 5.00th=[ 693], 10.00th=[ 693], 20.00th=[ 693], | 30.00th=[ 693], 40.00th=[ 693], 50.00th=[ 693], 60.00th=[ 693], | 70.00th=[ 693], 80.00th=[ 693], 90.00th=[ 693], 95.00th=[ 693], | 99.00th=[ 693], 99.50th=[ 693], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.44%, ctx=8, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=52.2MiB/s (54.7MB/s), 52.2MiB/s-52.2MiB/s (54.7MB/s-54.7MB/s), io=40.0MiB (41.9MB), run=767-767msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout