== sanity test 398c: run fio to test AIO ================= 11:44:35 (1786635875) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.619349 s, 67.7 MB/s osc.lustre-OST0000-osc-ffff9c34d2537800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=181181: Thu Aug 13 11:45:18 2026 write: IOPS=82, BW=330KiB/s (338kB/s)(10.0MiB/31050msec); 0 zone resets slat (usec): min=51, max=15710, avg=411.90, stdev=1156.07 clat (msec): min=33, max=299, avg=192.35, stdev=38.77 lat (msec): min=34, max=299, avg=192.76, stdev=38.79 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 128], 10.00th=[ 142], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 192], 60.00th=[ 201], | 70.00th=[ 211], 80.00th=[ 228], 90.00th=[ 245], 95.00th=[ 257], | 99.00th=[ 275], 99.50th=[ 284], 99.90th=[ 296], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 230, max= 511, per=24.86%, avg=327.69, stdev=58.86, samples=61 iops : min= 57, max= 127, avg=81.54, stdev=14.72, samples=61 lat (msec) : 50=0.08%, 100=1.05%, 250=91.80%, 500=7.07% cpu : usr=0.18%, sys=1.27%, ctx=2412, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181182: Thu Aug 13 11:45:18 2026 write: IOPS=82, BW=330KiB/s (337kB/s)(10.0MiB/31074msec); 0 zone resets slat (usec): min=51, max=16158, avg=422.32, stdev=1146.87 clat (msec): min=33, max=299, avg=192.77, stdev=38.83 lat (msec): min=34, max=299, avg=193.20, stdev=38.77 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 134], 10.00th=[ 144], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 184], 50.00th=[ 192], 60.00th=[ 201], | 70.00th=[ 213], 80.00th=[ 226], 90.00th=[ 245], 95.00th=[ 257], | 99.00th=[ 279], 99.50th=[ 284], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 254, max= 519, per=24.86%, avg=327.61, stdev=56.99, samples=61 iops : min= 63, max= 129, avg=81.52, stdev=14.26, samples=61 lat (msec) : 50=0.16%, 100=0.90%, 250=91.33%, 500=7.62% cpu : usr=0.18%, sys=1.43%, ctx=2337, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181183: Thu Aug 13 11:45:18 2026 write: IOPS=82, BW=332KiB/s (340kB/s)(10.0MiB/30861msec); 0 zone resets slat (usec): min=47, max=14039, avg=380.87, stdev=951.46 clat (msec): min=25, max=357, avg=192.25, stdev=40.43 lat (msec): min=25, max=357, avg=192.64, stdev=40.47 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 126], 10.00th=[ 140], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 182], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 213], 80.00th=[ 226], 90.00th=[ 245], 95.00th=[ 257], | 99.00th=[ 275], 99.50th=[ 288], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 253, max= 503, per=25.13%, avg=331.18, stdev=57.24, samples=61 iops : min= 63, max= 125, avg=82.43, stdev=14.30, samples=61 lat (msec) : 50=0.08%, 100=1.84%, 250=90.62%, 500=7.46% cpu : usr=0.23%, sys=1.25%, ctx=2071, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181184: Thu Aug 13 11:45:18 2026 write: IOPS=82, BW=330KiB/s (338kB/s)(10.0MiB/31056msec); 0 zone resets slat (usec): min=51, max=12700, avg=358.18, stdev=889.27 clat (msec): min=70, max=309, avg=192.58, stdev=38.25 lat (msec): min=70, max=309, avg=192.94, stdev=38.27 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 130], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 186], 50.00th=[ 194], 60.00th=[ 203], | 70.00th=[ 211], 80.00th=[ 226], 90.00th=[ 243], 95.00th=[ 255], | 99.00th=[ 275], 99.50th=[ 284], 99.90th=[ 296], 99.95th=[ 309], | 99.99th=[ 309] bw ( KiB/s): min= 206, max= 503, per=24.91%, avg=328.30, stdev=55.01, samples=61 iops : min= 51, max= 125, avg=81.67, stdev=13.74, samples=61 lat (msec) : 100=0.66%, 250=93.24%, 500=6.09% cpu : usr=0.19%, sys=1.40%, ctx=2407, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1318KiB/s (1350kB/s), 330KiB/s-332KiB/s (337kB/s-340kB/s), io=40.0MiB (41.9MB), run=30861-31074msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=181199: Thu Aug 13 11:46:09 2026 read: IOPS=29, BW=118KiB/s (121kB/s)(5048KiB/42671msec) slat (usec): min=3164, max=91865, avg=18812.91, stdev=11554.48 clat (msec): min=31, max=562, avg=251.90, stdev=77.38 lat (msec): min=50, max=598, avg=270.73, stdev=81.22 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 157], 10.00th=[ 174], 20.00th=[ 192], | 30.00th=[ 207], 40.00th=[ 222], 50.00th=[ 234], 60.00th=[ 251], | 70.00th=[ 271], 80.00th=[ 309], 90.00th=[ 359], 95.00th=[ 418], | 99.00th=[ 481], 99.50th=[ 498], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 36, max= 208, per=25.13%, avg=117.86, stdev=35.96, samples=84 iops : min= 9, max= 52, avg=29.00, stdev= 9.05, samples=84 write: IOPS=30, BW=122KiB/s (125kB/s)(5192KiB/42671msec); 0 zone resets slat (usec): min=58, max=40636, avg=518.72, stdev=1757.32 clat (msec): min=58, max=557, avg=258.20, stdev=73.07 lat (msec): min=59, max=557, avg=258.72, stdev=73.16 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 167], 10.00th=[ 182], 20.00th=[ 201], | 30.00th=[ 215], 40.00th=[ 230], 50.00th=[ 243], 60.00th=[ 259], | 70.00th=[ 279], 80.00th=[ 309], 90.00th=[ 368], 95.00th=[ 405], | 99.00th=[ 477], 99.50th=[ 485], 99.90th=[ 523], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 28, max= 207, per=24.83%, avg=121.43, stdev=37.13, samples=84 iops : min= 7, max= 51, avg=29.87, stdev= 9.31, samples=84 lat (msec) : 50=0.08%, 100=0.12%, 250=57.15%, 500=42.27%, 750=0.39% cpu : usr=0.23%, sys=1.90%, ctx=4285, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181200: Thu Aug 13 11:46:09 2026 read: IOPS=29, BW=116KiB/s (119kB/s)(4976KiB/42721msec) slat (usec): min=3496, max=87285, avg=19074.76, stdev=11530.26 clat (msec): min=38, max=693, avg=253.41, stdev=80.54 lat (msec): min=59, max=711, avg=272.50, stdev=84.18 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 161], 10.00th=[ 176], 20.00th=[ 192], | 30.00th=[ 207], 40.00th=[ 224], 50.00th=[ 236], 60.00th=[ 251], | 70.00th=[ 271], 80.00th=[ 305], 90.00th=[ 363], 95.00th=[ 418], | 99.00th=[ 510], 99.50th=[ 558], 99.90th=[ 676], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 7, max= 184, per=24.58%, avg=115.29, stdev=36.62, samples=84 iops : min= 1, max= 46, avg=28.33, stdev= 9.20, samples=84 write: IOPS=30, BW=123KiB/s (126kB/s)(5264KiB/42721msec); 0 zone resets slat (usec): min=57, max=16285, avg=444.58, stdev=1318.92 clat (msec): min=60, max=632, avg=257.95, stdev=73.88 lat (msec): min=61, max=632, avg=258.40, stdev=73.95 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 169], 10.00th=[ 180], 20.00th=[ 201], | 30.00th=[ 213], 40.00th=[ 228], 50.00th=[ 243], 60.00th=[ 257], | 70.00th=[ 279], 80.00th=[ 309], 90.00th=[ 359], 95.00th=[ 414], | 99.00th=[ 485], 99.50th=[ 535], 99.90th=[ 625], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 39, max= 200, per=25.11%, avg=122.77, stdev=38.62, samples=84 iops : min= 9, max= 50, avg=30.20, stdev= 9.73, samples=84 lat (msec) : 50=0.04%, 100=0.31%, 250=56.88%, 500=41.88%, 750=0.90% cpu : usr=0.25%, sys=1.63%, ctx=3907, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181201: Thu Aug 13 11:46:09 2026 read: IOPS=29, BW=117KiB/s (120kB/s)(4984KiB/42630msec) slat (msec): min=3, max=116, avg=19.04, stdev=11.62 clat (msec): min=103, max=639, avg=251.20, stdev=78.97 lat (msec): min=126, max=657, avg=270.25, stdev=82.38 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 157], 10.00th=[ 174], 20.00th=[ 188], | 30.00th=[ 205], 40.00th=[ 220], 50.00th=[ 234], 60.00th=[ 251], | 70.00th=[ 268], 80.00th=[ 300], 90.00th=[ 355], 95.00th=[ 418], | 99.00th=[ 506], 99.50th=[ 567], 99.90th=[ 609], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 31, max= 208, per=24.94%, avg=116.95, stdev=38.93, samples=84 iops : min= 7, max= 52, avg=28.76, stdev= 9.79, samples=84 write: IOPS=30, BW=123KiB/s (126kB/s)(5256KiB/42630msec); 0 zone resets slat (usec): min=57, max=24577, avg=462.21, stdev=1409.42 clat (msec): min=39, max=581, avg=259.57, stdev=76.77 lat (msec): min=39, max=581, avg=260.03, stdev=76.84 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 167], 10.00th=[ 180], 20.00th=[ 197], | 30.00th=[ 215], 40.00th=[ 230], 50.00th=[ 245], 60.00th=[ 264], | 70.00th=[ 279], 80.00th=[ 313], 90.00th=[ 359], 95.00th=[ 414], | 99.00th=[ 514], 99.50th=[ 558], 99.90th=[ 584], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 39, max= 191, per=25.07%, avg=122.57, stdev=36.29, samples=84 iops : min= 9, max= 47, avg=30.15, stdev= 9.14, samples=84 lat (msec) : 50=0.04%, 250=55.90%, 500=42.66%, 750=1.41% cpu : usr=0.24%, sys=1.65%, ctx=4017, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181202: Thu Aug 13 11:46:09 2026 read: IOPS=29, BW=119KiB/s (122kB/s)(5060KiB/42441msec) slat (usec): min=92, max=100422, avg=19143.14, stdev=11423.58 clat (msec): min=10, max=588, avg=252.27, stdev=77.68 lat (msec): min=10, max=612, avg=271.43, stdev=81.72 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 192], | 30.00th=[ 207], 40.00th=[ 222], 50.00th=[ 236], 60.00th=[ 255], | 70.00th=[ 275], 80.00th=[ 309], 90.00th=[ 359], 95.00th=[ 414], | 99.00th=[ 485], 99.50th=[ 502], 99.90th=[ 550], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 48, max= 198, per=25.25%, avg=118.43, stdev=34.48, samples=84 iops : min= 12, max= 49, avg=29.14, stdev= 8.64, samples=84 write: IOPS=30, BW=122KiB/s (125kB/s)(5180KiB/42441msec); 0 zone resets slat (usec): min=58, max=31016, avg=531.10, stdev=1584.00 clat (msec): min=78, max=538, avg=257.90, stdev=75.11 lat (msec): min=78, max=539, avg=258.44, stdev=75.12 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 165], 10.00th=[ 178], 20.00th=[ 199], | 30.00th=[ 215], 40.00th=[ 230], 50.00th=[ 243], 60.00th=[ 259], | 70.00th=[ 279], 80.00th=[ 309], 90.00th=[ 363], 95.00th=[ 414], | 99.00th=[ 502], 99.50th=[ 514], 99.90th=[ 535], 99.95th=[ 542], | 99.99th=[ 542] bw ( KiB/s): min= 31, max= 231, per=24.80%, avg=121.26, stdev=37.82, samples=84 iops : min= 7, max= 57, avg=29.83, stdev= 9.46, samples=84 lat (msec) : 20=0.12%, 50=0.12%, 100=0.23%, 250=55.90%, 500=42.89% lat (msec) : 750=0.74% cpu : usr=0.23%, sys=1.84%, ctx=4200, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=470KiB/s (481kB/s), 116KiB/s-119KiB/s (119kB/s-122kB/s), io=19.6MiB (20.5MB), run=42441-42721msec WRITE: bw=489KiB/s (501kB/s), 122KiB/s-123KiB/s (125kB/s-126kB/s), io=20.4MiB (21.4MB), run=42441-42721msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=181212: Thu Aug 13 11:46:19 2026 read: IOPS=2, BW=84.6MiB/s (88.7MB/s)(40.0MiB/473msec) slat (nsec): min=33598k, max=33598k, avg=33598183.00, stdev= 0.00 clat (nsec): min=405719k, max=405719k, avg=405718848.00, stdev= 0.00 lat (nsec): min=439333k, max=439333k, avg=439333333.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.75%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=84.6MiB/s (88.7MB/s), 84.6MiB/s-84.6MiB/s (88.7MB/s-88.7MB/s), io=40.0MiB (41.9MB), run=473-473msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout