== sanity test 398c: run fio to test AIO ================= 04:33:12 (1787473992) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.638566 s, 65.7 MB/s osc.lustre-OST0000-osc-ffff9aa486a9f800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=178848: Sun Aug 23 04:33:55 2026 write: IOPS=80, BW=320KiB/s (328kB/s)(10.0MiB/31970msec); 0 zone resets slat (usec): min=35, max=16245, avg=277.98, stdev=931.68 clat (msec): min=35, max=389, avg=198.37, stdev=60.98 lat (msec): min=36, max=389, avg=198.65, stdev=60.96 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 120], 10.00th=[ 134], 20.00th=[ 148], | 30.00th=[ 161], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 205], | 70.00th=[ 224], 80.00th=[ 247], 90.00th=[ 279], 95.00th=[ 321], | 99.00th=[ 368], 99.50th=[ 376], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 128, max= 509, per=24.73%, avg=316.52, stdev=84.10, samples=63 iops : min= 32, max= 127, avg=78.73, stdev=21.02, samples=63 lat (msec) : 50=0.39%, 100=2.15%, 250=79.02%, 500=18.44% cpu : usr=0.16%, sys=1.04%, ctx=2127, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178849: Sun Aug 23 04:33:55 2026 write: IOPS=80, BW=320KiB/s (328kB/s)(10.0MiB/31977msec); 0 zone resets slat (usec): min=35, max=28880, avg=273.76, stdev=1048.41 clat (msec): min=28, max=386, avg=198.27, stdev=59.50 lat (msec): min=29, max=386, avg=198.54, stdev=59.53 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 120], 10.00th=[ 129], 20.00th=[ 148], | 30.00th=[ 163], 40.00th=[ 171], 50.00th=[ 188], 60.00th=[ 207], | 70.00th=[ 224], 80.00th=[ 249], 90.00th=[ 284], 95.00th=[ 313], | 99.00th=[ 351], 99.50th=[ 368], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 190, max= 512, per=24.71%, avg=316.32, stdev=81.04, samples=63 iops : min= 47, max= 128, avg=78.68, stdev=20.28, samples=63 lat (msec) : 50=0.04%, 100=1.84%, 250=78.95%, 500=19.18% cpu : usr=0.16%, sys=0.82%, ctx=1465, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178850: Sun Aug 23 04:33:55 2026 write: IOPS=80, BW=321KiB/s (328kB/s)(10.0MiB/31946msec); 0 zone resets slat (usec): min=35, max=12105, avg=263.68, stdev=809.72 clat (msec): min=82, max=420, avg=199.19, stdev=60.50 lat (msec): min=82, max=426, avg=199.46, stdev=60.53 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 122], 10.00th=[ 132], 20.00th=[ 150], | 30.00th=[ 161], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 205], | 70.00th=[ 224], 80.00th=[ 249], 90.00th=[ 284], 95.00th=[ 313], | 99.00th=[ 363], 99.50th=[ 388], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 422] bw ( KiB/s): min= 128, max= 541, per=24.86%, avg=318.17, stdev=89.60, samples=63 iops : min= 32, max= 135, avg=79.16, stdev=22.43, samples=63 lat (msec) : 100=2.66%, 250=77.54%, 500=19.80% cpu : usr=0.14%, sys=0.94%, ctx=2133, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=178851: Sun Aug 23 04:33:55 2026 write: IOPS=80, BW=320KiB/s (328kB/s)(10.0MiB/31954msec); 0 zone resets slat (usec): min=51, max=13342, avg=301.27, stdev=906.26 clat (msec): min=22, max=393, avg=198.21, stdev=60.63 lat (msec): min=23, max=393, avg=198.52, stdev=60.65 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 115], 10.00th=[ 134], 20.00th=[ 148], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 188], 60.00th=[ 209], | 70.00th=[ 228], 80.00th=[ 247], 90.00th=[ 279], 95.00th=[ 309], | 99.00th=[ 363], 99.50th=[ 376], 99.90th=[ 388], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 134, max= 483, per=24.84%, avg=317.97, stdev=87.57, samples=63 iops : min= 33, max= 120, avg=79.10, stdev=21.89, samples=63 lat (msec) : 50=0.35%, 100=1.95%, 250=80.00%, 500=17.70% cpu : usr=0.16%, sys=1.24%, ctx=2646, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1281KiB/s (1312kB/s), 320KiB/s-321KiB/s (328kB/s-328kB/s), io=40.0MiB (41.9MB), run=31946-31977msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=178865: Sun Aug 23 04:34:41 2026 read: IOPS=34, BW=137KiB/s (141kB/s)(5048KiB/36729msec) slat (usec): min=3080, max=62088, avg=16377.79, stdev=8997.25 clat (msec): min=33, max=447, avg=216.41, stdev=52.99 lat (msec): min=42, max=467, avg=232.79, stdev=54.95 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 140], 10.00th=[ 153], 20.00th=[ 174], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 228], | 70.00th=[ 241], 80.00th=[ 257], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 376], 99.50th=[ 393], 99.90th=[ 439], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 41, max= 247, per=25.19%, avg=137.29, stdev=32.81, samples=72 iops : min= 10, max= 61, avg=33.81, stdev= 8.15, samples=72 write: IOPS=35, BW=141KiB/s (145kB/s)(5192KiB/36729msec); 0 zone resets slat (usec): min=39, max=27575, avg=341.66, stdev=1297.97 clat (msec): min=45, max=436, avg=221.65, stdev=47.94 lat (msec): min=45, max=437, avg=221.99, stdev=47.97 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 150], 10.00th=[ 163], 20.00th=[ 182], | 30.00th=[ 194], 40.00th=[ 205], 50.00th=[ 218], 60.00th=[ 230], | 70.00th=[ 249], 80.00th=[ 266], 90.00th=[ 288], 95.00th=[ 300], | 99.00th=[ 347], 99.50th=[ 355], 99.90th=[ 401], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 13, max= 231, per=24.94%, avg=141.42, stdev=36.69, samples=72 iops : min= 3, max= 57, avg=34.83, stdev= 9.15, samples=72 lat (msec) : 50=0.08%, 100=0.20%, 250=73.98%, 500=25.74% cpu : usr=0.24%, sys=1.70%, ctx=4063, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178866: Sun Aug 23 04:34:41 2026 read: IOPS=34, BW=137KiB/s (140kB/s)(4976KiB/36380msec) slat (usec): min=86, max=84832, avg=16548.06, stdev=9320.14 clat (msec): min=27, max=585, avg=214.95, stdev=56.65 lat (msec): min=37, max=600, avg=231.51, stdev=58.88 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 136], 10.00th=[ 150], 20.00th=[ 169], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 257], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 380], 99.50th=[ 409], 99.90th=[ 575], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 39, max= 223, per=24.90%, avg=135.72, stdev=35.24, samples=72 iops : min= 9, max= 55, avg=33.40, stdev= 8.80, samples=72 write: IOPS=36, BW=145KiB/s (148kB/s)(5264KiB/36380msec); 0 zone resets slat (usec): min=40, max=19977, avg=291.40, stdev=846.58 clat (msec): min=88, max=585, avg=222.32, stdev=51.17 lat (msec): min=89, max=585, avg=222.61, stdev=51.15 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 150], 10.00th=[ 163], 20.00th=[ 180], | 30.00th=[ 192], 40.00th=[ 205], 50.00th=[ 220], 60.00th=[ 232], | 70.00th=[ 247], 80.00th=[ 264], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 355], 99.50th=[ 372], 99.90th=[ 550], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 55, max= 223, per=25.41%, avg=144.08, stdev=36.28, samples=72 iops : min= 13, max= 55, avg=35.51, stdev= 9.06, samples=72 lat (msec) : 50=0.12%, 100=0.35%, 250=74.22%, 500=25.12%, 750=0.20% cpu : usr=0.22%, sys=1.72%, ctx=3962, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178867: Sun Aug 23 04:34:41 2026 read: IOPS=33, BW=136KiB/s (139kB/s)(4984KiB/36730msec) slat (usec): min=3276, max=93790, avg=16472.17, stdev=9319.05 clat (msec): min=73, max=446, avg=215.82, stdev=50.67 lat (msec): min=83, max=460, avg=232.30, stdev=52.59 clat percentiles (msec): | 1.00th=[ 120], 5.00th=[ 142], 10.00th=[ 155], 20.00th=[ 169], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 228], | 70.00th=[ 243], 80.00th=[ 257], 90.00th=[ 279], 95.00th=[ 296], | 99.00th=[ 347], 99.50th=[ 414], 99.90th=[ 443], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 48, max= 208, per=24.95%, avg=136.00, stdev=35.70, samples=72 iops : min= 12, max= 52, avg=33.47, stdev= 8.88, samples=72 write: IOPS=35, BW=143KiB/s (147kB/s)(5256KiB/36730msec); 0 zone resets slat (usec): min=42, max=11556, avg=278.65, stdev=748.66 clat (msec): min=51, max=434, avg=223.34, stdev=50.42 lat (msec): min=51, max=434, avg=223.62, stdev=50.41 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 148], 10.00th=[ 163], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 207], 50.00th=[ 222], 60.00th=[ 234], | 70.00th=[ 247], 80.00th=[ 266], 90.00th=[ 292], 95.00th=[ 309], | 99.00th=[ 368], 99.50th=[ 414], 99.90th=[ 418], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 54, max= 238, per=25.19%, avg=142.85, stdev=35.57, samples=72 iops : min= 13, max= 59, avg=35.18, stdev= 8.87, samples=72 lat (msec) : 100=0.08%, 250=73.75%, 500=26.17% cpu : usr=0.22%, sys=1.61%, ctx=3867, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=178868: Sun Aug 23 04:34:41 2026 read: IOPS=34, BW=137KiB/s (141kB/s)(5060KiB/36811msec) slat (usec): min=3232, max=97920, avg=16557.78, stdev=9234.89 clat (msec): min=26, max=416, avg=215.07, stdev=50.27 lat (msec): min=44, max=429, avg=231.64, stdev=51.85 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 142], 10.00th=[ 157], 20.00th=[ 174], | 30.00th=[ 188], 40.00th=[ 201], 50.00th=[ 213], 60.00th=[ 226], | 70.00th=[ 239], 80.00th=[ 253], 90.00th=[ 284], 95.00th=[ 300], | 99.00th=[ 355], 99.50th=[ 380], 99.90th=[ 401], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 20, max= 208, per=25.07%, avg=136.65, stdev=31.85, samples=72 iops : min= 5, max= 52, avg=33.65, stdev= 7.93, samples=72 write: IOPS=35, BW=141KiB/s (144kB/s)(5180KiB/36811msec); 0 zone resets slat (usec): min=40, max=44344, avg=358.42, stdev=1704.94 clat (msec): min=44, max=409, avg=223.61, stdev=50.23 lat (msec): min=44, max=409, avg=223.98, stdev=50.37 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 148], 10.00th=[ 163], 20.00th=[ 182], | 30.00th=[ 194], 40.00th=[ 207], 50.00th=[ 220], 60.00th=[ 236], | 70.00th=[ 251], 80.00th=[ 266], 90.00th=[ 288], 95.00th=[ 309], | 99.00th=[ 359], 99.50th=[ 376], 99.90th=[ 393], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 27, max= 279, per=24.67%, avg=139.86, stdev=35.93, samples=72 iops : min= 6, max= 69, avg=34.43, stdev= 8.98, samples=72 lat (msec) : 50=0.23%, 100=0.31%, 250=73.95%, 500=25.51% cpu : usr=0.21%, sys=1.79%, ctx=4143, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=545KiB/s (558kB/s), 136KiB/s-137KiB/s (139kB/s-141kB/s), io=19.6MiB (20.5MB), run=36380-36811msec WRITE: bw=568KiB/s (581kB/s), 141KiB/s-145KiB/s (144kB/s-148kB/s), io=20.4MiB (21.4MB), run=36380-36811msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=178879: Sun Aug 23 04:34:53 2026 read: IOPS=1, BW=59.0MiB/s (62.9MB/s)(40.0MiB/667msec) slat (nsec): min=31980k, max=31980k, avg=31980181.00, stdev= 0.00 clat (nsec): min=613177k, max=613177k, avg=613176944.00, stdev= 0.00 lat (nsec): min=645163k, max=645163k, avg=645163339.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 617], 5.00th=[ 617], 10.00th=[ 617], 20.00th=[ 617], | 30.00th=[ 617], 40.00th=[ 617], 50.00th=[ 617], 60.00th=[ 617], | 70.00th=[ 617], 80.00th=[ 617], 90.00th=[ 617], 95.00th=[ 617], | 99.00th=[ 617], 99.50th=[ 617], 99.90th=[ 617], 99.95th=[ 617], | 99.99th=[ 617] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.80%, ctx=5, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=59.0MiB/s (62.9MB/s), 59.0MiB/s-59.0MiB/s (62.9MB/s-62.9MB/s), io=40.0MiB (41.9MB), run=667-667msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout