== sanity test 398c: run fio to test AIO ================= 11:57:33 (1781625453) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.410366 s, 102 MB/s osc.lustre-OST0000-osc-ffff928860cd8000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147906: Tue Jun 16 11:59:39 2026 write: IOPS=21, BW=87.3KiB/s (89.4kB/s)(10.0MiB/117323msec); 0 zone resets slat (usec): min=38, max=17071, avg=257.59, stdev=718.33 clat (msec): min=279, max=1295, avg=732.15, stdev=162.01 lat (msec): min=279, max=1295, avg=732.41, stdev=162.02 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 468], 10.00th=[ 542], 20.00th=[ 609], | 30.00th=[ 659], 40.00th=[ 693], 50.00th=[ 735], 60.00th=[ 768], | 70.00th=[ 810], 80.00th=[ 860], 90.00th=[ 927], 95.00th=[ 1011], | 99.00th=[ 1116], 99.50th=[ 1167], 99.90th=[ 1267], 99.95th=[ 1284], | 99.99th=[ 1301] bw ( KiB/s): min= 15, max= 256, per=24.94%, avg=87.03, stdev=33.17, samples=232 iops : min= 3, max= 64, avg=21.37, stdev= 8.33, samples=232 lat (msec) : 500=6.80%, 750=48.52%, 1000=39.30%, 2000=5.39% cpu : usr=0.03%, sys=0.34%, ctx=2589, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147907: Tue Jun 16 11:59:39 2026 write: IOPS=21, BW=87.3KiB/s (89.4kB/s)(10.0MiB/117300msec); 0 zone resets slat (usec): min=38, max=23028, avg=268.55, stdev=776.45 clat (msec): min=200, max=1318, avg=732.01, stdev=166.00 lat (msec): min=200, max=1318, avg=732.28, stdev=165.98 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 464], 10.00th=[ 542], 20.00th=[ 609], | 30.00th=[ 642], 40.00th=[ 684], 50.00th=[ 726], 60.00th=[ 768], | 70.00th=[ 810], 80.00th=[ 860], 90.00th=[ 936], 95.00th=[ 1020], | 99.00th=[ 1133], 99.50th=[ 1167], 99.90th=[ 1301], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 7, max= 224, per=24.86%, avg=86.76, stdev=29.42, samples=233 iops : min= 1, max= 56, avg=21.30, stdev= 7.39, samples=233 lat (msec) : 250=0.23%, 500=6.52%, 750=48.55%, 1000=39.06%, 2000=5.62% cpu : usr=0.05%, sys=0.35%, ctx=2776, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147908: Tue Jun 16 11:59:39 2026 write: IOPS=21, BW=87.4KiB/s (89.5kB/s)(10.0MiB/117219msec); 0 zone resets slat (usec): min=39, max=20659, avg=258.16, stdev=769.01 clat (msec): min=151, max=1278, avg=731.98, stdev=168.11 lat (msec): min=151, max=1278, avg=732.24, stdev=168.11 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 464], 10.00th=[ 535], 20.00th=[ 609], | 30.00th=[ 651], 40.00th=[ 693], 50.00th=[ 735], 60.00th=[ 776], | 70.00th=[ 810], 80.00th=[ 860], 90.00th=[ 927], 95.00th=[ 1020], | 99.00th=[ 1150], 99.50th=[ 1200], 99.90th=[ 1267], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 16, max= 256, per=25.07%, avg=87.51, stdev=29.88, samples=231 iops : min= 4, max= 64, avg=21.49, stdev= 7.50, samples=231 lat (msec) : 250=0.62%, 500=5.98%, 750=48.05%, 1000=39.30%, 2000=6.05% cpu : usr=0.06%, sys=0.34%, ctx=2408, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147909: Tue Jun 16 11:59:39 2026 write: IOPS=21, BW=87.3KiB/s (89.4kB/s)(10.0MiB/117327msec); 0 zone resets slat (usec): min=36, max=15869, avg=254.49, stdev=725.52 clat (msec): min=256, max=1292, avg=732.26, stdev=162.68 lat (msec): min=256, max=1292, avg=732.53, stdev=162.65 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 447], 10.00th=[ 542], 20.00th=[ 609], | 30.00th=[ 651], 40.00th=[ 684], 50.00th=[ 735], 60.00th=[ 776], | 70.00th=[ 818], 80.00th=[ 860], 90.00th=[ 927], 95.00th=[ 1020], | 99.00th=[ 1116], 99.50th=[ 1183], 99.90th=[ 1267], 99.95th=[ 1284], | 99.99th=[ 1301] bw ( KiB/s): min= 15, max= 248, per=24.83%, avg=86.67, stdev=31.01, samples=233 iops : min= 3, max= 62, avg=21.28, stdev= 7.78, samples=233 lat (msec) : 500=6.99%, 750=48.32%, 1000=39.10%, 2000=5.59% cpu : usr=0.07%, sys=0.33%, ctx=2563, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=349KiB/s (357kB/s), 87.3KiB/s-87.4KiB/s (89.4kB/s-89.5kB/s), io=40.0MiB (41.9MB), run=117219-117327msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147936: Tue Jun 16 12:00:58 2026 read: IOPS=17, BW=70.2KiB/s (71.9kB/s)(5048KiB/71897msec) slat (usec): min=57, max=42979, avg=10234.64, stdev=5764.46 clat (msec): min=12, max=691, avg=284.60, stdev=75.94 lat (msec): min=12, max=707, avg=294.85, stdev=76.99 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 180], 10.00th=[ 197], 20.00th=[ 222], | 30.00th=[ 243], 40.00th=[ 259], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 317], 80.00th=[ 347], 90.00th=[ 388], 95.00th=[ 426], | 99.00th=[ 477], 99.50th=[ 493], 99.90th=[ 651], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 8, max= 144, per=25.01%, avg=69.52, stdev=25.36, samples=143 iops : min= 2, max= 36, avg=16.99, stdev= 6.39, samples=143 write: IOPS=18, BW=72.2KiB/s (73.9kB/s)(5192KiB/71897msec); 0 zone resets slat (usec): min=41, max=9561, avg=321.94, stdev=714.94 clat (msec): min=83, max=1223, avg=598.51, stdev=162.31 lat (msec): min=84, max=1223, avg=598.83, stdev=162.29 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 351], 10.00th=[ 409], 20.00th=[ 464], | 30.00th=[ 510], 40.00th=[ 542], 50.00th=[ 584], 60.00th=[ 625], | 70.00th=[ 684], 80.00th=[ 726], 90.00th=[ 810], 95.00th=[ 894], | 99.00th=[ 1011], 99.50th=[ 1062], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 24, max= 127, per=24.70%, avg=71.38, stdev=22.17, samples=143 iops : min= 6, max= 31, avg=17.45, stdev= 5.56, samples=143 lat (msec) : 20=0.12%, 50=0.12%, 100=0.08%, 250=17.03%, 500=46.02% lat (msec) : 750=28.40%, 1000=7.66%, 2000=0.59% cpu : usr=0.12%, sys=0.88%, ctx=4938, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147937: Tue Jun 16 12:00:58 2026 read: IOPS=17, BW=68.0KiB/s (70.6kB/s)(4976KiB/72146msec) slat (usec): min=2206, max=40993, avg=10616.34, stdev=5968.13 clat (msec): min=20, max=630, avg=280.54, stdev=73.03 lat (msec): min=27, max=642, avg=291.16, stdev=74.08 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 180], 10.00th=[ 194], 20.00th=[ 222], | 30.00th=[ 241], 40.00th=[ 255], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 313], 80.00th=[ 338], 90.00th=[ 380], 95.00th=[ 418], | 99.00th=[ 464], 99.50th=[ 485], 99.90th=[ 592], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 16, max= 160, per=24.82%, avg=69.01, stdev=28.04, samples=142 iops : min= 4, max= 40, avg=16.85, stdev= 7.03, samples=142 write: IOPS=18, BW=72.0KiB/s (74.7kB/s)(5264KiB/72146msec); 0 zone resets slat (usec): min=44, max=9725, avg=297.26, stdev=613.34 clat (msec): min=144, max=1179, avg=600.18, stdev=160.41 lat (msec): min=144, max=1179, avg=600.48, stdev=160.39 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 355], 10.00th=[ 401], 20.00th=[ 464], | 30.00th=[ 506], 40.00th=[ 550], 50.00th=[ 600], 60.00th=[ 634], | 70.00th=[ 684], 80.00th=[ 726], 90.00th=[ 802], 95.00th=[ 894], | 99.00th=[ 995], 99.50th=[ 1020], 99.90th=[ 1099], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 8, max= 152, per=25.07%, avg=72.44, stdev=23.85, samples=143 iops : min= 2, max= 38, avg=17.72, stdev= 6.02, samples=143 lat (msec) : 50=0.20%, 100=0.12%, 250=17.70%, 500=45.12%, 750=28.55% lat (msec) : 1000=7.81%, 2000=0.51% cpu : usr=0.12%, sys=0.89%, ctx=4864, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147938: Tue Jun 16 12:00:58 2026 read: IOPS=17, BW=69.0KiB/s (70.7kB/s)(4984KiB/72179msec) slat (usec): min=2533, max=38523, avg=10482.81, stdev=6005.35 clat (msec): min=13, max=477, avg=283.70, stdev=72.11 lat (msec): min=18, max=490, avg=294.19, stdev=72.90 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 178], 10.00th=[ 201], 20.00th=[ 222], | 30.00th=[ 245], 40.00th=[ 264], 50.00th=[ 279], 60.00th=[ 296], | 70.00th=[ 317], 80.00th=[ 342], 90.00th=[ 380], 95.00th=[ 414], | 99.00th=[ 456], 99.50th=[ 464], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 16, max= 144, per=24.69%, avg=68.64, stdev=26.82, samples=143 iops : min= 4, max= 36, avg=16.77, stdev= 6.71, samples=143 write: IOPS=18, BW=72.8KiB/s (74.6kB/s)(5256KiB/72179msec); 0 zone resets slat (usec): min=41, max=15049, avg=317.71, stdev=710.83 clat (msec): min=171, max=1236, avg=598.03, stdev=162.94 lat (msec): min=171, max=1236, avg=598.35, stdev=162.98 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 355], 10.00th=[ 397], 20.00th=[ 460], | 30.00th=[ 502], 40.00th=[ 542], 50.00th=[ 592], 60.00th=[ 625], | 70.00th=[ 676], 80.00th=[ 726], 90.00th=[ 802], 95.00th=[ 902], | 99.00th=[ 1028], 99.50th=[ 1053], 99.90th=[ 1200], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 31, max= 134, per=24.87%, avg=71.88, stdev=23.22, samples=143 iops : min= 7, max= 33, avg=17.58, stdev= 5.84, samples=143 lat (msec) : 20=0.08%, 50=0.23%, 100=0.23%, 250=15.94%, 500=47.46% lat (msec) : 750=27.70%, 1000=7.66%, 2000=0.70% cpu : usr=0.13%, sys=0.88%, ctx=5065, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147939: Tue Jun 16 12:00:58 2026 read: IOPS=17, BW=70.2KiB/s (71.9kB/s)(5060KiB/72029msec) slat (usec): min=2345, max=42215, avg=10405.04, stdev=5659.82 clat (msec): min=11, max=675, avg=281.39, stdev=75.04 lat (msec): min=20, max=679, avg=291.80, stdev=75.90 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 180], 10.00th=[ 197], 20.00th=[ 222], | 30.00th=[ 241], 40.00th=[ 257], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 338], 90.00th=[ 376], 95.00th=[ 418], | 99.00th=[ 481], 99.50th=[ 506], 99.90th=[ 667], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 8, max= 143, per=25.06%, avg=69.68, stdev=26.86, samples=143 iops : min= 2, max= 35, avg=17.02, stdev= 6.74, samples=143 write: IOPS=17, BW=71.9KiB/s (73.6kB/s)(5180KiB/72029msec); 0 zone resets slat (usec): min=43, max=17980, avg=333.28, stdev=840.70 clat (msec): min=202, max=1231, avg=602.74, stdev=162.45 lat (msec): min=202, max=1232, avg=603.07, stdev=162.48 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 355], 10.00th=[ 401], 20.00th=[ 460], | 30.00th=[ 506], 40.00th=[ 550], 50.00th=[ 592], 60.00th=[ 634], | 70.00th=[ 684], 80.00th=[ 735], 90.00th=[ 810], 95.00th=[ 894], | 99.00th=[ 1020], 99.50th=[ 1062], 99.90th=[ 1150], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 16, max= 135, per=24.68%, avg=71.33, stdev=22.17, samples=143 iops : min= 4, max= 33, avg=17.44, stdev= 5.57, samples=143 lat (msec) : 20=0.08%, 50=0.16%, 100=0.12%, 250=17.34%, 500=46.05% lat (msec) : 750=26.84%, 1000=8.83%, 2000=0.59% cpu : usr=0.13%, sys=0.89%, ctx=4972, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=278KiB/s (285kB/s), 68.0KiB/s-70.2KiB/s (70.6kB/s-71.9kB/s), io=19.6MiB (20.5MB), run=71897-72179msec WRITE: bw=289KiB/s (296kB/s), 71.9KiB/s-72.0KiB/s (73.6kB/s-74.7kB/s), io=20.4MiB (21.4MB), run=71897-72179msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147964: Tue Jun 16 12:01:05 2026 read: IOPS=2, BW=119MiB/s (124MB/s)(40.0MiB/337msec) slat (nsec): min=18409k, max=18409k, avg=18408873.00, stdev= 0.00 clat (nsec): min=300950k, max=300950k, avg=300949724.00, stdev= 0.00 lat (nsec): min=319363k, max=319363k, avg=319363124.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.87%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=119MiB/s (124MB/s), 119MiB/s-119MiB/s (124MB/s-124MB/s), io=40.0MiB (41.9MB), run=337-337msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout