== sanity test 398c: run fio to test AIO ================= 04:39:39 (1788597579) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.413312 s, 101 MB/s osc.lustre-OST0000-osc-ffff9879a62cd000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145465: Sat Sep 5 04:43:00 2026 write: IOPS=13, BW=53.7KiB/s (54.0kB/s)(10.0MiB/190687msec); 0 zone resets slat (usec): min=22, max=24407, avg=239.80, stdev=1232.01 clat (msec): min=123, max=2688, avg=1190.44, stdev=402.10 lat (msec): min=124, max=2688, avg=1190.68, stdev=402.13 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 600], 10.00th=[ 735], 20.00th=[ 844], | 30.00th=[ 927], 40.00th=[ 1028], 50.00th=[ 1150], 60.00th=[ 1267], | 70.00th=[ 1401], 80.00th=[ 1536], 90.00th=[ 1754], 95.00th=[ 1888], | 99.00th=[ 2198], 99.50th=[ 2333], 99.90th=[ 2601], 99.95th=[ 2601], | 99.99th=[ 2702] bw ( KiB/s): min= 7, max= 144, per=25.77%, avg=55.15, stdev=29.31, samples=364 iops : min= 1, max= 36, avg=13.32, stdev= 7.35, samples=364 lat (msec) : 250=0.31%, 500=1.25%, 750=9.77%, 1000=26.29%, 2000=59.34% lat (msec) : >=2000=3.05% cpu : usr=0.04%, sys=0.20%, ctx=1806, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145466: Sat Sep 5 04:43:00 2026 write: IOPS=13, BW=53.7KiB/s (54.0kB/s)(10.0MiB/190729msec); 0 zone resets slat (usec): min=22, max=24853, avg=192.93, stdev=928.36 clat (msec): min=373, max=2655, avg=1190.58, stdev=396.55 lat (msec): min=373, max=2656, avg=1190.77, stdev=396.55 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 609], 10.00th=[ 709], 20.00th=[ 835], | 30.00th=[ 919], 40.00th=[ 1036], 50.00th=[ 1150], 60.00th=[ 1267], | 70.00th=[ 1385], 80.00th=[ 1552], 90.00th=[ 1754], 95.00th=[ 1905], | 99.00th=[ 2198], 99.50th=[ 2265], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 144, per=25.70%, avg=54.99, stdev=30.43, samples=365 iops : min= 1, max= 36, avg=13.28, stdev= 7.63, samples=365 lat (msec) : 500=1.72%, 750=10.59%, 1000=24.61%, 2000=60.47%, >=2000=2.62% cpu : usr=0.05%, sys=0.17%, ctx=1572, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145467: Sat Sep 5 04:43:00 2026 write: IOPS=13, BW=53.7KiB/s (54.0kB/s)(10.0MiB/190686msec); 0 zone resets slat (usec): min=22, max=11649, avg=156.73, stdev=538.97 clat (msec): min=184, max=2684, avg=1191.20, stdev=404.27 lat (msec): min=184, max=2684, avg=1191.36, stdev=404.27 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 625], 10.00th=[ 726], 20.00th=[ 827], | 30.00th=[ 927], 40.00th=[ 1028], 50.00th=[ 1150], 60.00th=[ 1267], | 70.00th=[ 1385], 80.00th=[ 1536], 90.00th=[ 1754], 95.00th=[ 1888], | 99.00th=[ 2232], 99.50th=[ 2400], 99.90th=[ 2635], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min= 7, max= 128, per=25.35%, avg=54.26, stdev=30.09, samples=370 iops : min= 1, max= 32, avg=13.09, stdev= 7.52, samples=370 lat (msec) : 250=0.62%, 500=0.86%, 750=10.82%, 1000=24.69%, 2000=60.16% lat (msec) : >=2000=2.85% cpu : usr=0.03%, sys=0.16%, ctx=1476, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145468: Sat Sep 5 04:43:00 2026 write: IOPS=13, BW=53.7KiB/s (54.0kB/s)(10.0MiB/190685msec); 0 zone resets slat (usec): min=21, max=17502, avg=191.57, stdev=865.89 clat (msec): min=331, max=2739, avg=1190.42, stdev=404.79 lat (msec): min=331, max=2739, avg=1190.61, stdev=404.82 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 600], 10.00th=[ 726], 20.00th=[ 835], | 30.00th=[ 927], 40.00th=[ 1036], 50.00th=[ 1150], 60.00th=[ 1267], | 70.00th=[ 1368], 80.00th=[ 1536], 90.00th=[ 1770], 95.00th=[ 1905], | 99.00th=[ 2232], 99.50th=[ 2400], 99.90th=[ 2534], 99.95th=[ 2735], | 99.99th=[ 2735] bw ( KiB/s): min= 7, max= 143, per=25.91%, avg=55.44, stdev=31.33, samples=362 iops : min= 1, max= 35, avg=13.39, stdev= 7.86, samples=362 lat (msec) : 500=1.64%, 750=10.35%, 1000=25.16%, 2000=59.69%, >=2000=3.16% cpu : usr=0.04%, sys=0.16%, ctx=1734, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=215KiB/s (220kB/s), 53.7KiB/s-53.7KiB/s (54.0kB/s-54.0kB/s), io=40.0MiB (41.9MB), run=190685-190729msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145504: Sat Sep 5 04:45:09 2026 read: IOPS=10, BW=42.2KiB/s (43.2kB/s)(5048KiB/119671msec) slat (usec): min=48, max=70411, avg=15399.14, stdev=10476.78 clat (msec): min=14, max=1060, avg=449.11, stdev=164.09 lat (msec): min=15, max=1100, avg=464.51, stdev=167.37 clat percentiles (msec): | 1.00th=[ 112], 5.00th=[ 218], 10.00th=[ 268], 20.00th=[ 321], | 30.00th=[ 359], 40.00th=[ 397], 50.00th=[ 430], 60.00th=[ 464], | 70.00th=[ 506], 80.00th=[ 575], 90.00th=[ 667], 95.00th=[ 768], | 99.00th=[ 919], 99.50th=[ 969], 99.90th=[ 1020], 99.95th=[ 1062], | 99.99th=[ 1062] bw ( KiB/s): min= 7, max= 120, per=25.33%, avg=42.31, stdev=24.47, samples=233 iops : min= 1, max= 30, avg=10.10, stdev= 6.19, samples=233 write: IOPS=10, BW=43.4KiB/s (44.4kB/s)(5192KiB/119671msec); 0 zone resets slat (usec): min=26, max=18879, avg=253.50, stdev=1074.52 clat (msec): min=198, max=2427, avg=1022.28, stdev=381.93 lat (msec): min=199, max=2427, avg=1022.54, stdev=381.96 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 477], 10.00th=[ 558], 20.00th=[ 676], | 30.00th=[ 768], 40.00th=[ 885], 50.00th=[ 986], 60.00th=[ 1083], | 70.00th=[ 1200], 80.00th=[ 1351], 90.00th=[ 1586], 95.00th=[ 1703], | 99.00th=[ 2039], 99.50th=[ 2123], 99.90th=[ 2232], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 128, per=24.66%, avg=42.91, stdev=20.89, samples=236 iops : min= 1, max= 32, avg=10.25, stdev= 5.27, samples=236 lat (msec) : 20=0.16%, 50=0.12%, 100=0.20%, 250=3.91%, 500=32.73% lat (msec) : 750=23.32%, 1000=15.12%, 2000=23.95%, >=2000=0.51% cpu : usr=0.09%, sys=0.50%, ctx=4335, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145505: Sat Sep 5 04:45:09 2026 read: IOPS=10, BW=41.5KiB/s (42.5kB/s)(4976KiB/119862msec) slat (usec): min=2205, max=65520, avg=15258.23, stdev=9968.50 clat (msec): min=20, max=1180, avg=449.95, stdev=159.67 lat (msec): min=38, max=1188, avg=465.22, stdev=162.54 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 218], 10.00th=[ 266], 20.00th=[ 317], | 30.00th=[ 359], 40.00th=[ 401], 50.00th=[ 435], 60.00th=[ 468], | 70.00th=[ 514], 80.00th=[ 575], 90.00th=[ 659], 95.00th=[ 760], | 99.00th=[ 869], 99.50th=[ 911], 99.90th=[ 1062], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 7, max= 128, per=24.86%, avg=41.51, stdev=23.68, samples=234 iops : min= 1, max= 32, avg= 9.89, stdev= 6.00, samples=234 write: IOPS=10, BW=43.9KiB/s (44.0kB/s)(5264KiB/119862msec); 0 zone resets slat (usec): min=23, max=11444, avg=190.89, stdev=582.42 clat (msec): min=268, max=2358, avg=1014.81, stdev=382.52 lat (msec): min=268, max=2358, avg=1015.00, stdev=382.54 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 489], 10.00th=[ 558], 20.00th=[ 667], | 30.00th=[ 768], 40.00th=[ 869], 50.00th=[ 978], 60.00th=[ 1062], | 70.00th=[ 1183], 80.00th=[ 1351], 90.00th=[ 1569], 95.00th=[ 1687], | 99.00th=[ 2140], 99.50th=[ 2165], 99.90th=[ 2299], 99.95th=[ 2366], | 99.99th=[ 2366] bw ( KiB/s): min= 7, max= 112, per=24.95%, avg=43.41, stdev=22.90, samples=237 iops : min= 1, max= 28, avg=10.37, stdev= 5.80, samples=237 lat (msec) : 50=0.16%, 100=0.12%, 250=3.75%, 500=31.91%, 750=24.69% lat (msec) : 1000=15.39%, 2000=23.24%, >=2000=0.74% cpu : usr=0.09%, sys=0.54%, ctx=4347, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145506: Sat Sep 5 04:45:09 2026 read: IOPS=10, BW=41.6KiB/s (42.6kB/s)(4984KiB/119922msec) slat (usec): min=2198, max=72843, avg=15381.62, stdev=10414.03 clat (msec): min=17, max=1088, avg=440.96, stdev=161.13 lat (msec): min=25, max=1112, avg=456.35, stdev=163.94 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 213], 10.00th=[ 255], 20.00th=[ 313], | 30.00th=[ 351], 40.00th=[ 393], 50.00th=[ 422], 60.00th=[ 456], | 70.00th=[ 493], 80.00th=[ 558], 90.00th=[ 667], 95.00th=[ 751], | 99.00th=[ 877], 99.50th=[ 969], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 160, per=25.25%, avg=42.16, stdev=25.49, samples=232 iops : min= 1, max= 40, avg=10.06, stdev= 6.46, samples=232 write: IOPS=10, BW=43.8KiB/s (44.9kB/s)(5256KiB/119922msec); 0 zone resets slat (usec): min=27, max=40882, avg=305.46, stdev=1608.09 clat (msec): min=223, max=2418, avg=1024.10, stdev=387.52 lat (msec): min=223, max=2418, avg=1024.40, stdev=387.60 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 493], 10.00th=[ 558], 20.00th=[ 667], | 30.00th=[ 768], 40.00th=[ 877], 50.00th=[ 1003], 60.00th=[ 1083], | 70.00th=[ 1200], 80.00th=[ 1351], 90.00th=[ 1569], 95.00th=[ 1720], | 99.00th=[ 2106], 99.50th=[ 2198], 99.90th=[ 2400], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 111, per=24.82%, avg=43.19, stdev=21.67, samples=237 iops : min= 1, max= 27, avg=10.31, stdev= 5.47, samples=237 lat (msec) : 20=0.08%, 50=0.16%, 100=0.08%, 250=3.98%, 500=33.28% lat (msec) : 750=23.44%, 1000=13.28%, 2000=25.00%, >=2000=0.70% cpu : usr=0.10%, sys=0.52%, ctx=4332, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145507: Sat Sep 5 04:45:09 2026 read: IOPS=10, BW=42.2KiB/s (43.2kB/s)(5060KiB/119954msec) slat (msec): min=2, max=107, avg=15.60, stdev=10.62 clat (msec): min=47, max=1014, avg=454.77, stdev=159.96 lat (msec): min=51, max=1073, avg=470.37, stdev=163.39 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 222], 10.00th=[ 271], 20.00th=[ 326], | 30.00th=[ 359], 40.00th=[ 397], 50.00th=[ 435], 60.00th=[ 472], | 70.00th=[ 514], 80.00th=[ 584], 90.00th=[ 684], 95.00th=[ 760], | 99.00th=[ 869], 99.50th=[ 969], 99.90th=[ 1003], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 7, max= 112, per=25.22%, avg=42.11, stdev=23.11, samples=235 iops : min= 1, max= 28, avg=10.05, stdev= 5.86, samples=235 write: IOPS=10, BW=43.2KiB/s (44.2kB/s)(5180KiB/119954msec); 0 zone resets slat (usec): min=23, max=37465, avg=261.39, stdev=1366.79 clat (msec): min=309, max=2425, avg=1018.68, stdev=387.50 lat (msec): min=309, max=2425, avg=1018.94, stdev=387.55 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 493], 10.00th=[ 550], 20.00th=[ 651], | 30.00th=[ 760], 40.00th=[ 877], 50.00th=[ 978], 60.00th=[ 1083], | 70.00th=[ 1200], 80.00th=[ 1368], 90.00th=[ 1586], 95.00th=[ 1687], | 99.00th=[ 2106], 99.50th=[ 2198], 99.90th=[ 2366], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 152, per=24.66%, avg=42.91, stdev=22.16, samples=236 iops : min= 1, max= 38, avg=10.24, stdev= 5.61, samples=236 lat (msec) : 50=0.04%, 100=0.16%, 250=3.48%, 500=32.38%, 750=25.47% lat (msec) : 1000=14.30%, 2000=23.44%, >=2000=0.74% cpu : usr=0.08%, sys=0.47%, ctx=4204, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=167KiB/s (171kB/s), 41.5KiB/s-42.2KiB/s (42.5kB/s-43.2kB/s), io=19.6MiB (20.5MB), run=119671-119954msec WRITE: bw=174KiB/s (178kB/s), 43.2KiB/s-43.9KiB/s (44.2kB/s-44.0kB/s), io=20.4MiB (21.4MB), run=119671-119954msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145528: Sat Sep 5 04:45:20 2026 read: IOPS=1, BW=58.4MiB/s (61.2MB/s)(40.0MiB/685msec) slat (nsec): min=27496k, max=27496k, avg=27496111.00, stdev= 0.00 clat (nsec): min=629907k, max=629907k, avg=629907129.00, stdev= 0.00 lat (nsec): min=657407k, max=657407k, avg=657406910.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 634], 5.00th=[ 634], 10.00th=[ 634], 20.00th=[ 634], | 30.00th=[ 634], 40.00th=[ 634], 50.00th=[ 634], 60.00th=[ 634], | 70.00th=[ 634], 80.00th=[ 634], 90.00th=[ 634], 95.00th=[ 634], | 99.00th=[ 634], 99.50th=[ 634], 99.90th=[ 634], 99.95th=[ 634], | 99.99th=[ 634] lat (msec) : 750=100.00% cpu : usr=0.15%, sys=1.90%, ctx=43, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=58.4MiB/s (61.2MB/s), 58.4MiB/s-58.4MiB/s (61.2MB/s-61.2MB/s), io=40.0MiB (41.9MB), run=685-685msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout