== sanity test 398c: run fio to test AIO ================= 17:58:54 (1781301534) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.481027 s, 87.2 MB/s osc.lustre-OST0000-osc-ffff88ee4a0ea000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=125164: Fri Jun 12 18:01:23 2026 write: IOPS=18, BW=73.2KiB/s (74.0kB/s)(10.0MiB/139897msec); 0 zone resets slat (usec): min=19, max=17768, avg=257.01, stdev=1071.73 clat (msec): min=278, max=1985, avg=873.44, stdev=261.83 lat (msec): min=278, max=1985, avg=873.70, stdev=261.83 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 456], 10.00th=[ 584], 20.00th=[ 676], | 30.00th=[ 743], 40.00th=[ 793], 50.00th=[ 835], 60.00th=[ 902], | 70.00th=[ 969], 80.00th=[ 1070], 90.00th=[ 1217], 95.00th=[ 1334], | 99.00th=[ 1687], 99.50th=[ 1754], 99.90th=[ 1888], 99.95th=[ 1955], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 255, per=25.50%, avg=74.46, stdev=33.25, samples=271 iops : min= 1, max= 63, avg=18.17, stdev= 8.33, samples=271 lat (msec) : 500=6.17%, 750=25.47%, 1000=42.58%, 2000=25.78% cpu : usr=0.05%, sys=0.29%, ctx=1501, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=125165: Fri Jun 12 18:01:23 2026 write: IOPS=18, BW=73.2KiB/s (74.0kB/s)(10.0MiB/139828msec); 0 zone resets slat (usec): min=20, max=17582, avg=204.89, stdev=859.12 clat (msec): min=291, max=1907, avg=873.10, stdev=265.83 lat (msec): min=291, max=1907, avg=873.31, stdev=265.86 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 460], 10.00th=[ 584], 20.00th=[ 676], | 30.00th=[ 735], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 902], | 70.00th=[ 969], 80.00th=[ 1062], 90.00th=[ 1217], 95.00th=[ 1368], | 99.00th=[ 1670], 99.50th=[ 1703], 99.90th=[ 1888], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 256, per=25.79%, avg=75.30, stdev=33.81, samples=268 iops : min= 1, max= 64, avg=18.38, stdev= 8.52, samples=268 lat (msec) : 500=5.66%, 750=26.17%, 1000=40.51%, 2000=27.66% cpu : usr=0.06%, sys=0.26%, ctx=1540, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=125166: Fri Jun 12 18:01:23 2026 write: IOPS=18, BW=73.3KiB/s (75.0kB/s)(10.0MiB/139721msec); 0 zone resets slat (usec): min=20, max=35494, avg=255.97, stdev=1212.86 clat (msec): min=111, max=1870, avg=872.57, stdev=264.83 lat (msec): min=111, max=1870, avg=872.83, stdev=264.88 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 447], 10.00th=[ 592], 20.00th=[ 676], | 30.00th=[ 735], 40.00th=[ 802], 50.00th=[ 852], 60.00th=[ 902], | 70.00th=[ 978], 80.00th=[ 1053], 90.00th=[ 1217], 95.00th=[ 1334], | 99.00th=[ 1687], 99.50th=[ 1754], 99.90th=[ 1854], 99.95th=[ 1854], | 99.99th=[ 1871] bw ( KiB/s): min= 8, max= 245, per=25.40%, avg=74.16, stdev=32.49, samples=272 iops : min= 2, max= 61, avg=18.10, stdev= 8.15, samples=272 lat (msec) : 250=0.59%, 500=5.51%, 750=26.02%, 1000=40.70%, 2000=27.19% cpu : usr=0.04%, sys=0.28%, ctx=1581, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=125167: Fri Jun 12 18:01:23 2026 write: IOPS=18, BW=73.2KiB/s (74.0kB/s)(10.0MiB/139896msec); 0 zone resets slat (usec): min=19, max=19654, avg=216.30, stdev=872.60 clat (msec): min=283, max=1990, avg=873.48, stdev=264.38 lat (msec): min=284, max=1990, avg=873.70, stdev=264.36 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 481], 10.00th=[ 584], 20.00th=[ 676], | 30.00th=[ 743], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 902], | 70.00th=[ 961], 80.00th=[ 1053], 90.00th=[ 1217], 95.00th=[ 1385], | 99.00th=[ 1687], 99.50th=[ 1804], 99.90th=[ 1888], 99.95th=[ 1955], | 99.99th=[ 1989] bw ( KiB/s): min= 7, max= 255, per=25.66%, avg=74.93, stdev=35.92, samples=269 iops : min= 1, max= 63, avg=18.29, stdev= 9.03, samples=269 lat (msec) : 500=5.94%, 750=25.94%, 1000=42.11%, 2000=26.02% cpu : usr=0.05%, sys=0.26%, ctx=1537, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=293KiB/s (300kB/s), 73.2KiB/s-73.3KiB/s (74.0kB/s-75.0kB/s), io=40.0MiB (41.9MB), run=139721-139897msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=125210: Fri Jun 12 18:03:01 2026 read: IOPS=14, BW=58.6KiB/s (60.0kB/s)(5048KiB/86117msec) slat (usec): min=1498, max=68096, avg=11464.78, stdev=7396.10 clat (msec): min=30, max=735, avg=349.28, stdev=98.33 lat (msec): min=39, max=743, avg=360.75, stdev=99.87 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 211], 10.00th=[ 241], 20.00th=[ 275], | 30.00th=[ 300], 40.00th=[ 321], 50.00th=[ 342], 60.00th=[ 363], | 70.00th=[ 384], 80.00th=[ 409], 90.00th=[ 472], 95.00th=[ 535], | 99.00th=[ 676], 99.50th=[ 701], 99.90th=[ 726], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 5, max= 134, per=25.26%, avg=58.61, stdev=24.10, samples=169 iops : min= 1, max= 33, avg=14.18, stdev= 6.05, samples=169 write: IOPS=15, BW=60.3KiB/s (61.7kB/s)(5192KiB/86117msec); 0 zone resets slat (usec): min=22, max=25138, avg=287.55, stdev=1125.18 clat (msec): min=155, max=1332, avg=707.70, stdev=192.47 lat (msec): min=157, max=1332, avg=707.99, stdev=192.37 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 397], 10.00th=[ 456], 20.00th=[ 531], | 30.00th=[ 600], 40.00th=[ 651], 50.00th=[ 701], 60.00th=[ 760], | 70.00th=[ 810], 80.00th=[ 869], 90.00th=[ 961], 95.00th=[ 1028], | 99.00th=[ 1200], 99.50th=[ 1217], 99.90th=[ 1318], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 15, max= 119, per=24.97%, avg=60.19, stdev=22.23, samples=169 iops : min= 3, max= 29, avg=14.58, stdev= 5.56, samples=169 lat (msec) : 50=0.20%, 100=0.12%, 250=5.78%, 500=47.66%, 750=25.23% lat (msec) : 1000=17.54%, 2000=3.48% cpu : usr=0.11%, sys=0.83%, ctx=3350, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=125211: Fri Jun 12 18:03:01 2026 read: IOPS=14, BW=57.7KiB/s (59.1kB/s)(4976KiB/86284msec) slat (usec): min=34, max=50064, avg=11175.78, stdev=7408.01 clat (msec): min=13, max=737, avg=352.39, stdev=98.52 lat (msec): min=26, max=758, avg=363.57, stdev=99.83 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 213], 10.00th=[ 243], 20.00th=[ 275], | 30.00th=[ 300], 40.00th=[ 321], 50.00th=[ 347], 60.00th=[ 363], | 70.00th=[ 388], 80.00th=[ 418], 90.00th=[ 468], 95.00th=[ 542], | 99.00th=[ 659], 99.50th=[ 684], 99.90th=[ 701], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 8, max= 160, per=24.62%, avg=57.11, stdev=24.97, samples=171 iops : min= 2, max= 40, avg=13.81, stdev= 6.28, samples=171 write: IOPS=15, BW=61.0KiB/s (62.5kB/s)(5264KiB/86284msec); 0 zone resets slat (usec): min=22, max=23981, avg=310.23, stdev=1188.72 clat (msec): min=207, max=1301, avg=704.39, stdev=187.68 lat (msec): min=207, max=1301, avg=704.71, stdev=187.72 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 405], 10.00th=[ 460], 20.00th=[ 542], | 30.00th=[ 600], 40.00th=[ 651], 50.00th=[ 701], 60.00th=[ 751], | 70.00th=[ 802], 80.00th=[ 877], 90.00th=[ 953], 95.00th=[ 1020], | 99.00th=[ 1167], 99.50th=[ 1200], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 8, max= 135, per=25.17%, avg=60.67, stdev=21.14, samples=170 iops : min= 2, max= 33, avg=14.71, stdev= 5.25, samples=170 lat (msec) : 20=0.04%, 50=0.08%, 100=0.04%, 250=6.09%, 500=46.88% lat (msec) : 750=26.48%, 1000=17.19%, 2000=3.20% cpu : usr=0.11%, sys=0.79%, ctx=3220, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=125212: Fri Jun 12 18:03:01 2026 read: IOPS=14, BW=57.7KiB/s (59.1kB/s)(4984KiB/86365msec) slat (usec): min=1289, max=54089, avg=11367.38, stdev=7348.42 clat (msec): min=4, max=733, avg=350.52, stdev=95.54 lat (msec): min=9, max=754, avg=361.89, stdev=97.32 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 211], 10.00th=[ 236], 20.00th=[ 275], | 30.00th=[ 300], 40.00th=[ 326], 50.00th=[ 342], 60.00th=[ 368], | 70.00th=[ 388], 80.00th=[ 418], 90.00th=[ 464], 95.00th=[ 523], | 99.00th=[ 659], 99.50th=[ 684], 99.90th=[ 726], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 8, max= 135, per=24.73%, avg=57.38, stdev=24.81, samples=170 iops : min= 2, max= 33, avg=13.90, stdev= 6.16, samples=170 write: IOPS=15, BW=60.9KiB/s (62.3kB/s)(5256KiB/86365msec); 0 zone resets slat (usec): min=20, max=17834, avg=299.84, stdev=956.66 clat (msec): min=145, max=1351, avg=705.16, stdev=190.16 lat (msec): min=145, max=1351, avg=705.46, stdev=190.18 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 397], 10.00th=[ 456], 20.00th=[ 535], | 30.00th=[ 609], 40.00th=[ 659], 50.00th=[ 709], 60.00th=[ 751], | 70.00th=[ 802], 80.00th=[ 877], 90.00th=[ 944], 95.00th=[ 1011], | 99.00th=[ 1167], 99.50th=[ 1217], 99.90th=[ 1301], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 15, max= 120, per=24.92%, avg=60.06, stdev=22.23, samples=170 iops : min= 3, max= 30, avg=14.56, stdev= 5.57, samples=170 lat (msec) : 10=0.04%, 100=0.16%, 250=6.17%, 500=46.33%, 750=26.52% lat (msec) : 1000=17.89%, 2000=2.89% cpu : usr=0.11%, sys=0.78%, ctx=3320, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=125213: Fri Jun 12 18:03:01 2026 read: IOPS=14, BW=58.6KiB/s (60.1kB/s)(5060KiB/86280msec) slat (usec): min=1746, max=55474, avg=11317.34, stdev=7220.31 clat (msec): min=39, max=736, avg=351.80, stdev=98.76 lat (msec): min=45, max=787, avg=363.13, stdev=100.28 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 211], 10.00th=[ 236], 20.00th=[ 275], | 30.00th=[ 296], 40.00th=[ 321], 50.00th=[ 342], 60.00th=[ 363], | 70.00th=[ 388], 80.00th=[ 418], 90.00th=[ 472], 95.00th=[ 542], | 99.00th=[ 693], 99.50th=[ 709], 99.90th=[ 726], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 7, max= 143, per=25.43%, avg=59.01, stdev=26.01, samples=169 iops : min= 1, max= 35, avg=14.28, stdev= 6.47, samples=169 write: IOPS=15, BW=60.0KiB/s (61.5kB/s)(5180KiB/86280msec); 0 zone resets slat (usec): min=27, max=15396, avg=311.00, stdev=1113.87 clat (msec): min=84, max=1337, avg=708.65, stdev=188.79 lat (msec): min=84, max=1337, avg=708.97, stdev=188.77 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 418], 10.00th=[ 464], 20.00th=[ 531], | 30.00th=[ 592], 40.00th=[ 651], 50.00th=[ 709], 60.00th=[ 760], | 70.00th=[ 818], 80.00th=[ 885], 90.00th=[ 953], 95.00th=[ 1020], | 99.00th=[ 1183], 99.50th=[ 1234], 99.90th=[ 1301], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 8, max= 118, per=24.61%, avg=59.30, stdev=22.30, samples=171 iops : min= 2, max= 29, avg=14.36, stdev= 5.57, samples=171 lat (msec) : 50=0.04%, 100=0.12%, 250=6.48%, 500=46.80%, 750=25.47% lat (msec) : 1000=17.89%, 2000=3.20% cpu : usr=0.14%, sys=0.86%, ctx=3388, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=232KiB/s (238kB/s), 57.7KiB/s-58.6KiB/s (59.1kB/s-60.1kB/s), io=19.6MiB (20.5MB), run=86117-86365msec WRITE: bw=242KiB/s (248kB/s), 60.0KiB/s-61.0KiB/s (61.5kB/s-62.5kB/s), io=20.4MiB (21.4MB), run=86117-86365msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=125230: Fri Jun 12 18:03:08 2026 read: IOPS=1, BW=74.6MiB/s (78.3MB/s)(40.0MiB/536msec) slat (nsec): min=68400k, max=68400k, avg=68400392.00, stdev= 0.00 clat (nsec): min=456622k, max=456622k, avg=456621610.00, stdev= 0.00 lat (nsec): min=525027k, max=525027k, avg=525027164.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 456], 10.00th=[ 456], 20.00th=[ 456], | 30.00th=[ 456], 40.00th=[ 456], 50.00th=[ 456], 60.00th=[ 456], | 70.00th=[ 456], 80.00th=[ 456], 90.00th=[ 456], 95.00th=[ 456], | 99.00th=[ 456], 99.50th=[ 456], 99.90th=[ 456], 99.95th=[ 456], | 99.99th=[ 456] lat (msec) : 500=100.00% cpu : usr=0.19%, sys=3.18%, ctx=3, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=74.6MiB/s (78.3MB/s), 74.6MiB/s-74.6MiB/s (78.3MB/s-78.3MB/s), io=40.0MiB (41.9MB), run=536-536msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck