== sanity test 398c: run fio to test AIO ================= 18:40:23 (1763336423) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.37368 s, 112 MB/s osc.lustre-OST0000-osc-ffff8a48859df000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=150061: Sun Nov 16 18:42:48 2025 write: IOPS=18, BW=74.3KiB/s (76.1kB/s)(10.0MiB/137826msec); 0 zone resets slat (usec): min=27, max=4182, avg=144.69, stdev=301.83 clat (msec): min=193, max=1364, avg=860.20, stdev=192.28 lat (msec): min=193, max=1364, avg=860.35, stdev=192.29 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 523], 10.00th=[ 642], 20.00th=[ 718], | 30.00th=[ 776], 40.00th=[ 818], 50.00th=[ 869], 60.00th=[ 911], | 70.00th=[ 953], 80.00th=[ 1020], 90.00th=[ 1099], 95.00th=[ 1183], | 99.00th=[ 1284], 99.50th=[ 1301], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1368] bw ( KiB/s): min= 15, max= 254, per=24.86%, avg=73.84, stdev=30.21, samples=273 iops : min= 3, max= 63, avg=18.00, stdev= 7.60, samples=273 lat (msec) : 250=1.29%, 500=2.97%, 750=21.91%, 1000=51.33%, 2000=22.50% cpu : usr=0.05%, sys=0.24%, ctx=1561, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150062: Sun Nov 16 18:42:48 2025 write: IOPS=18, BW=74.3KiB/s (76.1kB/s)(10.0MiB/137753msec); 0 zone resets slat (usec): min=24, max=6500, avg=162.15, stdev=363.80 clat (msec): min=231, max=1595, avg=860.05, stdev=192.83 lat (msec): min=231, max=1595, avg=860.21, stdev=192.84 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 523], 10.00th=[ 642], 20.00th=[ 718], | 30.00th=[ 776], 40.00th=[ 818], 50.00th=[ 860], 60.00th=[ 902], | 70.00th=[ 961], 80.00th=[ 1020], 90.00th=[ 1099], 95.00th=[ 1150], | 99.00th=[ 1301], 99.50th=[ 1334], 99.90th=[ 1418], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 15, max= 254, per=24.86%, avg=73.85, stdev=27.83, samples=273 iops : min= 3, max= 63, avg=18.00, stdev= 6.99, samples=273 lat (msec) : 250=0.62%, 500=3.32%, 750=22.50%, 1000=49.57%, 2000=23.98% cpu : usr=0.04%, sys=0.26%, ctx=1770, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150063: Sun Nov 16 18:42:48 2025 write: IOPS=18, BW=74.4KiB/s (76.2kB/s)(10.0MiB/137592msec); 0 zone resets slat (usec): min=26, max=7700, avg=174.41, stdev=413.41 clat (msec): min=133, max=1590, avg=859.40, stdev=198.76 lat (msec): min=133, max=1590, avg=859.58, stdev=198.76 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 514], 10.00th=[ 634], 20.00th=[ 718], | 30.00th=[ 776], 40.00th=[ 818], 50.00th=[ 860], 60.00th=[ 902], | 70.00th=[ 961], 80.00th=[ 1028], 90.00th=[ 1099], 95.00th=[ 1167], | 99.00th=[ 1318], 99.50th=[ 1368], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1586] bw ( KiB/s): min= 15, max= 234, per=24.94%, avg=74.08, stdev=26.17, samples=272 iops : min= 3, max= 58, avg=18.06, stdev= 6.55, samples=272 lat (msec) : 250=0.90%, 500=3.67%, 750=21.33%, 1000=49.49%, 2000=24.61% cpu : usr=0.03%, sys=0.28%, ctx=1897, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=150064: Sun Nov 16 18:42:48 2025 write: IOPS=18, BW=74.3KiB/s (76.1kB/s)(10.0MiB/137779msec); 0 zone resets slat (usec): min=23, max=6653, avg=176.34, stdev=430.58 clat (msec): min=138, max=1722, avg=859.83, stdev=196.97 lat (msec): min=138, max=1722, avg=860.01, stdev=196.98 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 531], 10.00th=[ 634], 20.00th=[ 718], | 30.00th=[ 768], 40.00th=[ 818], 50.00th=[ 860], 60.00th=[ 911], | 70.00th=[ 961], 80.00th=[ 1020], 90.00th=[ 1099], 95.00th=[ 1167], | 99.00th=[ 1318], 99.50th=[ 1351], 99.90th=[ 1452], 99.95th=[ 1569], | 99.99th=[ 1720] bw ( KiB/s): min= 8, max= 191, per=24.95%, avg=74.11, stdev=29.09, samples=272 iops : min= 2, max= 47, avg=18.06, stdev= 7.29, samples=272 lat (msec) : 250=1.25%, 500=2.97%, 750=22.11%, 1000=50.27%, 2000=23.40% cpu : usr=0.04%, sys=0.26%, ctx=1742, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=297KiB/s (304kB/s), 74.3KiB/s-74.4KiB/s (76.1kB/s-76.2kB/s), io=40.0MiB (41.9MB), run=137592-137826msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=150093: Sun Nov 16 18:44:19 2025 read: IOPS=14, BW=59.2KiB/s (60.6kB/s)(5048KiB/85287msec) slat (msec): min=2, max=406, avg=10.48, stdev=12.11 clat (msec): min=24, max=806, avg=336.23, stdev=87.23 lat (msec): min=37, max=813, avg=346.72, stdev=88.15 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 211], 10.00th=[ 232], 20.00th=[ 264], | 30.00th=[ 288], 40.00th=[ 309], 50.00th=[ 334], 60.00th=[ 355], | 70.00th=[ 380], 80.00th=[ 409], 90.00th=[ 447], 95.00th=[ 477], | 99.00th=[ 567], 99.50th=[ 617], 99.90th=[ 743], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 7, max= 151, per=25.00%, avg=58.76, stdev=25.73, samples=169 iops : min= 1, max= 37, avg=14.21, stdev= 6.47, samples=169 write: IOPS=15, BW=60.9KiB/s (62.3kB/s)(5192KiB/85287msec); 0 zone resets slat (usec): min=29, max=11712, avg=225.43, stdev=622.10 clat (msec): min=168, max=1264, avg=712.11, stdev=187.40 lat (msec): min=168, max=1264, avg=712.34, stdev=187.40 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 405], 10.00th=[ 472], 20.00th=[ 550], | 30.00th=[ 617], 40.00th=[ 667], 50.00th=[ 709], 60.00th=[ 751], | 70.00th=[ 810], 80.00th=[ 877], 90.00th=[ 969], 95.00th=[ 1028], | 99.00th=[ 1116], 99.50th=[ 1133], 99.90th=[ 1217], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 7, max= 136, per=24.68%, avg=60.22, stdev=23.85, samples=169 iops : min= 1, max= 34, avg=14.59, stdev= 6.00, samples=169 lat (msec) : 50=0.04%, 100=0.31%, 250=7.46%, 500=46.80%, 750=24.84% lat (msec) : 1000=16.91%, 2000=3.63% cpu : usr=0.09%, sys=0.77%, ctx=4459, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150094: Sun Nov 16 18:44:19 2025 read: IOPS=14, BW=58.3KiB/s (59.7kB/s)(4976KiB/85288msec) slat (usec): min=2703, max=36156, avg=10166.23, stdev=4884.96 clat (msec): min=25, max=738, avg=336.98, stdev=84.38 lat (msec): min=35, max=743, avg=347.15, stdev=85.39 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 211], 10.00th=[ 236], 20.00th=[ 268], | 30.00th=[ 292], 40.00th=[ 313], 50.00th=[ 330], 60.00th=[ 355], | 70.00th=[ 384], 80.00th=[ 409], 90.00th=[ 447], 95.00th=[ 481], | 99.00th=[ 531], 99.50th=[ 550], 99.90th=[ 634], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 15, max= 135, per=24.64%, avg=57.91, stdev=24.19, samples=169 iops : min= 3, max= 33, avg=14.00, stdev= 6.07, samples=169 write: IOPS=15, BW=61.7KiB/s (63.2kB/s)(5264KiB/85288msec); 0 zone resets slat (usec): min=32, max=8352, avg=241.05, stdev=557.73 clat (msec): min=138, max=1219, avg=706.97, stdev=188.02 lat (msec): min=138, max=1220, avg=707.21, stdev=188.04 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 388], 10.00th=[ 464], 20.00th=[ 550], | 30.00th=[ 617], 40.00th=[ 659], 50.00th=[ 701], 60.00th=[ 751], | 70.00th=[ 810], 80.00th=[ 869], 90.00th=[ 961], 95.00th=[ 1020], | 99.00th=[ 1099], 99.50th=[ 1167], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 23, max= 136, per=25.03%, avg=61.08, stdev=22.10, samples=169 iops : min= 5, max= 34, avg=14.80, stdev= 5.57, samples=169 lat (msec) : 50=0.08%, 100=0.20%, 250=7.07%, 500=46.33%, 750=25.74% lat (msec) : 1000=17.23%, 2000=3.36% cpu : usr=0.09%, sys=0.74%, ctx=4220, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150095: Sun Nov 16 18:44:19 2025 read: IOPS=14, BW=58.4KiB/s (59.8kB/s)(4984KiB/85286msec) slat (usec): min=34, max=34526, avg=9981.36, stdev=4764.13 clat (msec): min=15, max=743, avg=334.93, stdev=85.09 lat (msec): min=24, max=752, avg=344.91, stdev=86.43 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 205], 10.00th=[ 230], 20.00th=[ 266], | 30.00th=[ 288], 40.00th=[ 309], 50.00th=[ 330], 60.00th=[ 351], | 70.00th=[ 380], 80.00th=[ 405], 90.00th=[ 447], 95.00th=[ 477], | 99.00th=[ 531], 99.50th=[ 558], 99.90th=[ 617], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 7, max= 127, per=24.73%, avg=58.12, stdev=25.51, samples=169 iops : min= 1, max= 31, avg=14.05, stdev= 6.46, samples=169 write: IOPS=15, BW=61.6KiB/s (63.1kB/s)(5256KiB/85286msec); 0 zone resets slat (usec): min=28, max=5448, avg=195.44, stdev=380.93 clat (msec): min=104, max=1260, avg=710.65, stdev=195.56 lat (msec): min=104, max=1260, avg=710.85, stdev=195.58 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 397], 10.00th=[ 468], 20.00th=[ 550], | 30.00th=[ 617], 40.00th=[ 659], 50.00th=[ 701], 60.00th=[ 751], | 70.00th=[ 810], 80.00th=[ 877], 90.00th=[ 969], 95.00th=[ 1053], | 99.00th=[ 1116], 99.50th=[ 1150], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 15, max= 136, per=24.91%, avg=60.78, stdev=22.02, samples=169 iops : min= 3, max= 34, avg=14.72, stdev= 5.55, samples=169 lat (msec) : 20=0.04%, 50=0.08%, 100=0.16%, 250=7.34%, 500=46.56% lat (msec) : 750=24.69%, 1000=16.99%, 2000=4.14% cpu : usr=0.11%, sys=0.68%, ctx=3983, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=150096: Sun Nov 16 18:44:19 2025 read: IOPS=14, BW=59.4KiB/s (60.8kB/s)(5060KiB/85218msec) slat (usec): min=2461, max=33751, avg=10177.94, stdev=4857.54 clat (msec): min=27, max=727, avg=335.15, stdev=85.44 lat (msec): min=37, max=733, avg=345.33, stdev=86.62 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 203], 10.00th=[ 228], 20.00th=[ 268], | 30.00th=[ 288], 40.00th=[ 313], 50.00th=[ 334], 60.00th=[ 351], | 70.00th=[ 376], 80.00th=[ 405], 90.00th=[ 451], 95.00th=[ 477], | 99.00th=[ 550], 99.50th=[ 567], 99.90th=[ 676], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 7, max= 152, per=25.09%, avg=58.96, stdev=25.27, samples=169 iops : min= 1, max= 38, avg=14.26, stdev= 6.38, samples=169 write: IOPS=15, BW=60.8KiB/s (62.2kB/s)(5180KiB/85218msec); 0 zone resets slat (usec): min=31, max=4647, avg=196.50, stdev=394.48 clat (msec): min=113, max=1214, avg=713.67, stdev=193.31 lat (msec): min=114, max=1214, avg=713.87, stdev=193.30 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 397], 10.00th=[ 460], 20.00th=[ 558], | 30.00th=[ 617], 40.00th=[ 659], 50.00th=[ 709], 60.00th=[ 751], | 70.00th=[ 810], 80.00th=[ 877], 90.00th=[ 986], 95.00th=[ 1053], | 99.00th=[ 1150], 99.50th=[ 1167], 99.90th=[ 1200], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 15, max= 120, per=24.59%, avg=60.01, stdev=22.54, samples=169 iops : min= 3, max= 30, avg=14.53, stdev= 5.64, samples=169 lat (msec) : 50=0.16%, 100=0.23%, 250=7.15%, 500=47.42%, 750=24.18% lat (msec) : 1000=16.72%, 2000=4.14% cpu : usr=0.12%, sys=0.73%, ctx=4366, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=235KiB/s (241kB/s), 58.3KiB/s-59.4KiB/s (59.7kB/s-60.8kB/s), io=19.6MiB (20.5MB), run=85218-85288msec WRITE: bw=245KiB/s (251kB/s), 60.8KiB/s-61.7KiB/s (62.2kB/s-63.2kB/s), io=20.4MiB (21.4MB), run=85218-85288msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=150112: Sun Nov 16 18:44:24 2025 read: IOPS=3, BW=129MiB/s (135MB/s)(40.0MiB/310msec) slat (nsec): min=24090k, max=24090k, avg=24090053.00, stdev= 0.00 clat (nsec): min=271869k, max=271869k, avg=271868935.00, stdev= 0.00 lat (nsec): min=295963k, max=295963k, avg=295963456.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.88%, ctx=22, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=40.0MiB (41.9MB), run=310-310msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout