== sanity test 398c: run fio to test AIO ================= 21:01:43 (1788656503) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.608881 s, 68.9 MB/s osc.lustre-OST0000-osc-ffff898d00a94000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145684: Sat Sep 5 21:04:46 2026 write: IOPS=14, BW=59.5KiB/s (60.0kB/s)(10.0MiB/171963msec); 0 zone resets slat (usec): min=28, max=19361, avg=242.27, stdev=840.96 clat (msec): min=179, max=2450, avg=1073.66, stdev=324.94 lat (msec): min=179, max=2450, avg=1073.91, stdev=324.95 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 625], 10.00th=[ 726], 20.00th=[ 827], | 30.00th=[ 911], 40.00th=[ 986], 50.00th=[ 1045], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1301], 90.00th=[ 1452], 95.00th=[ 1670], | 99.00th=[ 2005], 99.50th=[ 2123], 99.90th=[ 2400], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 255, per=25.36%, avg=60.35, stdev=33.89, samples=333 iops : min= 1, max= 63, avg=14.55, stdev= 8.48, samples=333 lat (msec) : 250=1.25%, 500=2.03%, 750=9.41%, 1000=29.53%, 2000=56.84% lat (msec) : >=2000=0.94% cpu : usr=0.04%, sys=0.23%, ctx=1542, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145685: Sat Sep 5 21:04:46 2026 write: IOPS=14, BW=59.6KiB/s (61.0kB/s)(10.0MiB/171850msec); 0 zone resets slat (usec): min=28, max=22090, avg=258.59, stdev=974.22 clat (msec): min=153, max=2503, avg=1072.86, stdev=329.35 lat (msec): min=161, max=2503, avg=1073.13, stdev=329.38 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 634], 10.00th=[ 735], 20.00th=[ 827], | 30.00th=[ 894], 40.00th=[ 969], 50.00th=[ 1053], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1318], 90.00th=[ 1452], 95.00th=[ 1620], | 99.00th=[ 2089], 99.50th=[ 2232], 99.90th=[ 2433], 99.95th=[ 2467], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 192, per=25.73%, avg=61.25, stdev=33.45, samples=328 iops : min= 1, max= 48, avg=14.77, stdev= 8.39, samples=328 lat (msec) : 250=1.25%, 500=2.07%, 750=8.24%, 1000=32.38%, 2000=54.92% lat (msec) : >=2000=1.13% cpu : usr=0.05%, sys=0.23%, ctx=1812, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145686: Sat Sep 5 21:04:46 2026 write: IOPS=14, BW=59.7KiB/s (61.1kB/s)(10.0MiB/171610msec); 0 zone resets slat (usec): min=28, max=18110, avg=299.41, stdev=1071.99 clat (msec): min=111, max=2477, avg=1071.83, stdev=325.54 lat (msec): min=114, max=2479, avg=1072.13, stdev=325.52 clat percentiles (msec): | 1.00th=[ 259], 5.00th=[ 625], 10.00th=[ 726], 20.00th=[ 818], | 30.00th=[ 911], 40.00th=[ 986], 50.00th=[ 1053], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1301], 90.00th=[ 1469], 95.00th=[ 1670], | 99.00th=[ 2022], 99.50th=[ 2089], 99.90th=[ 2366], 99.95th=[ 2433], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 215, per=25.05%, avg=59.61, stdev=29.77, samples=337 iops : min= 1, max= 53, avg=14.36, stdev= 7.44, samples=337 lat (msec) : 250=0.90%, 500=2.73%, 750=8.83%, 1000=29.61%, 2000=56.84% lat (msec) : >=2000=1.09% cpu : usr=0.05%, sys=0.27%, ctx=2044, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145687: Sat Sep 5 21:04:46 2026 write: IOPS=14, BW=59.7KiB/s (61.1kB/s)(10.0MiB/171665msec); 0 zone resets slat (usec): min=29, max=18926, avg=266.37, stdev=927.09 clat (msec): min=160, max=2616, avg=1071.74, stdev=334.21 lat (msec): min=160, max=2616, avg=1072.00, stdev=334.22 clat percentiles (msec): | 1.00th=[ 247], 5.00th=[ 617], 10.00th=[ 718], 20.00th=[ 827], | 30.00th=[ 902], 40.00th=[ 969], 50.00th=[ 1036], 60.00th=[ 1116], | 70.00th=[ 1183], 80.00th=[ 1301], 90.00th=[ 1469], 95.00th=[ 1687], | 99.00th=[ 2089], 99.50th=[ 2198], 99.90th=[ 2534], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 191, per=25.43%, avg=60.52, stdev=30.83, samples=332 iops : min= 1, max= 47, avg=14.59, stdev= 7.72, samples=332 lat (msec) : 250=1.25%, 500=2.19%, 750=8.67%, 1000=32.46%, 2000=54.10% lat (msec) : >=2000=1.33% cpu : usr=0.05%, sys=0.26%, ctx=1818, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=238KiB/s (244kB/s), 59.5KiB/s-59.7KiB/s (60.0kB/s-61.1kB/s), io=40.0MiB (41.9MB), run=171610-171963msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145722: Sat Sep 5 21:06:41 2026 read: IOPS=12, BW=48.3KiB/s (49.5kB/s)(5048KiB/104437msec) slat (usec): min=42, max=61138, avg=14734.52, stdev=7653.20 clat (msec): min=24, max=836, avg=402.99, stdev=102.42 lat (msec): min=40, max=860, avg=417.74, stdev=103.55 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 264], 10.00th=[ 292], 20.00th=[ 326], | 30.00th=[ 355], 40.00th=[ 376], 50.00th=[ 397], 60.00th=[ 414], | 70.00th=[ 439], 80.00th=[ 472], 90.00th=[ 531], 95.00th=[ 584], | 99.00th=[ 743], 99.50th=[ 776], 99.90th=[ 818], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 7, max= 119, per=25.13%, avg=48.00, stdev=22.20, samples=206 iops : min= 1, max= 29, avg=11.45, stdev= 5.59, samples=206 write: IOPS=12, BW=49.7KiB/s (50.9kB/s)(5192KiB/104437msec); 0 zone resets slat (usec): min=36, max=17550, avg=409.54, stdev=1147.54 clat (msec): min=107, max=1693, avg=879.82, stdev=232.17 lat (msec): min=107, max=1693, avg=880.23, stdev=232.16 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 510], 10.00th=[ 600], 20.00th=[ 693], | 30.00th=[ 751], 40.00th=[ 818], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 1003], 80.00th=[ 1070], 90.00th=[ 1150], 95.00th=[ 1267], | 99.00th=[ 1519], 99.50th=[ 1552], 99.90th=[ 1687], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 104, per=24.65%, avg=49.05, stdev=19.39, samples=207 iops : min= 1, max= 26, avg=11.71, stdev= 4.85, samples=207 lat (msec) : 50=0.27%, 100=0.12%, 250=1.33%, 500=42.85%, 750=19.26% lat (msec) : 1000=20.90%, 2000=15.27% cpu : usr=0.13%, sys=0.83%, ctx=4804, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145723: Sat Sep 5 21:06:41 2026 read: IOPS=11, BW=47.5KiB/s (48.7kB/s)(4976KiB/104703msec) slat (usec): min=3430, max=54734, avg=15237.42, stdev=7561.65 clat (msec): min=14, max=838, avg=402.36, stdev=97.69 lat (msec): min=25, max=858, avg=417.61, stdev=98.40 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 266], 10.00th=[ 296], 20.00th=[ 326], | 30.00th=[ 347], 40.00th=[ 372], 50.00th=[ 397], 60.00th=[ 418], | 70.00th=[ 443], 80.00th=[ 468], 90.00th=[ 523], 95.00th=[ 575], | 99.00th=[ 676], 99.50th=[ 701], 99.90th=[ 802], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 7, max= 111, per=25.00%, avg=47.75, stdev=20.53, samples=204 iops : min= 1, max= 27, avg=11.38, stdev= 5.14, samples=204 write: IOPS=12, BW=50.3KiB/s (51.5kB/s)(5264KiB/104703msec); 0 zone resets slat (usec): min=30, max=30282, avg=471.26, stdev=1642.23 clat (msec): min=216, max=1669, avg=875.28, stdev=227.41 lat (msec): min=216, max=1669, avg=875.76, stdev=227.40 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 514], 10.00th=[ 584], 20.00th=[ 684], | 30.00th=[ 751], 40.00th=[ 810], 50.00th=[ 869], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1062], 90.00th=[ 1150], 95.00th=[ 1234], | 99.00th=[ 1536], 99.50th=[ 1620], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 119, per=24.89%, avg=49.54, stdev=19.23, samples=208 iops : min= 1, max= 29, avg=11.83, stdev= 4.80, samples=208 lat (msec) : 20=0.04%, 50=0.20%, 100=0.12%, 250=1.09%, 500=42.81% lat (msec) : 750=19.88%, 1000=21.25%, 2000=14.61% cpu : usr=0.12%, sys=0.82%, ctx=4759, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145724: Sat Sep 5 21:06:41 2026 read: IOPS=11, BW=47.6KiB/s (48.7kB/s)(4984KiB/104764msec) slat (usec): min=3157, max=52280, avg=14610.51, stdev=7371.93 clat (msec): min=15, max=833, avg=404.43, stdev=97.23 lat (msec): min=39, max=852, avg=419.06, stdev=97.89 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 275], 10.00th=[ 296], 20.00th=[ 326], | 30.00th=[ 359], 40.00th=[ 380], 50.00th=[ 397], 60.00th=[ 418], | 70.00th=[ 439], 80.00th=[ 468], 90.00th=[ 531], 95.00th=[ 584], | 99.00th=[ 693], 99.50th=[ 743], 99.90th=[ 827], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 7, max= 112, per=24.75%, avg=47.27, stdev=21.50, samples=207 iops : min= 1, max= 28, avg=11.27, stdev= 5.36, samples=207 write: IOPS=12, BW=50.2KiB/s (51.4kB/s)(5256KiB/104764msec); 0 zone resets slat (usec): min=33, max=28508, avg=415.85, stdev=1394.64 clat (msec): min=234, max=1617, avg=875.70, stdev=224.28 lat (msec): min=234, max=1618, avg=876.12, stdev=224.30 clat percentiles (msec): | 1.00th=[ 439], 5.00th=[ 527], 10.00th=[ 592], 20.00th=[ 684], | 30.00th=[ 760], 40.00th=[ 818], 50.00th=[ 869], 60.00th=[ 927], | 70.00th=[ 986], 80.00th=[ 1053], 90.00th=[ 1150], 95.00th=[ 1234], | 99.00th=[ 1519], 99.50th=[ 1569], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 7, max= 103, per=24.90%, avg=49.55, stdev=19.71, samples=207 iops : min= 1, max= 25, avg=11.84, stdev= 4.95, samples=207 lat (msec) : 20=0.04%, 50=0.08%, 100=0.12%, 250=0.98%, 500=42.30% lat (msec) : 750=19.73%, 1000=22.62%, 2000=14.14% cpu : usr=0.14%, sys=0.88%, ctx=5138, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145725: Sat Sep 5 21:06:41 2026 read: IOPS=12, BW=48.3KiB/s (49.4kB/s)(5060KiB/104813msec) slat (usec): min=3453, max=50323, avg=14675.90, stdev=7431.57 clat (msec): min=30, max=827, avg=405.75, stdev=96.07 lat (msec): min=50, max=855, avg=420.44, stdev=96.60 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 271], 10.00th=[ 300], 20.00th=[ 330], | 30.00th=[ 351], 40.00th=[ 376], 50.00th=[ 397], 60.00th=[ 418], | 70.00th=[ 443], 80.00th=[ 472], 90.00th=[ 535], 95.00th=[ 584], | 99.00th=[ 684], 99.50th=[ 743], 99.90th=[ 776], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 7, max= 111, per=25.18%, avg=48.09, stdev=21.40, samples=206 iops : min= 1, max= 27, avg=11.47, stdev= 5.37, samples=206 write: IOPS=12, BW=49.4KiB/s (50.6kB/s)(5180KiB/104813msec); 0 zone resets slat (usec): min=35, max=21948, avg=389.14, stdev=1240.42 clat (msec): min=243, max=1677, avg=881.46, stdev=228.46 lat (msec): min=244, max=1677, avg=881.86, stdev=228.44 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 527], 10.00th=[ 592], 20.00th=[ 684], | 30.00th=[ 760], 40.00th=[ 818], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1053], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1536], 99.50th=[ 1586], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 111, per=24.61%, avg=48.98, stdev=19.76, samples=207 iops : min= 1, max= 27, avg=11.69, stdev= 4.92, samples=207 lat (msec) : 50=0.04%, 100=0.08%, 250=1.05%, 500=42.89%, 750=19.41% lat (msec) : 1000=22.50%, 2000=14.02% cpu : usr=0.13%, sys=0.79%, ctx=5018, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=191KiB/s (196kB/s), 47.5KiB/s-48.3KiB/s (48.7kB/s-49.5kB/s), io=19.6MiB (20.5MB), run=104437-104813msec WRITE: bw=199KiB/s (204kB/s), 49.4KiB/s-50.3KiB/s (50.6kB/s-51.5kB/s), io=20.4MiB (21.4MB), run=104437-104813msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145743: Sat Sep 5 21:06:51 2026 read: IOPS=1, BW=77.4MiB/s (81.1MB/s)(40.0MiB/517msec) slat (nsec): min=41942k, max=41942k, avg=41941635.00, stdev= 0.00 clat (nsec): min=434410k, max=434410k, avg=434409718.00, stdev= 0.00 lat (nsec): min=476356k, max=476356k, avg=476356014.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 435], 10.00th=[ 435], 20.00th=[ 435], | 30.00th=[ 435], 40.00th=[ 435], 50.00th=[ 435], 60.00th=[ 435], | 70.00th=[ 435], 80.00th=[ 435], 90.00th=[ 435], 95.00th=[ 435], | 99.00th=[ 435], 99.50th=[ 435], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.52%, ctx=18, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=77.4MiB/s (81.1MB/s), 77.4MiB/s-77.4MiB/s (81.1MB/s-81.1MB/s), io=40.0MiB (41.9MB), run=517-517msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout