== sanity test 398c: run fio to test AIO ================= 18:10:37 (1779315037) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.363563 s, 115 MB/s osc.lustre-OST0000-osc-ffff9145ae70c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180430: Wed May 20 18:13:09 2026 write: IOPS=17, BW=71.7KiB/s (73.4kB/s)(10.0MiB/142780msec); 0 zone resets slat (usec): min=42, max=12369, avg=273.10, stdev=682.88 clat (msec): min=260, max=1496, avg=891.32, stdev=205.05 lat (msec): min=260, max=1496, avg=891.60, stdev=205.07 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 531], 10.00th=[ 651], 20.00th=[ 743], | 30.00th=[ 793], 40.00th=[ 852], 50.00th=[ 902], 60.00th=[ 944], | 70.00th=[ 995], 80.00th=[ 1053], 90.00th=[ 1150], 95.00th=[ 1217], | 99.00th=[ 1318], 99.50th=[ 1401], 99.90th=[ 1469], 99.95th=[ 1502], | 99.99th=[ 1502] bw ( KiB/s): min= 7, max= 254, per=24.82%, avg=70.99, stdev=28.96, samples=284 iops : min= 1, max= 63, avg=17.30, stdev= 7.25, samples=284 lat (msec) : 500=4.57%, 750=17.19%, 1000=48.52%, 2000=29.73% cpu : usr=0.05%, sys=0.29%, ctx=2281, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180431: Wed May 20 18:13:09 2026 write: IOPS=17, BW=71.9KiB/s (73.6kB/s)(10.0MiB/142448msec); 0 zone resets slat (usec): min=49, max=15730, avg=270.18, stdev=731.07 clat (msec): min=208, max=1636, avg=889.53, stdev=209.54 lat (msec): min=208, max=1637, avg=889.80, stdev=209.54 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 527], 10.00th=[ 651], 20.00th=[ 743], | 30.00th=[ 802], 40.00th=[ 852], 50.00th=[ 894], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1062], 90.00th=[ 1150], 95.00th=[ 1234], | 99.00th=[ 1351], 99.50th=[ 1401], 99.90th=[ 1469], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 15, max= 239, per=25.08%, avg=71.72, stdev=28.60, samples=281 iops : min= 3, max= 59, avg=17.48, stdev= 7.17, samples=281 lat (msec) : 250=0.62%, 500=4.14%, 750=16.72%, 1000=49.57%, 2000=28.95% cpu : usr=0.04%, sys=0.30%, ctx=2135, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180432: Wed May 20 18:13:09 2026 write: IOPS=17, BW=71.8KiB/s (73.5kB/s)(10.0MiB/142637msec); 0 zone resets slat (usec): min=42, max=14449, avg=307.38, stdev=893.35 clat (msec): min=102, max=1527, avg=890.67, stdev=206.50 lat (msec): min=103, max=1527, avg=890.98, stdev=206.46 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 518], 10.00th=[ 651], 20.00th=[ 743], | 30.00th=[ 802], 40.00th=[ 852], 50.00th=[ 902], 60.00th=[ 944], | 70.00th=[ 995], 80.00th=[ 1053], 90.00th=[ 1133], 95.00th=[ 1234], | 99.00th=[ 1334], 99.50th=[ 1385], 99.90th=[ 1469], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 231, per=24.91%, avg=71.24, stdev=30.92, samples=283 iops : min= 1, max= 57, avg=17.37, stdev= 7.73, samples=283 lat (msec) : 250=0.35%, 500=4.45%, 750=16.17%, 1000=50.12%, 2000=28.91% cpu : usr=0.04%, sys=0.30%, ctx=2160, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180433: Wed May 20 18:13:09 2026 write: IOPS=17, BW=71.7KiB/s (73.4kB/s)(10.0MiB/142780msec); 0 zone resets slat (usec): min=40, max=14965, avg=308.77, stdev=838.47 clat (msec): min=259, max=1461, avg=891.24, stdev=202.37 lat (msec): min=259, max=1461, avg=891.55, stdev=202.37 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 558], 10.00th=[ 659], 20.00th=[ 743], | 30.00th=[ 793], 40.00th=[ 852], 50.00th=[ 894], 60.00th=[ 944], | 70.00th=[ 995], 80.00th=[ 1062], 90.00th=[ 1150], 95.00th=[ 1217], | 99.00th=[ 1334], 99.50th=[ 1385], 99.90th=[ 1452], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 8, max= 254, per=25.27%, avg=72.28, stdev=29.79, samples=279 iops : min= 2, max= 63, avg=17.63, stdev= 7.45, samples=279 lat (msec) : 500=4.26%, 750=18.16%, 1000=48.87%, 2000=28.71% cpu : usr=0.04%, sys=0.32%, ctx=2494, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=287KiB/s (294kB/s), 71.7KiB/s-71.9KiB/s (73.4kB/s-73.6kB/s), io=40.0MiB (41.9MB), run=142448-142780msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180465: Wed May 20 18:14:44 2026 read: IOPS=14, BW=57.1KiB/s (58.5kB/s)(5048KiB/88339msec) slat (usec): min=2479, max=37714, avg=11222.95, stdev=5768.55 clat (msec): min=14, max=817, avg=347.34, stdev=86.78 lat (msec): min=23, max=826, avg=358.56, stdev=87.82 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 239], 10.00th=[ 255], 20.00th=[ 279], | 30.00th=[ 296], 40.00th=[ 317], 50.00th=[ 338], 60.00th=[ 359], | 70.00th=[ 384], 80.00th=[ 414], 90.00th=[ 456], 95.00th=[ 493], | 99.00th=[ 592], 99.50th=[ 667], 99.90th=[ 684], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 15, max= 128, per=25.12%, avg=56.76, stdev=22.28, samples=175 iops : min= 3, max= 32, avg=13.78, stdev= 5.55, samples=175 write: IOPS=14, BW=58.8KiB/s (60.2kB/s)(5192KiB/88339msec); 0 zone resets slat (usec): min=53, max=23830, avg=356.70, stdev=966.01 clat (msec): min=96, max=1426, avg=738.22, stdev=192.31 lat (msec): min=96, max=1426, avg=738.58, stdev=192.28 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 460], 10.00th=[ 510], 20.00th=[ 567], | 30.00th=[ 634], 40.00th=[ 684], 50.00th=[ 726], 60.00th=[ 768], | 70.00th=[ 827], 80.00th=[ 885], 90.00th=[ 1003], 95.00th=[ 1070], | 99.00th=[ 1250], 99.50th=[ 1284], 99.90th=[ 1368], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 119, per=24.70%, avg=58.29, stdev=21.23, samples=175 iops : min= 1, max= 29, avg=14.16, stdev= 5.29, samples=175 lat (msec) : 20=0.08%, 50=0.12%, 100=0.20%, 250=3.95%, 500=46.99% lat (msec) : 750=26.05%, 1000=17.50%, 2000=5.12% cpu : usr=0.09%, sys=0.68%, ctx=4847, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180466: Wed May 20 18:14:44 2026 read: IOPS=14, BW=56.3KiB/s (57.6kB/s)(4976KiB/88445msec) slat (usec): min=2593, max=54698, avg=11030.64, stdev=5860.99 clat (msec): min=31, max=677, avg=344.81, stdev=84.50 lat (msec): min=41, max=693, avg=355.85, stdev=85.62 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 232], 10.00th=[ 251], 20.00th=[ 275], | 30.00th=[ 300], 40.00th=[ 317], 50.00th=[ 338], 60.00th=[ 355], | 70.00th=[ 380], 80.00th=[ 409], 90.00th=[ 447], 95.00th=[ 498], | 99.00th=[ 609], 99.50th=[ 634], 99.90th=[ 667], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 7, max= 112, per=24.61%, avg=55.61, stdev=23.78, samples=176 iops : min= 1, max= 28, avg=13.49, stdev= 5.97, samples=176 write: IOPS=14, BW=59.5KiB/s (60.9kB/s)(5264KiB/88445msec); 0 zone resets slat (usec): min=52, max=11001, avg=324.11, stdev=721.23 clat (msec): min=176, max=1459, avg=737.06, stdev=193.26 lat (msec): min=176, max=1459, avg=737.39, stdev=193.27 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 447], 10.00th=[ 510], 20.00th=[ 575], | 30.00th=[ 634], 40.00th=[ 684], 50.00th=[ 726], 60.00th=[ 776], | 70.00th=[ 827], 80.00th=[ 885], 90.00th=[ 995], 95.00th=[ 1070], | 99.00th=[ 1250], 99.50th=[ 1301], 99.90th=[ 1418], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 7, max= 111, per=24.91%, avg=58.79, stdev=22.70, samples=176 iops : min= 1, max= 27, avg=14.29, stdev= 5.69, samples=176 lat (msec) : 50=0.12%, 100=0.08%, 250=4.88%, 500=45.51%, 750=25.98% lat (msec) : 1000=18.75%, 2000=4.69% cpu : usr=0.09%, sys=0.74%, ctx=4685, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180467: Wed May 20 18:14:44 2026 read: IOPS=14, BW=56.5KiB/s (57.9kB/s)(4984KiB/88193msec) slat (usec): min=55, max=49547, avg=10948.80, stdev=5745.64 clat (msec): min=11, max=936, avg=343.29, stdev=86.95 lat (msec): min=17, max=945, avg=354.24, stdev=88.27 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 228], 10.00th=[ 251], 20.00th=[ 275], | 30.00th=[ 296], 40.00th=[ 317], 50.00th=[ 334], 60.00th=[ 355], | 70.00th=[ 380], 80.00th=[ 409], 90.00th=[ 443], 95.00th=[ 485], | 99.00th=[ 600], 99.50th=[ 659], 99.90th=[ 869], 99.95th=[ 936], | 99.99th=[ 936] bw ( KiB/s): min= 8, max= 127, per=25.28%, avg=57.14, stdev=25.50, samples=172 iops : min= 2, max= 31, avg=13.88, stdev= 6.38, samples=172 write: IOPS=14, BW=59.6KiB/s (61.0kB/s)(5256KiB/88193msec); 0 zone resets slat (usec): min=53, max=13469, avg=296.94, stdev=635.56 clat (msec): min=94, max=1455, avg=737.08, stdev=200.79 lat (msec): min=94, max=1456, avg=737.37, stdev=200.77 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 439], 10.00th=[ 506], 20.00th=[ 567], | 30.00th=[ 617], 40.00th=[ 676], 50.00th=[ 726], 60.00th=[ 776], | 70.00th=[ 835], 80.00th=[ 894], 90.00th=[ 1003], 95.00th=[ 1083], | 99.00th=[ 1250], 99.50th=[ 1318], 99.90th=[ 1435], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 7, max= 120, per=24.92%, avg=58.81, stdev=21.65, samples=175 iops : min= 1, max= 30, avg=14.30, stdev= 5.43, samples=175 lat (msec) : 20=0.04%, 50=0.12%, 100=0.27%, 250=4.77%, 500=46.21% lat (msec) : 750=25.00%, 1000=18.09%, 2000=5.51% cpu : usr=0.08%, sys=0.68%, ctx=4730, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180468: Wed May 20 18:14:44 2026 read: IOPS=14, BW=57.3KiB/s (58.7kB/s)(5060KiB/88276msec) slat (usec): min=2028, max=42378, avg=11250.47, stdev=5733.24 clat (msec): min=9, max=662, avg=344.90, stdev=81.38 lat (msec): min=28, max=682, avg=356.16, stdev=82.56 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 234], 10.00th=[ 257], 20.00th=[ 279], | 30.00th=[ 296], 40.00th=[ 313], 50.00th=[ 334], 60.00th=[ 355], | 70.00th=[ 384], 80.00th=[ 409], 90.00th=[ 451], 95.00th=[ 493], | 99.00th=[ 567], 99.50th=[ 617], 99.90th=[ 659], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 7, max= 128, per=25.28%, avg=57.14, stdev=23.20, samples=174 iops : min= 1, max= 32, avg=13.87, stdev= 5.80, samples=174 write: IOPS=14, BW=58.7KiB/s (60.1kB/s)(5180KiB/88276msec); 0 zone resets slat (usec): min=55, max=12611, avg=325.57, stdev=774.10 clat (msec): min=154, max=1426, avg=740.75, stdev=197.44 lat (msec): min=154, max=1426, avg=741.08, stdev=197.43 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 456], 10.00th=[ 506], 20.00th=[ 567], | 30.00th=[ 625], 40.00th=[ 684], 50.00th=[ 726], 60.00th=[ 776], | 70.00th=[ 835], 80.00th=[ 894], 90.00th=[ 1003], 95.00th=[ 1099], | 99.00th=[ 1267], 99.50th=[ 1318], 99.90th=[ 1418], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 15, max= 119, per=24.77%, avg=58.45, stdev=21.53, samples=174 iops : min= 3, max= 29, avg=14.21, stdev= 5.40, samples=174 lat (msec) : 10=0.04%, 100=0.16%, 250=4.22%, 500=47.73%, 750=25.04% lat (msec) : 1000=17.54%, 2000=5.27% cpu : usr=0.10%, sys=0.73%, ctx=4886, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=227KiB/s (232kB/s), 56.3KiB/s-57.3KiB/s (57.6kB/s-58.7kB/s), io=19.6MiB (20.5MB), run=88193-88445msec WRITE: bw=236KiB/s (242kB/s), 58.7KiB/s-59.6KiB/s (60.1kB/s-61.0kB/s), io=20.4MiB (21.4MB), run=88193-88445msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180484: Wed May 20 18:14:51 2026 read: IOPS=2, BW=102MiB/s (107MB/s)(40.0MiB/392msec) slat (nsec): min=41223k, max=41223k, avg=41223360.00, stdev= 0.00 clat (nsec): min=329726k, max=329726k, avg=329725503.00, stdev= 0.00 lat (nsec): min=370955k, max=370955k, avg=370954945.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 330], 40.00th=[ 330], 50.00th=[ 330], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.07%, ctx=43, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=102MiB/s (107MB/s), 102MiB/s-102MiB/s (107MB/s-107MB/s), io=40.0MiB (41.9MB), run=392-392msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout