== sanity test 398c: run fio to test AIO ================= 04:48:27 (1781686107) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.779677 s, 53.8 MB/s osc.lustre-OST0000-osc-ffff9e3080682000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=159063: Wed Jun 17 04:50:46 2026 write: IOPS=19, BW=79.8KiB/s (81.7kB/s)(10.0MiB/128318msec); 0 zone resets slat (usec): min=39, max=22213, avg=324.11, stdev=887.31 clat (msec): min=149, max=1579, avg=801.21, stdev=192.97 lat (msec): min=149, max=1579, avg=801.54, stdev=192.99 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 518], 10.00th=[ 592], 20.00th=[ 659], | 30.00th=[ 709], 40.00th=[ 751], 50.00th=[ 785], 60.00th=[ 835], | 70.00th=[ 894], 80.00th=[ 953], 90.00th=[ 1045], 95.00th=[ 1116], | 99.00th=[ 1351], 99.50th=[ 1435], 99.90th=[ 1519], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 241, per=25.41%, avg=80.80, stdev=31.53, samples=250 iops : min= 1, max= 60, avg=19.77, stdev= 7.91, samples=250 lat (msec) : 250=0.62%, 500=4.02%, 750=35.16%, 1000=46.09%, 2000=14.10% cpu : usr=0.04%, sys=0.35%, ctx=2165, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159064: Wed Jun 17 04:50:46 2026 write: IOPS=19, BW=79.8KiB/s (81.7kB/s)(10.0MiB/128401msec); 0 zone resets slat (usec): min=39, max=19348, avg=338.86, stdev=805.46 clat (msec): min=29, max=1512, avg=801.21, stdev=184.69 lat (msec): min=33, max=1512, avg=801.55, stdev=184.71 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 535], 10.00th=[ 600], 20.00th=[ 659], | 30.00th=[ 709], 40.00th=[ 751], 50.00th=[ 793], 60.00th=[ 835], | 70.00th=[ 894], 80.00th=[ 953], 90.00th=[ 1036], 95.00th=[ 1099], | 99.00th=[ 1284], 99.50th=[ 1351], 99.90th=[ 1469], 99.95th=[ 1485], | 99.99th=[ 1519] bw ( KiB/s): min= 7, max= 254, per=24.91%, avg=79.22, stdev=28.15, samples=255 iops : min= 1, max= 63, avg=19.37, stdev= 7.07, samples=255 lat (msec) : 50=0.04%, 100=0.04%, 250=0.23%, 500=3.63%, 750=36.56% lat (msec) : 1000=45.82%, 2000=13.67% cpu : usr=0.06%, sys=0.38%, ctx=2751, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159065: Wed Jun 17 04:50:46 2026 write: IOPS=19, BW=79.7KiB/s (81.7kB/s)(10.0MiB/128402msec); 0 zone resets slat (usec): min=38, max=10927, avg=326.28, stdev=722.11 clat (msec): min=182, max=1570, avg=801.32, stdev=185.12 lat (msec): min=182, max=1570, avg=801.65, stdev=185.14 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 542], 10.00th=[ 609], 20.00th=[ 659], | 30.00th=[ 701], 40.00th=[ 743], 50.00th=[ 785], 60.00th=[ 835], | 70.00th=[ 894], 80.00th=[ 953], 90.00th=[ 1036], 95.00th=[ 1099], | 99.00th=[ 1284], 99.50th=[ 1334], 99.90th=[ 1502], 99.95th=[ 1536], | 99.99th=[ 1569] bw ( KiB/s): min= 7, max= 256, per=25.09%, avg=79.79, stdev=32.11, samples=253 iops : min= 1, max= 64, avg=19.51, stdev= 8.08, samples=253 lat (msec) : 250=0.62%, 500=2.62%, 750=37.77%, 1000=44.96%, 2000=14.02% cpu : usr=0.05%, sys=0.35%, ctx=2338, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=159066: Wed Jun 17 04:50:46 2026 write: IOPS=19, BW=79.7KiB/s (81.6kB/s)(10.0MiB/128477msec); 0 zone resets slat (usec): min=39, max=11739, avg=299.89, stdev=670.53 clat (msec): min=187, max=1570, avg=801.73, stdev=183.79 lat (msec): min=189, max=1570, avg=802.03, stdev=183.79 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 531], 10.00th=[ 592], 20.00th=[ 667], | 30.00th=[ 709], 40.00th=[ 751], 50.00th=[ 785], 60.00th=[ 844], | 70.00th=[ 885], 80.00th=[ 944], 90.00th=[ 1028], 95.00th=[ 1099], | 99.00th=[ 1284], 99.50th=[ 1318], 99.90th=[ 1536], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 7, max= 254, per=25.21%, avg=80.16, stdev=33.13, samples=252 iops : min= 1, max= 63, avg=19.61, stdev= 8.33, samples=252 lat (msec) : 250=0.04%, 500=3.91%, 750=35.55%, 1000=47.03%, 2000=13.48% cpu : usr=0.04%, sys=0.33%, ctx=2071, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=319KiB/s (326kB/s), 79.7KiB/s-79.8KiB/s (81.6kB/s-81.7kB/s), io=40.0MiB (41.9MB), run=128318-128477msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=159094: Wed Jun 17 04:52:21 2026 read: IOPS=14, BW=57.1KiB/s (58.4kB/s)(5048KiB/88477msec) slat (usec): min=2828, max=45271, avg=11411.61, stdev=5247.04 clat (msec): min=16, max=632, avg=341.17, stdev=69.74 lat (msec): min=22, max=639, avg=352.59, stdev=70.39 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 234], 10.00th=[ 259], 20.00th=[ 284], | 30.00th=[ 305], 40.00th=[ 321], 50.00th=[ 342], 60.00th=[ 359], | 70.00th=[ 376], 80.00th=[ 397], 90.00th=[ 430], 95.00th=[ 451], | 99.00th=[ 527], 99.50th=[ 542], 99.90th=[ 575], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 8, max= 128, per=25.00%, avg=56.50, stdev=22.63, samples=176 iops : min= 2, max= 32, avg=13.73, stdev= 5.67, samples=176 write: IOPS=14, BW=58.7KiB/s (60.1kB/s)(5192KiB/88477msec); 0 zone resets slat (usec): min=54, max=17042, avg=352.39, stdev=802.99 clat (msec): min=140, max=1229, avg=745.68, stdev=184.56 lat (msec): min=140, max=1229, avg=746.04, stdev=184.56 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 443], 10.00th=[ 506], 20.00th=[ 584], | 30.00th=[ 642], 40.00th=[ 701], 50.00th=[ 751], 60.00th=[ 793], | 70.00th=[ 844], 80.00th=[ 911], 90.00th=[ 995], 95.00th=[ 1053], | 99.00th=[ 1167], 99.50th=[ 1183], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 15, max= 168, per=24.66%, avg=57.94, stdev=22.14, samples=176 iops : min= 3, max= 42, avg=14.10, stdev= 5.54, samples=176 lat (msec) : 20=0.04%, 50=0.04%, 100=0.16%, 250=3.79%, 500=49.38% lat (msec) : 750=21.41%, 1000=20.47%, 2000=4.73% cpu : usr=0.10%, sys=0.77%, ctx=4499, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159095: Wed Jun 17 04:52:21 2026 read: IOPS=14, BW=56.0KiB/s (57.4kB/s)(4976KiB/88782msec) slat (usec): min=2693, max=42038, avg=11601.18, stdev=5506.21 clat (msec): min=19, max=578, avg=340.60, stdev=70.69 lat (msec): min=39, max=598, avg=352.21, stdev=71.24 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 234], 10.00th=[ 257], 20.00th=[ 284], | 30.00th=[ 305], 40.00th=[ 326], 50.00th=[ 338], 60.00th=[ 355], | 70.00th=[ 376], 80.00th=[ 397], 90.00th=[ 426], 95.00th=[ 456], | 99.00th=[ 514], 99.50th=[ 535], 99.90th=[ 575], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 8, max= 136, per=24.46%, avg=55.28, stdev=22.84, samples=176 iops : min= 2, max= 34, avg=13.43, stdev= 5.70, samples=176 write: IOPS=14, BW=59.3KiB/s (60.7kB/s)(5264KiB/88782msec); 0 zone resets slat (usec): min=57, max=12541, avg=356.92, stdev=754.98 clat (msec): min=167, max=1253, avg=744.93, stdev=185.46 lat (msec): min=167, max=1254, avg=745.29, stdev=185.46 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 447], 10.00th=[ 506], 20.00th=[ 584], | 30.00th=[ 651], 40.00th=[ 701], 50.00th=[ 743], 60.00th=[ 793], | 70.00th=[ 844], 80.00th=[ 911], 90.00th=[ 986], 95.00th=[ 1053], | 99.00th=[ 1150], 99.50th=[ 1200], 99.90th=[ 1234], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 119, per=24.92%, avg=58.55, stdev=20.94, samples=176 iops : min= 1, max= 29, avg=14.24, stdev= 5.23, samples=176 lat (msec) : 20=0.04%, 50=0.04%, 100=0.23%, 250=3.87%, 500=48.28% lat (msec) : 750=22.34%, 1000=20.82%, 2000=4.38% cpu : usr=0.10%, sys=0.79%, ctx=4723, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159096: Wed Jun 17 04:52:21 2026 read: IOPS=14, BW=56.2KiB/s (57.5kB/s)(4984KiB/88743msec) slat (usec): min=83, max=51013, avg=11525.44, stdev=5608.85 clat (msec): min=13, max=566, avg=340.53, stdev=69.50 lat (msec): min=13, max=579, avg=352.06, stdev=70.48 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 241], 10.00th=[ 266], 20.00th=[ 284], | 30.00th=[ 305], 40.00th=[ 326], 50.00th=[ 338], 60.00th=[ 355], | 70.00th=[ 376], 80.00th=[ 393], 90.00th=[ 430], 95.00th=[ 456], | 99.00th=[ 502], 99.50th=[ 527], 99.90th=[ 558], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 7, max= 127, per=24.63%, avg=55.66, stdev=24.86, samples=176 iops : min= 1, max= 31, avg=13.52, stdev= 6.25, samples=176 write: IOPS=14, BW=59.2KiB/s (60.6kB/s)(5256KiB/88743msec); 0 zone resets slat (usec): min=40, max=19274, avg=423.67, stdev=1037.57 clat (msec): min=85, max=1283, avg=745.92, stdev=192.97 lat (msec): min=85, max=1283, avg=746.35, stdev=192.99 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 435], 10.00th=[ 502], 20.00th=[ 584], | 30.00th=[ 651], 40.00th=[ 701], 50.00th=[ 751], 60.00th=[ 802], | 70.00th=[ 852], 80.00th=[ 911], 90.00th=[ 986], 95.00th=[ 1053], | 99.00th=[ 1150], 99.50th=[ 1167], 99.90th=[ 1250], 99.95th=[ 1284], | 99.99th=[ 1284] bw ( KiB/s): min= 15, max= 121, per=24.74%, avg=58.13, stdev=22.04, samples=176 iops : min= 3, max= 30, avg=14.14, stdev= 5.51, samples=176 lat (msec) : 20=0.08%, 50=0.23%, 100=0.12%, 250=3.28%, 500=49.57% lat (msec) : 750=20.55%, 1000=21.33%, 2000=4.84% cpu : usr=0.09%, sys=0.78%, ctx=4918, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=159097: Wed Jun 17 04:52:21 2026 read: IOPS=14, BW=57.2KiB/s (58.5kB/s)(5060KiB/88530msec) slat (usec): min=2736, max=39494, avg=11420.85, stdev=5440.79 clat (msec): min=12, max=583, avg=339.92, stdev=70.74 lat (msec): min=19, max=607, avg=351.35, stdev=71.35 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 234], 10.00th=[ 257], 20.00th=[ 284], | 30.00th=[ 305], 40.00th=[ 321], 50.00th=[ 338], 60.00th=[ 355], | 70.00th=[ 376], 80.00th=[ 393], 90.00th=[ 430], 95.00th=[ 456], | 99.00th=[ 518], 99.50th=[ 542], 99.90th=[ 575], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 7, max= 120, per=25.24%, avg=57.05, stdev=22.77, samples=175 iops : min= 1, max= 30, avg=13.87, stdev= 5.70, samples=175 write: IOPS=14, BW=58.5KiB/s (59.9kB/s)(5180KiB/88530msec); 0 zone resets slat (usec): min=55, max=11908, avg=340.31, stdev=683.68 clat (msec): min=226, max=1218, avg=749.13, stdev=182.27 lat (msec): min=230, max=1219, avg=749.47, stdev=182.26 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 456], 10.00th=[ 510], 20.00th=[ 584], | 30.00th=[ 651], 40.00th=[ 709], 50.00th=[ 751], 60.00th=[ 802], | 70.00th=[ 852], 80.00th=[ 911], 90.00th=[ 978], 95.00th=[ 1053], | 99.00th=[ 1150], 99.50th=[ 1167], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 15, max= 127, per=24.57%, avg=57.74, stdev=19.35, samples=176 iops : min= 3, max= 31, avg=14.05, stdev= 4.86, samples=176 lat (msec) : 20=0.08%, 50=0.08%, 100=0.16%, 250=3.83%, 500=48.63% lat (msec) : 750=21.99%, 1000=20.90%, 2000=4.34% cpu : usr=0.11%, sys=0.79%, ctx=4675, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=226KiB/s (231kB/s), 56.0KiB/s-57.2KiB/s (57.4kB/s-58.5kB/s), io=19.6MiB (20.5MB), run=88477-88782msec WRITE: bw=235KiB/s (241kB/s), 58.5KiB/s-59.3KiB/s (59.9kB/s-60.7kB/s), io=20.4MiB (21.4MB), run=88477-88782msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=159113: Wed Jun 17 04:52:28 2026 read: IOPS=2, BW=107MiB/s (112MB/s)(40.0MiB/373msec) slat (nsec): min=19777k, max=19777k, avg=19777104.00, stdev= 0.00 clat (nsec): min=339116k, max=339116k, avg=339116022.00, stdev= 0.00 lat (nsec): min=358899k, max=358899k, avg=358898850.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 338], 10.00th=[ 338], 20.00th=[ 338], | 30.00th=[ 338], 40.00th=[ 338], 50.00th=[ 338], 60.00th=[ 338], | 70.00th=[ 338], 80.00th=[ 338], 90.00th=[ 338], 95.00th=[ 338], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.96%, ctx=43, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=107MiB/s (112MB/s), 107MiB/s-107MiB/s (112MB/s-112MB/s), io=40.0MiB (41.9MB), run=373-373msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout