== sanity test 398c: run fio to test AIO ================= 14:27:45 (1788805665) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.315939 s, 133 MB/s osc.lustre-OST0000-osc-ffff986190cf7000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145695: Mon Sep 7 14:30:09 2026 write: IOPS=18, BW=75.1KiB/s (76.9kB/s)(10.0MiB/136322msec); 0 zone resets slat (usec): min=28, max=10394, avg=220.93, stdev=694.58 clat (msec): min=309, max=1622, avg=851.05, stdev=206.09 lat (msec): min=310, max=1622, avg=851.28, stdev=206.09 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 531], 10.00th=[ 617], 20.00th=[ 693], | 30.00th=[ 751], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 877], | 70.00th=[ 936], 80.00th=[ 1011], 90.00th=[ 1099], 95.00th=[ 1217], | 99.00th=[ 1401], 99.50th=[ 1485], 99.90th=[ 1620], 99.95th=[ 1620], | 99.99th=[ 1620] bw ( KiB/s): min= 7, max= 255, per=25.87%, avg=77.61, stdev=33.69, samples=260 iops : min= 1, max= 63, avg=18.96, stdev= 8.45, samples=260 lat (msec) : 500=4.10%, 750=25.62%, 1000=49.41%, 2000=20.86% cpu : usr=0.05%, sys=0.26%, ctx=2071, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145696: Mon Sep 7 14:30:09 2026 write: IOPS=18, BW=75.2KiB/s (77.0kB/s)(10.0MiB/136160msec); 0 zone resets slat (usec): min=28, max=12851, avg=195.19, stdev=568.14 clat (msec): min=183, max=1630, avg=850.08, stdev=203.13 lat (msec): min=183, max=1630, avg=850.28, stdev=203.13 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 558], 10.00th=[ 617], 20.00th=[ 693], | 30.00th=[ 743], 40.00th=[ 802], 50.00th=[ 844], 60.00th=[ 894], | 70.00th=[ 944], 80.00th=[ 1003], 90.00th=[ 1099], 95.00th=[ 1183], | 99.00th=[ 1368], 99.50th=[ 1485], 99.90th=[ 1569], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 248, per=25.00%, avg=74.99, stdev=31.64, samples=269 iops : min= 1, max= 62, avg=18.29, stdev= 7.95, samples=269 lat (msec) : 250=0.62%, 500=2.70%, 750=27.27%, 1000=48.67%, 2000=20.74% cpu : usr=0.05%, sys=0.24%, ctx=1703, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145697: Mon Sep 7 14:30:09 2026 write: IOPS=18, BW=75.2KiB/s (77.0kB/s)(10.0MiB/136158msec); 0 zone resets slat (usec): min=29, max=15253, avg=206.53, stdev=726.00 clat (msec): min=247, max=1634, avg=850.04, stdev=206.52 lat (msec): min=247, max=1634, avg=850.25, stdev=206.52 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 542], 10.00th=[ 617], 20.00th=[ 684], | 30.00th=[ 751], 40.00th=[ 802], 50.00th=[ 835], 60.00th=[ 877], | 70.00th=[ 944], 80.00th=[ 1011], 90.00th=[ 1083], 95.00th=[ 1217], | 99.00th=[ 1435], 99.50th=[ 1519], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 15, max= 256, per=25.47%, avg=76.42, stdev=31.63, samples=264 iops : min= 3, max= 64, avg=18.65, stdev= 7.97, samples=264 lat (msec) : 250=0.62%, 500=2.89%, 750=27.15%, 1000=47.77%, 2000=21.56% cpu : usr=0.05%, sys=0.24%, ctx=1689, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145698: Mon Sep 7 14:30:09 2026 write: IOPS=18, BW=75.2KiB/s (76.0kB/s)(10.0MiB/136223msec); 0 zone resets slat (usec): min=27, max=9661, avg=223.14, stdev=597.39 clat (msec): min=144, max=1639, avg=850.84, stdev=209.50 lat (msec): min=144, max=1639, avg=851.06, stdev=209.50 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 542], 10.00th=[ 617], 20.00th=[ 684], | 30.00th=[ 751], 40.00th=[ 802], 50.00th=[ 852], 60.00th=[ 885], | 70.00th=[ 944], 80.00th=[ 1003], 90.00th=[ 1116], 95.00th=[ 1200], | 99.00th=[ 1385], 99.50th=[ 1485], 99.90th=[ 1586], 99.95th=[ 1620], | 99.99th=[ 1636] bw ( KiB/s): min= 7, max= 225, per=24.89%, avg=74.67, stdev=30.65, samples=270 iops : min= 1, max= 56, avg=18.21, stdev= 7.73, samples=270 lat (msec) : 250=0.62%, 500=3.52%, 750=26.05%, 1000=49.41%, 2000=20.39% cpu : usr=0.06%, sys=0.23%, ctx=2146, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=300KiB/s (308kB/s), 75.1KiB/s-75.2KiB/s (76.9kB/s-77.0kB/s), io=40.0MiB (41.9MB), run=136158-136322msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145736: Mon Sep 7 14:31:40 2026 read: IOPS=15, BW=60.8KiB/s (62.2kB/s)(5048KiB/83082msec) slat (usec): min=2692, max=54222, avg=10732.41, stdev=5511.15 clat (msec): min=17, max=588, avg=329.71, stdev=77.10 lat (msec): min=25, max=607, avg=340.45, stdev=77.93 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 215], 10.00th=[ 245], 20.00th=[ 271], | 30.00th=[ 292], 40.00th=[ 309], 50.00th=[ 326], 60.00th=[ 347], | 70.00th=[ 363], 80.00th=[ 393], 90.00th=[ 426], 95.00th=[ 464], | 99.00th=[ 518], 99.50th=[ 542], 99.90th=[ 567], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 15, max= 143, per=24.96%, avg=60.16, stdev=24.71, samples=165 iops : min= 3, max= 35, avg=14.57, stdev= 6.20, samples=165 write: IOPS=15, BW=62.5KiB/s (63.0kB/s)(5192KiB/83082msec); 0 zone resets slat (usec): min=31, max=10038, avg=257.92, stdev=664.91 clat (msec): min=283, max=1216, avg=690.55, stdev=159.62 lat (msec): min=284, max=1217, avg=690.81, stdev=159.62 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 414], 10.00th=[ 493], 20.00th=[ 558], | 30.00th=[ 617], 40.00th=[ 642], 50.00th=[ 684], 60.00th=[ 726], | 70.00th=[ 776], 80.00th=[ 827], 90.00th=[ 885], 95.00th=[ 953], | 99.00th=[ 1116], 99.50th=[ 1183], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 15, max= 136, per=24.62%, avg=61.80, stdev=21.66, samples=165 iops : min= 3, max= 34, avg=14.98, stdev= 5.44, samples=165 lat (msec) : 20=0.08%, 50=0.16%, 100=0.27%, 250=5.20%, 500=48.40% lat (msec) : 750=28.09%, 1000=16.25%, 2000=1.56% cpu : usr=0.09%, sys=0.72%, ctx=4773, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145737: Mon Sep 7 14:31:40 2026 read: IOPS=14, BW=59.9KiB/s (61.4kB/s)(4976KiB/83014msec) slat (usec): min=34, max=41272, avg=10709.71, stdev=5660.17 clat (msec): min=9, max=892, avg=328.72, stdev=86.15 lat (msec): min=16, max=907, avg=339.43, stdev=86.96 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 218], 10.00th=[ 239], 20.00th=[ 268], | 30.00th=[ 288], 40.00th=[ 309], 50.00th=[ 326], 60.00th=[ 342], | 70.00th=[ 359], 80.00th=[ 388], 90.00th=[ 426], 95.00th=[ 468], | 99.00th=[ 550], 99.50th=[ 651], 99.90th=[ 885], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 15, max= 151, per=24.72%, avg=59.59, stdev=24.64, samples=164 iops : min= 3, max= 37, avg=14.43, stdev= 6.13, samples=164 write: IOPS=15, BW=63.4KiB/s (64.9kB/s)(5264KiB/83014msec); 0 zone resets slat (usec): min=32, max=14777, avg=271.69, stdev=705.04 clat (msec): min=96, max=1237, avg=687.40, stdev=171.11 lat (msec): min=96, max=1237, avg=687.68, stdev=171.13 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 418], 10.00th=[ 481], 20.00th=[ 550], | 30.00th=[ 609], 40.00th=[ 642], 50.00th=[ 676], 60.00th=[ 726], | 70.00th=[ 776], 80.00th=[ 835], 90.00th=[ 902], 95.00th=[ 969], | 99.00th=[ 1116], 99.50th=[ 1183], 99.90th=[ 1217], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 8, max= 119, per=25.11%, avg=63.04, stdev=21.62, samples=164 iops : min= 2, max= 29, avg=15.29, stdev= 5.43, samples=164 lat (msec) : 10=0.04%, 20=0.08%, 50=0.12%, 100=0.66%, 250=5.94% lat (msec) : 500=47.58%, 750=27.15%, 1000=16.56%, 2000=1.88% cpu : usr=0.10%, sys=0.68%, ctx=4631, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145738: Mon Sep 7 14:31:40 2026 read: IOPS=14, BW=59.0KiB/s (61.4kB/s)(4984KiB/83082msec) slat (usec): min=2355, max=56170, avg=10944.86, stdev=5505.68 clat (msec): min=13, max=713, avg=331.56, stdev=78.11 lat (msec): min=20, max=727, avg=342.51, stdev=78.93 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 218], 10.00th=[ 243], 20.00th=[ 271], | 30.00th=[ 292], 40.00th=[ 313], 50.00th=[ 330], 60.00th=[ 347], | 70.00th=[ 363], 80.00th=[ 397], 90.00th=[ 435], 95.00th=[ 468], | 99.00th=[ 523], 99.50th=[ 542], 99.90th=[ 592], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 15, max= 144, per=24.73%, avg=59.59, stdev=23.97, samples=165 iops : min= 3, max= 36, avg=14.42, stdev= 5.99, samples=165 write: IOPS=15, BW=63.3KiB/s (64.8kB/s)(5256KiB/83082msec); 0 zone resets slat (usec): min=32, max=20300, avg=261.17, stdev=712.08 clat (msec): min=225, max=1237, avg=685.00, stdev=161.74 lat (msec): min=225, max=1237, avg=685.26, stdev=161.76 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 414], 10.00th=[ 477], 20.00th=[ 550], | 30.00th=[ 609], 40.00th=[ 642], 50.00th=[ 676], 60.00th=[ 718], | 70.00th=[ 776], 80.00th=[ 827], 90.00th=[ 885], 95.00th=[ 944], | 99.00th=[ 1083], 99.50th=[ 1150], 99.90th=[ 1200], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 8, max= 120, per=24.83%, avg=62.33, stdev=20.93, samples=165 iops : min= 2, max= 30, avg=15.11, stdev= 5.23, samples=165 lat (msec) : 20=0.12%, 100=0.27%, 250=5.70%, 500=48.32%, 750=27.85% lat (msec) : 1000=16.48%, 2000=1.25% cpu : usr=0.12%, sys=0.69%, ctx=4756, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145739: Mon Sep 7 14:31:40 2026 read: IOPS=15, BW=61.2KiB/s (62.6kB/s)(5060KiB/82711msec) slat (usec): min=2285, max=45422, avg=10915.81, stdev=5541.82 clat (msec): min=26, max=1000, avg=332.49, stdev=79.63 lat (msec): min=33, max=1007, avg=343.41, stdev=80.36 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 222], 10.00th=[ 245], 20.00th=[ 271], | 30.00th=[ 296], 40.00th=[ 309], 50.00th=[ 326], 60.00th=[ 342], | 70.00th=[ 363], 80.00th=[ 393], 90.00th=[ 435], 95.00th=[ 464], | 99.00th=[ 527], 99.50th=[ 575], 99.90th=[ 785], 99.95th=[ 1003], | 99.99th=[ 1003] bw ( KiB/s): min= 15, max= 168, per=25.20%, avg=60.74, stdev=22.57, samples=164 iops : min= 3, max= 42, avg=14.71, stdev= 5.68, samples=164 write: IOPS=15, BW=62.6KiB/s (64.1kB/s)(5180KiB/82711msec); 0 zone resets slat (usec): min=32, max=9199, avg=261.09, stdev=666.28 clat (msec): min=266, max=1295, avg=684.30, stdev=164.45 lat (msec): min=266, max=1295, avg=684.56, stdev=164.45 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 401], 10.00th=[ 472], 20.00th=[ 550], | 30.00th=[ 600], 40.00th=[ 642], 50.00th=[ 676], 60.00th=[ 726], | 70.00th=[ 768], 80.00th=[ 827], 90.00th=[ 885], 95.00th=[ 944], | 99.00th=[ 1099], 99.50th=[ 1183], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 16, max= 119, per=24.68%, avg=61.95, stdev=20.49, samples=164 iops : min= 4, max= 29, avg=15.02, stdev= 5.12, samples=164 lat (msec) : 50=0.12%, 100=0.16%, 250=5.51%, 500=49.30%, 750=27.34% lat (msec) : 1000=16.17%, 2000=1.41% cpu : usr=0.11%, sys=0.71%, ctx=5026, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=242KiB/s (247kB/s), 59.9KiB/s-61.2KiB/s (61.4kB/s-62.6kB/s), io=19.6MiB (20.5MB), run=82711-83082msec WRITE: bw=251KiB/s (257kB/s), 62.5KiB/s-63.4KiB/s (63.0kB/s-64.9kB/s), io=20.4MiB (21.4MB), run=82711-83082msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145755: Mon Sep 7 14:31:47 2026 read: IOPS=2, BW=97.3MiB/s (102MB/s)(40.0MiB/411msec) slat (nsec): min=59717k, max=59717k, avg=59717192.00, stdev= 0.00 clat (nsec): min=328440k, max=328440k, avg=328440141.00, stdev= 0.00 lat (nsec): min=388161k, max=388161k, avg=388161290.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 330], 10.00th=[ 330], 20.00th=[ 330], | 30.00th=[ 330], 40.00th=[ 330], 50.00th=[ 330], 60.00th=[ 330], | 70.00th=[ 330], 80.00th=[ 330], 90.00th=[ 330], 95.00th=[ 330], | 99.00th=[ 330], 99.50th=[ 330], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] lat (msec) : 500=100.00% cpu : usr=0.24%, sys=3.66%, ctx=42, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=97.3MiB/s (102MB/s), 97.3MiB/s-97.3MiB/s (102MB/s-102MB/s), io=40.0MiB (41.9MB), run=411-411msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout