== sanity test 398c: run fio to test AIO ================= 11:04:37 (1761318277) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.272945 s, 154 MB/s osc.lustre-OST0000-osc-ffff980542c72000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=123613: Fri Oct 24 11:06:36 2025 write: IOPS=22, BW=90.7KiB/s (92.9kB/s)(10.0MiB/112899msec); 0 zone resets slat (usec): min=25, max=7598, avg=191.85, stdev=406.16 clat (msec): min=110, max=1247, avg=705.04, stdev=158.04 lat (msec): min=111, max=1247, avg=705.23, stdev=158.05 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 460], 10.00th=[ 527], 20.00th=[ 592], | 30.00th=[ 625], 40.00th=[ 676], 50.00th=[ 701], 60.00th=[ 735], | 70.00th=[ 776], 80.00th=[ 818], 90.00th=[ 911], 95.00th=[ 978], | 99.00th=[ 1116], 99.50th=[ 1167], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1250] bw ( KiB/s): min= 15, max= 231, per=24.87%, avg=90.03, stdev=29.87, samples=224 iops : min= 3, max= 57, avg=22.01, stdev= 7.47, samples=224 lat (msec) : 250=0.62%, 500=7.07%, 750=57.03%, 1000=31.33%, 2000=3.95% cpu : usr=0.07%, sys=0.33%, ctx=1492, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123614: Fri Oct 24 11:06:36 2025 write: IOPS=22, BW=90.9KiB/s (93.1kB/s)(10.0MiB/112680msec); 0 zone resets slat (usec): min=23, max=5504, avg=201.31, stdev=398.48 clat (msec): min=125, max=1247, avg=703.66, stdev=156.37 lat (msec): min=126, max=1247, avg=703.86, stdev=156.40 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 451], 10.00th=[ 531], 20.00th=[ 592], | 30.00th=[ 625], 40.00th=[ 667], 50.00th=[ 701], 60.00th=[ 726], | 70.00th=[ 776], 80.00th=[ 818], 90.00th=[ 911], 95.00th=[ 961], | 99.00th=[ 1099], 99.50th=[ 1150], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 239, per=24.98%, avg=90.43, stdev=30.15, samples=223 iops : min= 1, max= 59, avg=22.12, stdev= 7.54, samples=223 lat (msec) : 250=0.74%, 500=6.60%, 750=57.85%, 1000=31.64%, 2000=3.16% cpu : usr=0.06%, sys=0.36%, ctx=1467, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123615: Fri Oct 24 11:06:36 2025 write: IOPS=22, BW=90.6KiB/s (92.8kB/s)(10.0MiB/112976msec); 0 zone resets slat (usec): min=23, max=8376, avg=201.43, stdev=451.39 clat (msec): min=213, max=1247, avg=705.42, stdev=154.93 lat (msec): min=213, max=1247, avg=705.62, stdev=154.95 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 468], 10.00th=[ 527], 20.00th=[ 584], | 30.00th=[ 625], 40.00th=[ 676], 50.00th=[ 701], 60.00th=[ 735], | 70.00th=[ 776], 80.00th=[ 827], 90.00th=[ 902], 95.00th=[ 961], | 99.00th=[ 1116], 99.50th=[ 1167], 99.90th=[ 1234], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 254, per=24.89%, avg=90.11, stdev=32.19, samples=224 iops : min= 1, max= 63, avg=22.04, stdev= 8.08, samples=224 lat (msec) : 250=0.62%, 500=6.91%, 750=55.00%, 1000=34.41%, 2000=3.05% cpu : usr=0.06%, sys=0.33%, ctx=1415, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123616: Fri Oct 24 11:06:36 2025 write: IOPS=22, BW=90.6KiB/s (92.8kB/s)(10.0MiB/113005msec); 0 zone resets slat (usec): min=26, max=5476, avg=216.43, stdev=454.07 clat (msec): min=51, max=1232, avg=705.67, stdev=153.76 lat (msec): min=54, max=1232, avg=705.89, stdev=153.76 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 472], 10.00th=[ 527], 20.00th=[ 584], | 30.00th=[ 625], 40.00th=[ 667], 50.00th=[ 701], 60.00th=[ 735], | 70.00th=[ 776], 80.00th=[ 827], 90.00th=[ 902], 95.00th=[ 969], | 99.00th=[ 1099], 99.50th=[ 1167], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 7, max= 254, per=24.89%, avg=90.12, stdev=29.31, samples=224 iops : min= 1, max= 63, avg=22.04, stdev= 7.36, samples=224 lat (msec) : 100=0.04%, 250=0.04%, 500=7.11%, 750=56.72%, 1000=32.66% lat (msec) : 2000=3.44% cpu : usr=0.06%, sys=0.37%, ctx=1583, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=362KiB/s (371kB/s), 90.6KiB/s-90.9KiB/s (92.8kB/s-93.1kB/s), io=40.0MiB (41.9MB), run=112680-113005msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=123642: Fri Oct 24 11:07:56 2025 read: IOPS=17, BW=68.2KiB/s (69.9kB/s)(5048KiB/73977msec) slat (usec): min=2326, max=37889, avg=8324.77, stdev=3937.33 clat (msec): min=8, max=622, avg=300.15, stdev=75.26 lat (msec): min=13, max=630, avg=308.49, stdev=76.16 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 192], 10.00th=[ 215], 20.00th=[ 241], | 30.00th=[ 257], 40.00th=[ 275], 50.00th=[ 292], 60.00th=[ 305], | 70.00th=[ 330], 80.00th=[ 363], 90.00th=[ 397], 95.00th=[ 439], | 99.00th=[ 514], 99.50th=[ 550], 99.90th=[ 617], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 15, max= 135, per=25.01%, avg=67.54, stdev=25.95, samples=147 iops : min= 3, max= 33, avg=16.34, stdev= 6.53, samples=147 write: IOPS=17, BW=70.2KiB/s (71.9kB/s)(5192KiB/73977msec); 0 zone resets slat (usec): min=27, max=7177, avg=266.31, stdev=566.45 clat (msec): min=225, max=1161, avg=610.22, stdev=165.72 lat (msec): min=225, max=1161, avg=610.49, stdev=165.78 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 372], 10.00th=[ 409], 20.00th=[ 464], | 30.00th=[ 510], 40.00th=[ 550], 50.00th=[ 592], 60.00th=[ 642], | 70.00th=[ 701], 80.00th=[ 760], 90.00th=[ 844], 95.00th=[ 911], | 99.00th=[ 1003], 99.50th=[ 1028], 99.90th=[ 1133], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 15, max= 143, per=24.69%, avg=69.37, stdev=21.46, samples=147 iops : min= 3, max= 35, avg=16.80, stdev= 5.34, samples=147 lat (msec) : 10=0.04%, 50=0.12%, 100=0.04%, 250=12.73%, 500=49.92% lat (msec) : 750=26.64%, 1000=10.00%, 2000=0.51% cpu : usr=0.12%, sys=0.92%, ctx=2885, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123643: Fri Oct 24 11:07:56 2025 read: IOPS=16, BW=67.1KiB/s (68.7kB/s)(4976KiB/74168msec) slat (usec): min=100, max=28393, avg=8355.25, stdev=3801.94 clat (msec): min=13, max=622, avg=303.48, stdev=76.94 lat (msec): min=19, max=630, avg=311.85, stdev=77.67 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 199], 10.00th=[ 218], 20.00th=[ 241], | 30.00th=[ 259], 40.00th=[ 279], 50.00th=[ 296], 60.00th=[ 313], | 70.00th=[ 334], 80.00th=[ 363], 90.00th=[ 409], 95.00th=[ 447], | 99.00th=[ 518], 99.50th=[ 550], 99.90th=[ 575], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 16, max= 135, per=24.61%, avg=66.44, stdev=24.38, samples=147 iops : min= 4, max= 33, avg=16.06, stdev= 6.17, samples=147 write: IOPS=17, BW=70.0KiB/s (72.7kB/s)(5264KiB/74168msec); 0 zone resets slat (usec): min=24, max=7917, avg=253.00, stdev=553.88 clat (msec): min=170, max=1123, avg=606.03, stdev=167.08 lat (msec): min=170, max=1123, avg=606.28, stdev=167.10 clat percentiles (msec): | 1.00th=[ 268], 5.00th=[ 363], 10.00th=[ 401], 20.00th=[ 456], | 30.00th=[ 506], 40.00th=[ 550], 50.00th=[ 592], 60.00th=[ 634], | 70.00th=[ 693], 80.00th=[ 743], 90.00th=[ 852], 95.00th=[ 919], | 99.00th=[ 1003], 99.50th=[ 1011], 99.90th=[ 1099], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 16, max= 127, per=25.04%, avg=70.37, stdev=23.22, samples=147 iops : min= 4, max= 31, avg=17.05, stdev= 5.76, samples=147 lat (msec) : 20=0.08%, 50=0.08%, 100=0.12%, 250=12.27%, 500=50.20% lat (msec) : 750=27.70%, 1000=8.98%, 2000=0.59% cpu : usr=0.13%, sys=0.90%, ctx=2856, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123644: Fri Oct 24 11:07:56 2025 read: IOPS=16, BW=67.2KiB/s (68.8kB/s)(4984KiB/74168msec) slat (usec): min=2590, max=31656, avg=8391.59, stdev=3899.69 clat (msec): min=11, max=622, avg=302.50, stdev=76.70 lat (msec): min=20, max=632, avg=310.90, stdev=77.52 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 197], 10.00th=[ 220], 20.00th=[ 239], | 30.00th=[ 257], 40.00th=[ 279], 50.00th=[ 292], 60.00th=[ 309], | 70.00th=[ 334], 80.00th=[ 372], 90.00th=[ 409], 95.00th=[ 435], | 99.00th=[ 493], 99.50th=[ 542], 99.90th=[ 617], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min= 23, max= 175, per=24.74%, avg=66.79, stdev=28.62, samples=147 iops : min= 5, max= 43, avg=16.16, stdev= 7.21, samples=147 write: IOPS=17, BW=70.9KiB/s (72.6kB/s)(5256KiB/74168msec); 0 zone resets slat (usec): min=21, max=8987, avg=272.86, stdev=617.83 clat (msec): min=141, max=1065, avg=606.90, stdev=167.10 lat (msec): min=143, max=1066, avg=607.18, stdev=167.14 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 359], 10.00th=[ 401], 20.00th=[ 456], | 30.00th=[ 506], 40.00th=[ 550], 50.00th=[ 592], 60.00th=[ 642], | 70.00th=[ 684], 80.00th=[ 751], 90.00th=[ 852], 95.00th=[ 902], | 99.00th=[ 1003], 99.50th=[ 1028], 99.90th=[ 1062], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 8, max= 143, per=24.91%, avg=70.00, stdev=22.35, samples=147 iops : min= 2, max= 35, avg=16.96, stdev= 5.60, samples=147 lat (msec) : 20=0.08%, 50=0.16%, 100=0.16%, 250=12.46%, 500=50.08% lat (msec) : 750=27.03%, 1000=9.53%, 2000=0.51% cpu : usr=0.14%, sys=0.92%, ctx=3045, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123645: Fri Oct 24 11:07:56 2025 read: IOPS=17, BW=68.6KiB/s (70.3kB/s)(5060KiB/73729msec) slat (usec): min=40, max=41975, avg=8050.20, stdev=3863.55 clat (msec): min=5, max=989, avg=299.51, stdev=81.97 lat (msec): min=6, max=999, avg=307.56, stdev=82.80 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 192], 10.00th=[ 211], 20.00th=[ 239], | 30.00th=[ 255], 40.00th=[ 275], 50.00th=[ 292], 60.00th=[ 309], | 70.00th=[ 326], 80.00th=[ 359], 90.00th=[ 401], 95.00th=[ 435], | 99.00th=[ 531], 99.50th=[ 550], 99.90th=[ 919], 99.95th=[ 986], | 99.99th=[ 986] bw ( KiB/s): min= 7, max= 144, per=25.45%, avg=68.71, stdev=26.90, samples=145 iops : min= 1, max= 36, avg=16.64, stdev= 6.76, samples=145 write: IOPS=17, BW=70.3KiB/s (71.9kB/s)(5180KiB/73729msec); 0 zone resets slat (usec): min=28, max=9935, avg=277.05, stdev=645.02 clat (msec): min=92, max=1633, avg=609.41, stdev=180.24 lat (msec): min=92, max=1634, avg=609.69, stdev=180.27 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 363], 10.00th=[ 405], 20.00th=[ 464], | 30.00th=[ 510], 40.00th=[ 550], 50.00th=[ 592], 60.00th=[ 634], | 70.00th=[ 676], 80.00th=[ 751], 90.00th=[ 852], 95.00th=[ 919], | 99.00th=[ 1003], 99.50th=[ 1318], 99.90th=[ 1636], 99.95th=[ 1636], | 99.99th=[ 1636] bw ( KiB/s): min= 24, max= 127, per=24.75%, avg=69.53, stdev=21.98, samples=146 iops : min= 6, max= 31, avg=16.84, stdev= 5.50, samples=146 lat (msec) : 10=0.04%, 20=0.04%, 50=0.20%, 100=0.16%, 250=13.32% lat (msec) : 500=48.67%, 750=27.50%, 1000=9.45%, 2000=0.62% cpu : usr=0.11%, sys=0.92%, ctx=2926, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=271KiB/s (277kB/s), 67.1KiB/s-68.6KiB/s (68.7kB/s-70.3kB/s), io=19.6MiB (20.5MB), run=73729-74168msec WRITE: bw=282KiB/s (288kB/s), 70.2KiB/s-70.0KiB/s (71.9kB/s-72.7kB/s), io=20.4MiB (21.4MB), run=73729-74168msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=123659: Fri Oct 24 11:08:01 2025 read: IOPS=3, BW=152MiB/s (159MB/s)(40.0MiB/264msec) slat (nsec): min=17511k, max=17511k, avg=17510807.00, stdev= 0.00 clat (nsec): min=235737k, max=235737k, avg=235736551.00, stdev= 0.00 lat (nsec): min=253251k, max=253251k, avg=253251485.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 236], 5.00th=[ 236], 10.00th=[ 236], 20.00th=[ 236], | 30.00th=[ 236], 40.00th=[ 236], 50.00th=[ 236], 60.00th=[ 236], | 70.00th=[ 236], 80.00th=[ 236], 90.00th=[ 236], 95.00th=[ 236], | 99.00th=[ 236], 99.50th=[ 236], 99.90th=[ 236], 99.95th=[ 236], | 99.99th=[ 236] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.08%, ctx=2, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=152MiB/s (159MB/s), 152MiB/s-152MiB/s (159MB/s-159MB/s), io=40.0MiB (41.9MB), run=264-264msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck