== sanity test 398c: run fio to test AIO ================= 22:03:50 (1788919430) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.179868 s, 233 MB/s osc.lustre-OST0000-osc-ffff90ad9917c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=152108: Tue Sep 8 22:05:03 2026 write: IOPS=37, BW=148KiB/s (152kB/s)(10.0MiB/69161msec); 0 zone resets slat (usec): min=38, max=6449, avg=249.53, stdev=340.61 clat (msec): min=48, max=911, avg=431.77, stdev=101.87 lat (msec): min=48, max=911, avg=432.02, stdev=101.87 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 268], 10.00th=[ 313], 20.00th=[ 363], | 30.00th=[ 393], 40.00th=[ 418], 50.00th=[ 439], 60.00th=[ 460], | 70.00th=[ 481], 80.00th=[ 510], 90.00th=[ 550], 95.00th=[ 584], | 99.00th=[ 667], 99.50th=[ 693], 99.90th=[ 810], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 55, max= 414, per=24.96%, avg=147.49, stdev=40.76, samples=137 iops : min= 13, max= 103, avg=36.57, stdev=10.21, samples=137 lat (msec) : 50=0.62%, 100=0.62%, 250=3.40%, 500=72.66%, 750=22.46% lat (msec) : 1000=0.23% cpu : usr=0.06%, sys=0.55%, ctx=3073, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=152109: Tue Sep 8 22:05:03 2026 write: IOPS=36, BW=148KiB/s (151kB/s)(10.0MiB/69283msec); 0 zone resets slat (usec): min=37, max=4721, avg=264.28, stdev=348.76 clat (msec): min=85, max=775, avg=432.42, stdev=96.40 lat (msec): min=85, max=776, avg=432.69, stdev=96.41 clat percentiles (msec): | 1.00th=[ 142], 5.00th=[ 275], 10.00th=[ 317], 20.00th=[ 363], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 435], 60.00th=[ 460], | 70.00th=[ 481], 80.00th=[ 510], 90.00th=[ 550], 95.00th=[ 584], | 99.00th=[ 642], 99.50th=[ 693], 99.90th=[ 735], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 80, max= 376, per=24.79%, avg=146.49, stdev=37.68, samples=138 iops : min= 20, max= 94, avg=36.33, stdev= 9.45, samples=138 lat (msec) : 100=0.62%, 250=3.20%, 500=73.32%, 750=22.77%, 1000=0.08% cpu : usr=0.06%, sys=0.61%, ctx=3179, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=152110: Tue Sep 8 22:05:03 2026 write: IOPS=37, BW=148KiB/s (152kB/s)(10.0MiB/69160msec); 0 zone resets slat (usec): min=35, max=3386, avg=257.29, stdev=296.09 clat (msec): min=46, max=860, avg=431.67, stdev=98.41 lat (msec): min=47, max=860, avg=431.93, stdev=98.40 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 271], 10.00th=[ 317], 20.00th=[ 359], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 435], 60.00th=[ 460], | 70.00th=[ 481], 80.00th=[ 506], 90.00th=[ 550], 95.00th=[ 584], | 99.00th=[ 651], 99.50th=[ 701], 99.90th=[ 743], 99.95th=[ 810], | 99.99th=[ 860] bw ( KiB/s): min= 72, max= 388, per=24.95%, avg=147.47, stdev=37.18, samples=137 iops : min= 18, max= 97, avg=36.57, stdev= 9.31, samples=137 lat (msec) : 50=0.08%, 100=0.70%, 250=3.55%, 500=72.89%, 750=22.70% lat (msec) : 1000=0.08% cpu : usr=0.07%, sys=0.59%, ctx=3187, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=152111: Tue Sep 8 22:05:03 2026 write: IOPS=36, BW=148KiB/s (151kB/s)(10.0MiB/69283msec); 0 zone resets slat (usec): min=50, max=5588, avg=244.34, stdev=304.44 clat (msec): min=75, max=754, avg=432.42, stdev=96.11 lat (msec): min=75, max=754, avg=432.67, stdev=96.12 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 275], 10.00th=[ 321], 20.00th=[ 359], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 439], 60.00th=[ 460], | 70.00th=[ 485], 80.00th=[ 506], 90.00th=[ 550], 95.00th=[ 584], | 99.00th=[ 642], 99.50th=[ 693], 99.90th=[ 726], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 56, max= 363, per=24.79%, avg=146.51, stdev=35.77, samples=138 iops : min= 14, max= 90, avg=36.33, stdev= 8.93, samples=138 lat (msec) : 100=0.62%, 250=3.09%, 500=73.28%, 750=22.93%, 1000=0.08% cpu : usr=0.07%, sys=0.59%, ctx=3153, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=591KiB/s (605kB/s), 148KiB/s-148KiB/s (151kB/s-152kB/s), io=40.0MiB (41.9MB), run=69160-69283msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=152132: Tue Sep 8 22:05:47 2026 read: IOPS=31, BW=126KiB/s (129kB/s)(5048KiB/40193msec) slat (usec): min=1886, max=13750, avg=5088.26, stdev=1873.33 clat (msec): min=5, max=439, avg=156.94, stdev=36.90 lat (msec): min=9, max=445, avg=162.04, stdev=37.19 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 109], 10.00th=[ 117], 20.00th=[ 129], | 30.00th=[ 140], 40.00th=[ 146], 50.00th=[ 155], 60.00th=[ 163], | 70.00th=[ 171], 80.00th=[ 182], 90.00th=[ 199], 95.00th=[ 213], | 99.00th=[ 259], 99.50th=[ 271], 99.90th=[ 435], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 40, max= 240, per=25.17%, avg=125.59, stdev=40.34, samples=79 iops : min= 10, max= 60, avg=31.16, stdev=10.09, samples=79 write: IOPS=32, BW=129KiB/s (132kB/s)(5192KiB/40193msec); 0 zone resets slat (usec): min=40, max=5282, avg=263.11, stdev=288.39 clat (msec): min=92, max=613, avg=336.61, stdev=87.28 lat (msec): min=92, max=614, avg=336.87, stdev=87.29 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 213], 10.00th=[ 232], 20.00th=[ 262], | 30.00th=[ 288], 40.00th=[ 305], 50.00th=[ 326], 60.00th=[ 351], | 70.00th=[ 376], 80.00th=[ 414], 90.00th=[ 460], 95.00th=[ 502], | 99.00th=[ 558], 99.50th=[ 567], 99.90th=[ 600], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 63, max= 199, per=24.89%, avg=129.18, stdev=27.58, samples=79 iops : min= 15, max= 49, avg=32.05, stdev= 6.87, samples=79 lat (msec) : 10=0.04%, 20=0.04%, 50=0.35%, 100=0.82%, 250=55.51% lat (msec) : 500=40.74%, 750=2.50% cpu : usr=0.16%, sys=1.42%, ctx=5032, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152133: Tue Sep 8 22:05:47 2026 read: IOPS=30, BW=124KiB/s (127kB/s)(4976KiB/40192msec) slat (usec): min=2010, max=15476, avg=5123.28, stdev=1926.68 clat (msec): min=3, max=464, avg=160.02, stdev=37.37 lat (msec): min=8, max=469, avg=165.15, stdev=37.65 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 112], 10.00th=[ 121], 20.00th=[ 132], | 30.00th=[ 142], 40.00th=[ 150], 50.00th=[ 159], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 184], 90.00th=[ 201], 95.00th=[ 218], | 99.00th=[ 257], 99.50th=[ 292], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 48, max= 200, per=24.79%, avg=123.70, stdev=33.40, samples=79 iops : min= 12, max= 50, avg=30.68, stdev= 8.37, samples=79 write: IOPS=32, BW=131KiB/s (134kB/s)(5264KiB/40192msec); 0 zone resets slat (usec): min=60, max=6985, avg=278.67, stdev=371.98 clat (msec): min=67, max=617, avg=331.40, stdev=88.15 lat (msec): min=68, max=617, avg=331.68, stdev=88.16 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 211], 10.00th=[ 230], 20.00th=[ 259], | 30.00th=[ 279], 40.00th=[ 300], 50.00th=[ 321], 60.00th=[ 342], | 70.00th=[ 372], 80.00th=[ 405], 90.00th=[ 456], 95.00th=[ 498], | 99.00th=[ 550], 99.50th=[ 558], 99.90th=[ 584], 99.95th=[ 617], | 99.99th=[ 617] bw ( KiB/s): min= 71, max= 184, per=25.24%, avg=130.99, stdev=31.05, samples=79 iops : min= 17, max= 46, avg=32.51, stdev= 7.73, samples=79 lat (msec) : 4=0.04%, 20=0.04%, 50=0.20%, 100=0.82%, 250=55.74% lat (msec) : 500=40.82%, 750=2.34% cpu : usr=0.14%, sys=1.48%, ctx=4991, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152134: Tue Sep 8 22:05:47 2026 read: IOPS=31, BW=124KiB/s (127kB/s)(4984KiB/40154msec) slat (usec): min=55, max=12854, avg=5093.09, stdev=1903.89 clat (msec): min=6, max=522, avg=156.21, stdev=38.40 lat (msec): min=12, max=529, avg=161.31, stdev=38.72 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 109], 10.00th=[ 117], 20.00th=[ 131], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 157], 60.00th=[ 165], | 70.00th=[ 174], 80.00th=[ 182], 90.00th=[ 197], 95.00th=[ 213], | 99.00th=[ 247], 99.50th=[ 257], 99.90th=[ 518], 99.95th=[ 523], | 99.99th=[ 523] bw ( KiB/s): min= 48, max= 248, per=25.04%, avg=124.96, stdev=38.07, samples=79 iops : min= 12, max= 62, avg=31.00, stdev= 9.51, samples=79 write: IOPS=32, BW=131KiB/s (134kB/s)(5256KiB/40154msec); 0 zone resets slat (usec): min=57, max=3132, avg=260.13, stdev=272.85 clat (msec): min=58, max=656, avg=335.23, stdev=87.72 lat (msec): min=58, max=656, avg=335.49, stdev=87.73 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 218], 10.00th=[ 236], 20.00th=[ 268], | 30.00th=[ 284], 40.00th=[ 305], 50.00th=[ 326], 60.00th=[ 347], | 70.00th=[ 372], 80.00th=[ 405], 90.00th=[ 460], 95.00th=[ 502], | 99.00th=[ 558], 99.50th=[ 592], 99.90th=[ 617], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 72, max= 214, per=25.18%, avg=130.68, stdev=29.20, samples=79 iops : min= 18, max= 53, avg=32.43, stdev= 7.27, samples=79 lat (msec) : 10=0.04%, 20=0.23%, 50=0.51%, 100=1.05%, 250=54.06% lat (msec) : 500=41.45%, 750=2.66% cpu : usr=0.12%, sys=1.44%, ctx=4997, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=152135: Tue Sep 8 22:05:47 2026 read: IOPS=31, BW=126KiB/s (129kB/s)(5060KiB/40192msec) slat (usec): min=1882, max=15004, avg=5130.14, stdev=1807.68 clat (msec): min=9, max=371, avg=158.15, stdev=34.08 lat (msec): min=13, max=376, avg=163.29, stdev=34.40 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 110], 10.00th=[ 120], 20.00th=[ 132], | 30.00th=[ 140], 40.00th=[ 148], 50.00th=[ 157], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 186], 90.00th=[ 203], 95.00th=[ 213], | 99.00th=[ 241], 99.50th=[ 255], 99.90th=[ 292], 99.95th=[ 372], | 99.99th=[ 372] bw ( KiB/s): min= 48, max= 192, per=25.36%, avg=126.56, stdev=33.53, samples=79 iops : min= 12, max= 48, avg=31.41, stdev= 8.33, samples=79 write: IOPS=32, BW=129KiB/s (132kB/s)(5180KiB/40192msec); 0 zone resets slat (usec): min=53, max=3484, avg=281.09, stdev=288.47 clat (msec): min=111, max=598, avg=335.74, stdev=85.17 lat (msec): min=111, max=598, avg=336.02, stdev=85.17 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 218], 10.00th=[ 239], 20.00th=[ 266], | 30.00th=[ 288], 40.00th=[ 300], 50.00th=[ 321], 60.00th=[ 347], | 70.00th=[ 372], 80.00th=[ 409], 90.00th=[ 456], 95.00th=[ 506], | 99.00th=[ 558], 99.50th=[ 575], 99.90th=[ 600], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 55, max= 200, per=24.77%, avg=128.57, stdev=25.63, samples=79 iops : min= 13, max= 50, avg=31.90, stdev= 6.40, samples=79 lat (msec) : 10=0.04%, 20=0.16%, 50=0.16%, 100=0.78%, 250=54.92% lat (msec) : 500=41.13%, 750=2.81% cpu : usr=0.15%, sys=1.46%, ctx=5075, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=499KiB/s (511kB/s), 124KiB/s-126KiB/s (127kB/s-129kB/s), io=19.6MiB (20.5MB), run=40154-40193msec WRITE: bw=520KiB/s (532kB/s), 129KiB/s-131KiB/s (132kB/s-134kB/s), io=20.4MiB (21.4MB), run=40154-40193msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=152144: Tue Sep 8 22:05:52 2026 read: IOPS=4, BW=177MiB/s (186MB/s)(40.0MiB/226msec) slat (nsec): min=18620k, max=18620k, avg=18620036.00, stdev= 0.00 clat (nsec): min=196004k, max=196004k, avg=196003871.00, stdev= 0.00 lat (nsec): min=214629k, max=214629k, avg=214628896.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 197], 10.00th=[ 197], 20.00th=[ 197], | 30.00th=[ 197], 40.00th=[ 197], 50.00th=[ 197], 60.00th=[ 197], | 70.00th=[ 197], 80.00th=[ 197], 90.00th=[ 197], 95.00th=[ 197], | 99.00th=[ 197], 99.50th=[ 197], 99.90th=[ 197], 99.95th=[ 197], | 99.99th=[ 197] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.78%, ctx=25, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=177MiB/s (186MB/s), 177MiB/s-177MiB/s (186MB/s-186MB/s), io=40.0MiB (41.9MB), run=226-226msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout