== sanity test 398c: run fio to test AIO ================= 06:21:12 (1787394072) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.467425 s, 89.7 MB/s osc.lustre-OST0000-osc-ffff8fdbc9bb9800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180449: Sat Aug 22 06:21:42 2026 write: IOPS=113, BW=453KiB/s (464kB/s)(10.0MiB/22583msec); 0 zone resets slat (usec): min=39, max=15593, avg=251.73, stdev=702.07 clat (msec): min=15, max=241, avg=140.06, stdev=35.59 lat (msec): min=15, max=241, avg=140.31, stdev=35.60 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 85], 10.00th=[ 97], 20.00th=[ 111], | 30.00th=[ 121], 40.00th=[ 130], 50.00th=[ 138], 60.00th=[ 146], | 70.00th=[ 157], 80.00th=[ 169], 90.00th=[ 188], 95.00th=[ 205], | 99.00th=[ 224], 99.50th=[ 232], 99.90th=[ 239], 99.95th=[ 241], | 99.99th=[ 241] bw ( KiB/s): min= 270, max= 674, per=25.04%, avg=452.27, stdev=95.36, samples=44 iops : min= 67, max= 168, avg=112.75, stdev=23.84, samples=44 lat (msec) : 20=0.12%, 50=0.16%, 100=10.90%, 250=88.83% cpu : usr=0.21%, sys=1.37%, ctx=2760, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180450: Sat Aug 22 06:21:42 2026 write: IOPS=112, BW=452KiB/s (463kB/s)(10.0MiB/22671msec); 0 zone resets slat (usec): min=39, max=18257, avg=222.30, stdev=642.06 clat (msec): min=14, max=246, avg=140.40, stdev=34.51 lat (msec): min=14, max=246, avg=140.63, stdev=34.51 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 88], 10.00th=[ 100], 20.00th=[ 113], | 30.00th=[ 122], 40.00th=[ 131], 50.00th=[ 138], 60.00th=[ 146], | 70.00th=[ 157], 80.00th=[ 171], 90.00th=[ 188], 95.00th=[ 205], | 99.00th=[ 228], 99.50th=[ 243], 99.90th=[ 247], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 215, max= 642, per=24.85%, avg=448.75, stdev=94.69, samples=44 iops : min= 53, max= 160, avg=111.86, stdev=23.75, samples=44 lat (msec) : 20=0.04%, 50=0.20%, 100=10.82%, 250=88.95% cpu : usr=0.11%, sys=1.32%, ctx=2603, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180451: Sat Aug 22 06:21:42 2026 write: IOPS=113, BW=452KiB/s (463kB/s)(10.0MiB/22647msec); 0 zone resets slat (usec): min=37, max=14611, avg=230.14, stdev=721.70 clat (msec): min=11, max=254, avg=140.40, stdev=33.95 lat (msec): min=12, max=254, avg=140.63, stdev=33.97 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 88], 10.00th=[ 100], 20.00th=[ 111], | 30.00th=[ 122], 40.00th=[ 129], 50.00th=[ 140], 60.00th=[ 146], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 184], 95.00th=[ 199], | 99.00th=[ 234], 99.50th=[ 251], 99.90th=[ 255], 99.95th=[ 255], | 99.99th=[ 255] bw ( KiB/s): min= 216, max= 753, per=24.93%, avg=450.27, stdev=94.17, samples=44 iops : min= 54, max= 188, avg=112.25, stdev=23.57, samples=44 lat (msec) : 20=0.08%, 50=0.04%, 100=12.07%, 250=87.30%, 500=0.51% cpu : usr=0.19%, sys=1.20%, ctx=2384, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180452: Sat Aug 22 06:21:42 2026 write: IOPS=113, BW=455KiB/s (466kB/s)(10.0MiB/22481msec); 0 zone resets slat (usec): min=35, max=12491, avg=245.97, stdev=635.98 clat (msec): min=13, max=250, avg=140.09, stdev=35.03 lat (msec): min=20, max=250, avg=140.34, stdev=35.03 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 88], 10.00th=[ 95], 20.00th=[ 111], | 30.00th=[ 121], 40.00th=[ 128], 50.00th=[ 138], 60.00th=[ 146], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 190], 95.00th=[ 201], | 99.00th=[ 224], 99.50th=[ 243], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 318, max= 690, per=25.13%, avg=453.93, stdev=94.08, samples=44 iops : min= 79, max= 172, avg=113.14, stdev=23.56, samples=44 lat (msec) : 20=0.04%, 50=0.23%, 100=11.68%, 250=87.97%, 500=0.08% cpu : usr=0.15%, sys=1.32%, ctx=2864, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1807KiB/s (1850kB/s), 452KiB/s-455KiB/s (463kB/s-466kB/s), io=40.0MiB (41.9MB), run=22481-22671msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180465: Sat Aug 22 06:22:16 2026 read: IOPS=47, BW=189KiB/s (193kB/s)(5048KiB/26720msec) slat (usec): min=89, max=58113, avg=11771.40, stdev=6961.30 clat (msec): min=11, max=336, avg=157.74, stdev=44.94 lat (msec): min=18, max=348, avg=169.52, stdev=46.85 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 93], 10.00th=[ 105], 20.00th=[ 122], | 30.00th=[ 133], 40.00th=[ 144], 50.00th=[ 155], 60.00th=[ 167], | 70.00th=[ 178], 80.00th=[ 190], 90.00th=[ 215], 95.00th=[ 236], | 99.00th=[ 292], 99.50th=[ 305], 99.90th=[ 330], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 79, max= 316, per=25.32%, avg=188.38, stdev=44.07, samples=52 iops : min= 19, max= 79, avg=46.65, stdev=11.09, samples=52 write: IOPS=48, BW=194KiB/s (199kB/s)(5192KiB/26720msec); 0 zone resets slat (usec): min=38, max=13455, avg=311.71, stdev=801.71 clat (msec): min=48, max=327, avg=163.70, stdev=43.71 lat (msec): min=48, max=327, avg=164.01, stdev=43.67 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 99], 10.00th=[ 110], 20.00th=[ 127], | 30.00th=[ 140], 40.00th=[ 150], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 222], 95.00th=[ 243], | 99.00th=[ 284], 99.50th=[ 288], 99.90th=[ 326], 99.95th=[ 326], | 99.99th=[ 326] bw ( KiB/s): min= 103, max= 358, per=25.13%, avg=194.73, stdev=48.75, samples=52 iops : min= 25, max= 89, avg=48.25, stdev=12.19, samples=52 lat (msec) : 20=0.08%, 50=0.16%, 100=6.52%, 250=89.49%, 500=3.75% cpu : usr=0.30%, sys=2.32%, ctx=4807, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180466: Sat Aug 22 06:22:16 2026 read: IOPS=46, BW=185KiB/s (189kB/s)(4976KiB/26933msec) slat (usec): min=2154, max=48552, avg=11842.10, stdev=6945.19 clat (msec): min=14, max=335, avg=157.78, stdev=41.84 lat (msec): min=21, max=347, avg=169.63, stdev=43.51 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 96], 10.00th=[ 110], 20.00th=[ 125], | 30.00th=[ 134], 40.00th=[ 144], 50.00th=[ 153], 60.00th=[ 163], | 70.00th=[ 176], 80.00th=[ 192], 90.00th=[ 213], 95.00th=[ 232], | 99.00th=[ 279], 99.50th=[ 292], 99.90th=[ 321], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 83, max= 311, per=24.70%, avg=183.75, stdev=46.48, samples=53 iops : min= 20, max= 77, avg=45.51, stdev=11.66, samples=53 write: IOPS=48, BW=195KiB/s (200kB/s)(5264KiB/26933msec); 0 zone resets slat (usec): min=38, max=16163, avg=275.17, stdev=742.65 clat (msec): min=45, max=336, avg=164.59, stdev=42.24 lat (msec): min=46, max=336, avg=164.87, stdev=42.27 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 103], 10.00th=[ 115], 20.00th=[ 130], | 30.00th=[ 142], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 171], | 70.00th=[ 182], 80.00th=[ 194], 90.00th=[ 215], 95.00th=[ 241], | 99.00th=[ 292], 99.50th=[ 300], 99.90th=[ 334], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 104, max= 332, per=25.19%, avg=195.19, stdev=49.60, samples=53 iops : min= 26, max= 83, avg=48.36, stdev=12.46, samples=53 lat (msec) : 20=0.04%, 50=0.08%, 100=5.08%, 250=91.56%, 500=3.24% cpu : usr=0.23%, sys=2.26%, ctx=4565, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180467: Sat Aug 22 06:22:16 2026 read: IOPS=46, BW=185KiB/s (189kB/s)(4984KiB/26957msec) slat (usec): min=2019, max=66311, avg=12086.76, stdev=7126.36 clat (msec): min=13, max=320, avg=158.48, stdev=43.11 lat (msec): min=16, max=336, avg=170.57, stdev=44.68 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 94], 10.00th=[ 105], 20.00th=[ 125], | 30.00th=[ 136], 40.00th=[ 144], 50.00th=[ 155], 60.00th=[ 167], | 70.00th=[ 178], 80.00th=[ 192], 90.00th=[ 211], 95.00th=[ 232], | 99.00th=[ 288], 99.50th=[ 296], 99.90th=[ 317], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 76, max= 334, per=24.73%, avg=184.02, stdev=49.39, samples=53 iops : min= 19, max= 83, avg=45.55, stdev=12.37, samples=53 write: IOPS=48, BW=195KiB/s (200kB/s)(5256KiB/26957msec); 0 zone resets slat (usec): min=46, max=7143, avg=254.88, stdev=508.64 clat (msec): min=23, max=330, avg=163.39, stdev=42.11 lat (msec): min=23, max=330, avg=163.65, stdev=42.13 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 101], 10.00th=[ 113], 20.00th=[ 128], | 30.00th=[ 142], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 169], | 70.00th=[ 182], 80.00th=[ 197], 90.00th=[ 215], 95.00th=[ 234], | 99.00th=[ 288], 99.50th=[ 305], 99.90th=[ 317], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 76, max= 311, per=24.92%, avg=193.09, stdev=49.70, samples=53 iops : min= 19, max= 77, avg=47.87, stdev=12.45, samples=53 lat (msec) : 20=0.04%, 50=0.20%, 100=5.86%, 250=90.66%, 500=3.24% cpu : usr=0.24%, sys=2.01%, ctx=4620, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180468: Sat Aug 22 06:22:16 2026 read: IOPS=46, BW=188KiB/s (192kB/s)(5060KiB/26918msec) slat (usec): min=2673, max=45915, avg=11844.76, stdev=6854.12 clat (msec): min=52, max=328, avg=157.32, stdev=44.76 lat (msec): min=65, max=358, avg=169.17, stdev=46.48 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 94], 10.00th=[ 106], 20.00th=[ 123], | 30.00th=[ 132], 40.00th=[ 142], 50.00th=[ 153], 60.00th=[ 163], | 70.00th=[ 174], 80.00th=[ 192], 90.00th=[ 215], 95.00th=[ 241], | 99.00th=[ 296], 99.50th=[ 309], 99.90th=[ 326], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 78, max= 304, per=25.25%, avg=187.83, stdev=49.46, samples=53 iops : min= 19, max= 76, avg=46.51, stdev=12.40, samples=53 write: IOPS=48, BW=192KiB/s (197kB/s)(5180KiB/26918msec); 0 zone resets slat (usec): min=40, max=8244, avg=270.73, stdev=591.52 clat (msec): min=74, max=333, avg=164.55, stdev=42.06 lat (msec): min=75, max=335, avg=164.82, stdev=42.06 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 100], 10.00th=[ 113], 20.00th=[ 130], | 30.00th=[ 144], 40.00th=[ 153], 50.00th=[ 161], 60.00th=[ 171], | 70.00th=[ 184], 80.00th=[ 199], 90.00th=[ 222], 95.00th=[ 236], | 99.00th=[ 279], 99.50th=[ 300], 99.90th=[ 326], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 92, max= 303, per=24.70%, avg=191.42, stdev=47.51, samples=53 iops : min= 23, max= 75, avg=47.42, stdev=11.91, samples=53 lat (msec) : 100=6.33%, 250=89.92%, 500=3.75% cpu : usr=0.25%, sys=2.22%, ctx=4659, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=744KiB/s (762kB/s), 185KiB/s-189KiB/s (189kB/s-193kB/s), io=19.6MiB (20.5MB), run=26720-26957msec WRITE: bw=775KiB/s (794kB/s), 192KiB/s-195KiB/s (197kB/s-200kB/s), io=20.4MiB (21.4MB), run=26720-26957msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180477: Sat Aug 22 06:22:26 2026 read: IOPS=2, BW=83.7MiB/s (87.7MB/s)(40.0MiB/478msec) slat (nsec): min=57336k, max=57336k, avg=57336183.00, stdev= 0.00 clat (nsec): min=405474k, max=405474k, avg=405473728.00, stdev= 0.00 lat (nsec): min=462814k, max=462814k, avg=462814447.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] lat (msec) : 500=100.00% cpu : usr=0.63%, sys=3.35%, ctx=12, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=83.7MiB/s (87.7MB/s), 83.7MiB/s-83.7MiB/s (87.7MB/s-87.7MB/s), io=40.0MiB (41.9MB), run=478-478msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout