== sanity test 398c: run fio to test AIO ================= 15:09:59 (1787857799) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.572858 s, 73.2 MB/s osc.lustre-OST0000-osc-ffff8a97c4fa0800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=144922: Thu Aug 27 15:13:26 2026 write: IOPS=13, BW=52.6KiB/s (53.9kB/s)(10.0MiB/194669msec); 0 zone resets slat (usec): min=22, max=27662, avg=232.46, stdev=1074.37 clat (msec): min=195, max=3173, avg=1215.69, stdev=407.19 lat (msec): min=195, max=3173, avg=1215.92, stdev=407.19 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 701], 10.00th=[ 768], 20.00th=[ 885], | 30.00th=[ 1003], 40.00th=[ 1083], 50.00th=[ 1167], 60.00th=[ 1250], | 70.00th=[ 1368], 80.00th=[ 1519], 90.00th=[ 1720], 95.00th=[ 1854], | 99.00th=[ 2735], 99.50th=[ 2869], 99.90th=[ 3037], 99.95th=[ 3171], | 99.99th=[ 3171] bw ( KiB/s): min= 7, max= 225, per=25.35%, avg=53.23, stdev=28.34, samples=377 iops : min= 1, max= 56, avg=12.81, stdev= 7.11, samples=377 lat (msec) : 250=0.62%, 500=1.25%, 750=7.11%, 1000=20.98%, 2000=66.72% lat (msec) : >=2000=3.32% cpu : usr=0.05%, sys=0.17%, ctx=1498, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144923: Thu Aug 27 15:13:26 2026 write: IOPS=13, BW=52.6KiB/s (53.8kB/s)(10.0MiB/194804msec); 0 zone resets slat (usec): min=22, max=12510, avg=220.01, stdev=789.53 clat (msec): min=157, max=3084, avg=1216.18, stdev=394.16 lat (msec): min=157, max=3085, avg=1216.40, stdev=394.12 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 693], 10.00th=[ 785], 20.00th=[ 902], | 30.00th=[ 995], 40.00th=[ 1083], 50.00th=[ 1167], 60.00th=[ 1250], | 70.00th=[ 1351], 80.00th=[ 1536], 90.00th=[ 1720], 95.00th=[ 1854], | 99.00th=[ 2668], 99.50th=[ 2802], 99.90th=[ 2970], 99.95th=[ 2970], | 99.99th=[ 3071] bw ( KiB/s): min= 7, max= 191, per=25.24%, avg=52.99, stdev=27.28, samples=379 iops : min= 1, max= 47, avg=12.76, stdev= 6.83, samples=379 lat (msec) : 250=0.08%, 500=1.52%, 750=5.98%, 1000=22.66%, 2000=67.03% lat (msec) : >=2000=2.73% cpu : usr=0.04%, sys=0.20%, ctx=1691, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144924: Thu Aug 27 15:13:26 2026 write: IOPS=13, BW=52.6KiB/s (53.9kB/s)(10.0MiB/194663msec); 0 zone resets slat (usec): min=22, max=20865, avg=245.51, stdev=1011.67 clat (msec): min=286, max=3019, avg=1215.23, stdev=385.46 lat (msec): min=286, max=3019, avg=1215.48, stdev=385.44 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 718], 10.00th=[ 776], 20.00th=[ 911], | 30.00th=[ 1003], 40.00th=[ 1083], 50.00th=[ 1167], 60.00th=[ 1250], | 70.00th=[ 1351], 80.00th=[ 1519], 90.00th=[ 1687], 95.00th=[ 1838], | 99.00th=[ 2635], 99.50th=[ 2802], 99.90th=[ 2903], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 128, per=25.30%, avg=53.13, stdev=26.82, samples=378 iops : min= 1, max= 32, avg=12.78, stdev= 6.68, samples=378 lat (msec) : 500=1.33%, 750=6.76%, 1000=21.33%, 2000=67.97%, >=2000=2.62% cpu : usr=0.03%, sys=0.19%, ctx=1540, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=144925: Thu Aug 27 15:13:26 2026 write: IOPS=13, BW=52.6KiB/s (53.9kB/s)(10.0MiB/194662msec); 0 zone resets slat (usec): min=22, max=20419, avg=286.13, stdev=1278.18 clat (msec): min=79, max=3080, avg=1214.81, stdev=397.41 lat (msec): min=79, max=3080, avg=1215.10, stdev=397.39 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 693], 10.00th=[ 768], 20.00th=[ 902], | 30.00th=[ 1011], 40.00th=[ 1083], 50.00th=[ 1167], 60.00th=[ 1250], | 70.00th=[ 1351], 80.00th=[ 1519], 90.00th=[ 1703], 95.00th=[ 1871], | 99.00th=[ 2668], 99.50th=[ 2802], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min= 7, max= 143, per=25.78%, avg=54.14, stdev=28.34, samples=371 iops : min= 1, max= 35, avg=13.04, stdev= 7.07, samples=371 lat (msec) : 100=0.08%, 250=0.20%, 500=1.37%, 750=7.38%, 1000=19.84% lat (msec) : 2000=68.44%, >=2000=2.70% cpu : usr=0.04%, sys=0.19%, ctx=1831, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=210KiB/s (215kB/s), 52.6KiB/s-52.6KiB/s (53.8kB/s-53.9kB/s), io=40.0MiB (41.9MB), run=194662-194804msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=144962: Thu Aug 27 15:15:43 2026 read: IOPS=9, BW=39.4KiB/s (40.3kB/s)(5048KiB/128231msec) slat (usec): min=2618, max=80850, avg=16764.67, stdev=9993.47 clat (msec): min=18, max=1153, avg=497.42, stdev=158.45 lat (msec): min=23, max=1184, avg=514.19, stdev=160.62 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 288], 10.00th=[ 330], 20.00th=[ 376], | 30.00th=[ 409], 40.00th=[ 443], 50.00th=[ 477], 60.00th=[ 506], | 70.00th=[ 550], 80.00th=[ 609], 90.00th=[ 693], 95.00th=[ 802], | 99.00th=[ 1003], 99.50th=[ 1053], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 103, per=25.16%, avg=39.25, stdev=20.28, samples=251 iops : min= 1, max= 25, avg= 9.31, stdev= 5.09, samples=251 write: IOPS=10, BW=40.5KiB/s (41.5kB/s)(5192KiB/128231msec); 0 zone resets slat (usec): min=25, max=23034, avg=288.94, stdev=1152.55 clat (msec): min=304, max=2394, avg=1076.78, stdev=352.54 lat (msec): min=305, max=2394, avg=1077.07, stdev=352.51 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 600], 10.00th=[ 684], 20.00th=[ 785], | 30.00th=[ 852], 40.00th=[ 936], 50.00th=[ 1011], 60.00th=[ 1116], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1552], 95.00th=[ 1787], | 99.00th=[ 2106], 99.50th=[ 2232], 99.90th=[ 2333], 99.95th=[ 2400], | 99.99th=[ 2400] bw ( KiB/s): min= 7, max= 104, per=24.74%, avg=40.08, stdev=18.54, samples=252 iops : min= 1, max= 26, avg= 9.52, stdev= 4.65, samples=252 lat (msec) : 20=0.04%, 50=0.08%, 100=0.08%, 250=0.86%, 500=28.28% lat (msec) : 750=25.04%, 1000=18.71%, 2000=26.02%, >=2000=0.90% cpu : usr=0.08%, sys=0.54%, ctx=4047, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144963: Thu Aug 27 15:15:43 2026 read: IOPS=9, BW=38.9KiB/s (39.8kB/s)(4976KiB/128023msec) slat (usec): min=51, max=66185, avg=16844.78, stdev=9993.75 clat (msec): min=23, max=1152, avg=503.66, stdev=169.77 lat (msec): min=27, max=1166, avg=520.51, stdev=171.72 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 275], 10.00th=[ 313], 20.00th=[ 372], | 30.00th=[ 414], 40.00th=[ 447], 50.00th=[ 481], 60.00th=[ 523], | 70.00th=[ 567], 80.00th=[ 625], 90.00th=[ 718], 95.00th=[ 818], | 99.00th=[ 995], 99.50th=[ 1083], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 7, max= 103, per=24.66%, avg=38.47, stdev=20.31, samples=252 iops : min= 1, max= 25, avg= 9.12, stdev= 5.09, samples=252 write: IOPS=10, BW=41.1KiB/s (42.1kB/s)(5264KiB/128023msec); 0 zone resets slat (usec): min=29, max=24546, avg=303.58, stdev=1113.16 clat (msec): min=172, max=2297, avg=1063.33, stdev=352.04 lat (msec): min=172, max=2297, avg=1063.64, stdev=352.05 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 575], 10.00th=[ 667], 20.00th=[ 768], | 30.00th=[ 852], 40.00th=[ 927], 50.00th=[ 1020], 60.00th=[ 1116], | 70.00th=[ 1217], 80.00th=[ 1334], 90.00th=[ 1519], 95.00th=[ 1770], | 99.00th=[ 2022], 99.50th=[ 2232], 99.90th=[ 2299], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 96, per=25.30%, avg=40.99, stdev=19.06, samples=251 iops : min= 1, max= 24, avg= 9.75, stdev= 4.78, samples=251 lat (msec) : 50=0.20%, 100=0.16%, 250=1.29%, 500=26.17%, 750=26.02% lat (msec) : 1000=18.91%, 2000=26.64%, >=2000=0.62% cpu : usr=0.09%, sys=0.59%, ctx=4340, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144964: Thu Aug 27 15:15:43 2026 read: IOPS=9, BW=38.9KiB/s (39.8kB/s)(4984KiB/128234msec) slat (usec): min=2669, max=80912, avg=16958.03, stdev=10223.77 clat (msec): min=35, max=1220, avg=499.54, stdev=167.01 lat (msec): min=42, max=1231, avg=516.51, stdev=168.97 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 275], 10.00th=[ 317], 20.00th=[ 368], | 30.00th=[ 409], 40.00th=[ 443], 50.00th=[ 477], 60.00th=[ 518], | 70.00th=[ 558], 80.00th=[ 617], 90.00th=[ 709], 95.00th=[ 827], | 99.00th=[ 995], 99.50th=[ 1083], 99.90th=[ 1133], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 135, per=24.82%, avg=38.73, stdev=20.74, samples=252 iops : min= 1, max= 33, avg= 9.18, stdev= 5.22, samples=252 write: IOPS=10, BW=40.0KiB/s (41.0kB/s)(5256KiB/128234msec); 0 zone resets slat (usec): min=25, max=27154, avg=315.33, stdev=1256.61 clat (msec): min=352, max=2438, avg=1068.29, stdev=350.61 lat (msec): min=352, max=2438, avg=1068.61, stdev=350.59 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 584], 10.00th=[ 659], 20.00th=[ 760], | 30.00th=[ 852], 40.00th=[ 936], 50.00th=[ 1020], 60.00th=[ 1116], | 70.00th=[ 1217], 80.00th=[ 1334], 90.00th=[ 1519], 95.00th=[ 1754], | 99.00th=[ 2106], 99.50th=[ 2265], 99.90th=[ 2299], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 95, per=25.61%, avg=41.49, stdev=19.60, samples=246 iops : min= 1, max= 23, avg= 9.87, stdev= 4.91, samples=246 lat (msec) : 50=0.12%, 100=0.16%, 250=1.21%, 500=27.15%, 750=26.29% lat (msec) : 1000=17.54%, 2000=26.91%, >=2000=0.62% cpu : usr=0.10%, sys=0.57%, ctx=4408, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=144965: Thu Aug 27 15:15:43 2026 read: IOPS=9, BW=39.7KiB/s (40.7kB/s)(5060KiB/127428msec) slat (usec): min=3261, max=81186, avg=16461.94, stdev=9882.33 clat (msec): min=22, max=1160, avg=497.01, stdev=164.58 lat (msec): min=32, max=1183, avg=513.48, stdev=166.91 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 275], 10.00th=[ 313], 20.00th=[ 368], | 30.00th=[ 405], 40.00th=[ 443], 50.00th=[ 481], 60.00th=[ 514], | 70.00th=[ 550], 80.00th=[ 609], 90.00th=[ 701], 95.00th=[ 793], | 99.00th=[ 1011], 99.50th=[ 1116], 99.90th=[ 1150], 99.95th=[ 1167], | 99.99th=[ 1167] bw ( KiB/s): min= 7, max= 112, per=25.91%, avg=40.42, stdev=19.70, samples=245 iops : min= 1, max= 28, avg= 9.60, stdev= 4.96, samples=245 write: IOPS=10, BW=40.7KiB/s (41.6kB/s)(5180KiB/127428msec); 0 zone resets slat (usec): min=31, max=21260, avg=324.62, stdev=1264.98 clat (msec): min=354, max=2284, avg=1069.71, stdev=353.35 lat (msec): min=354, max=2284, avg=1070.03, stdev=353.35 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 584], 10.00th=[ 676], 20.00th=[ 768], | 30.00th=[ 852], 40.00th=[ 927], 50.00th=[ 1011], 60.00th=[ 1099], | 70.00th=[ 1200], 80.00th=[ 1351], 90.00th=[ 1569], 95.00th=[ 1754], | 99.00th=[ 2072], 99.50th=[ 2165], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 102, per=25.45%, avg=41.23, stdev=18.91, samples=245 iops : min= 1, max= 25, avg= 9.81, stdev= 4.74, samples=245 lat (msec) : 50=0.16%, 100=0.20%, 250=1.05%, 500=26.95%, 750=27.03% lat (msec) : 1000=18.44%, 2000=25.27%, >=2000=0.90% cpu : usr=0.10%, sys=0.60%, ctx=4166, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=156KiB/s (160kB/s), 38.9KiB/s-39.7KiB/s (39.8kB/s-40.7kB/s), io=19.6MiB (20.5MB), run=127428-128234msec WRITE: bw=163KiB/s (167kB/s), 40.5KiB/s-41.1KiB/s (41.5kB/s-42.1kB/s), io=20.4MiB (21.4MB), run=127428-128234msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=144989: Thu Aug 27 15:15:55 2026 read: IOPS=1, BW=65.4MiB/s (68.5MB/s)(40.0MiB/612msec) slat (nsec): min=50370k, max=50370k, avg=50369726.00, stdev= 0.00 clat (nsec): min=543783k, max=543783k, avg=543783390.00, stdev= 0.00 lat (nsec): min=594159k, max=594159k, avg=594158993.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 542], 10.00th=[ 542], 20.00th=[ 542], | 30.00th=[ 542], 40.00th=[ 542], 50.00th=[ 542], 60.00th=[ 542], | 70.00th=[ 542], 80.00th=[ 542], 90.00th=[ 542], 95.00th=[ 542], | 99.00th=[ 542], 99.50th=[ 542], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 542] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.13%, ctx=43, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=65.4MiB/s (68.5MB/s), 65.4MiB/s-65.4MiB/s (68.5MB/s-68.5MB/s), io=40.0MiB (41.9MB), run=612-612msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout