== sanity test 398c: run fio to test AIO ================= 04:20:32 (1787127632) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.59226 s, 70.8 MB/s osc.lustre-OST0000-osc-ffff8c207007e000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145297: Wed Aug 19 04:23:06 2026 write: IOPS=17, BW=71.2KiB/s (72.9kB/s)(10.0MiB/143840msec); 0 zone resets slat (usec): min=49, max=16961, avg=306.85, stdev=856.89 clat (msec): min=95, max=1544, avg=898.02, stdev=212.67 lat (msec): min=99, max=1545, avg=898.33, stdev=212.65 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 592], 10.00th=[ 651], 20.00th=[ 735], | 30.00th=[ 776], 40.00th=[ 827], 50.00th=[ 877], 60.00th=[ 927], | 70.00th=[ 986], 80.00th=[ 1083], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1435], 99.50th=[ 1502], 99.90th=[ 1536], 99.95th=[ 1536], | 99.99th=[ 1552] bw ( KiB/s): min= 7, max= 140, per=25.08%, avg=70.99, stdev=27.45, samples=284 iops : min= 1, max= 35, avg=17.31, stdev= 6.88, samples=284 lat (msec) : 100=0.04%, 250=0.62%, 500=0.86%, 750=22.42%, 1000=47.73% lat (msec) : 2000=28.32% cpu : usr=0.05%, sys=0.33%, ctx=2194, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145298: Wed Aug 19 04:23:06 2026 write: IOPS=17, BW=70.9KiB/s (72.7kB/s)(10.0MiB/144327msec); 0 zone resets slat (usec): min=47, max=22439, avg=310.70, stdev=903.15 clat (msec): min=241, max=1604, avg=900.81, stdev=212.46 lat (msec): min=242, max=1605, avg=901.12, stdev=212.51 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 600], 10.00th=[ 651], 20.00th=[ 726], | 30.00th=[ 768], 40.00th=[ 827], 50.00th=[ 877], 60.00th=[ 936], | 70.00th=[ 995], 80.00th=[ 1070], 90.00th=[ 1200], 95.00th=[ 1301], | 99.00th=[ 1452], 99.50th=[ 1502], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 1603] bw ( KiB/s): min= 7, max= 128, per=25.18%, avg=71.27, stdev=34.39, samples=283 iops : min= 1, max= 32, avg=17.37, stdev= 8.62, samples=283 lat (msec) : 250=0.12%, 500=1.48%, 750=24.10%, 1000=45.94%, 2000=28.36% cpu : usr=0.06%, sys=0.30%, ctx=1907, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145299: Wed Aug 19 04:23:06 2026 write: IOPS=17, BW=71.1KiB/s (72.8kB/s)(10.0MiB/144088msec); 0 zone resets slat (usec): min=50, max=24278, avg=348.22, stdev=1024.41 clat (msec): min=222, max=1584, avg=899.42, stdev=211.29 lat (msec): min=222, max=1584, avg=899.77, stdev=211.33 clat percentiles (msec): | 1.00th=[ 502], 5.00th=[ 584], 10.00th=[ 642], 20.00th=[ 735], | 30.00th=[ 785], 40.00th=[ 835], 50.00th=[ 885], 60.00th=[ 919], | 70.00th=[ 986], 80.00th=[ 1062], 90.00th=[ 1183], 95.00th=[ 1284], | 99.00th=[ 1452], 99.50th=[ 1502], 99.90th=[ 1552], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 136, per=25.35%, avg=71.74, stdev=31.99, samples=281 iops : min= 1, max= 34, avg=17.49, stdev= 8.03, samples=281 lat (msec) : 250=0.04%, 500=0.90%, 750=22.66%, 1000=48.44%, 2000=27.97% cpu : usr=0.04%, sys=0.31%, ctx=1852, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145300: Wed Aug 19 04:23:06 2026 write: IOPS=17, BW=70.0KiB/s (72.7kB/s)(10.0MiB/144304msec); 0 zone resets slat (usec): min=51, max=11738, avg=334.76, stdev=905.43 clat (msec): min=306, max=1604, avg=900.69, stdev=208.17 lat (msec): min=307, max=1604, avg=901.03, stdev=208.22 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 592], 10.00th=[ 659], 20.00th=[ 735], | 30.00th=[ 785], 40.00th=[ 835], 50.00th=[ 877], 60.00th=[ 927], | 70.00th=[ 986], 80.00th=[ 1062], 90.00th=[ 1183], 95.00th=[ 1284], | 99.00th=[ 1469], 99.50th=[ 1519], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 1603] bw ( KiB/s): min= 7, max= 128, per=24.82%, avg=70.25, stdev=27.79, samples=287 iops : min= 1, max= 32, avg=17.13, stdev= 6.98, samples=287 lat (msec) : 500=1.33%, 750=21.29%, 1000=50.23%, 2000=27.15% cpu : usr=0.05%, sys=0.33%, ctx=2137, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=284KiB/s (291kB/s), 70.9KiB/s-71.2KiB/s (72.7kB/s-72.9kB/s), io=40.0MiB (41.9MB), run=143840-144327msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145331: Wed Aug 19 04:24:50 2026 read: IOPS=13, BW=52.3KiB/s (53.5kB/s)(5048KiB/96597msec) slat (usec): min=3180, max=45571, avg=12785.41, stdev=6378.02 clat (msec): min=8, max=678, avg=373.71, stdev=88.23 lat (msec): min=17, max=702, avg=386.50, stdev=89.35 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 247], 10.00th=[ 279], 20.00th=[ 305], | 30.00th=[ 330], 40.00th=[ 347], 50.00th=[ 368], 60.00th=[ 388], | 70.00th=[ 409], 80.00th=[ 439], 90.00th=[ 485], 95.00th=[ 527], | 99.00th=[ 625], 99.50th=[ 642], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 8, max= 119, per=25.01%, avg=51.76, stdev=22.78, samples=192 iops : min= 2, max= 29, avg=12.52, stdev= 5.76, samples=192 write: IOPS=13, BW=53.7KiB/s (55.0kB/s)(5192KiB/96597msec); 0 zone resets slat (usec): min=54, max=9523, avg=347.40, stdev=710.41 clat (msec): min=180, max=1471, avg=811.96, stdev=174.80 lat (msec): min=180, max=1471, avg=812.31, stdev=174.79 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 542], 10.00th=[ 592], 20.00th=[ 659], | 30.00th=[ 718], 40.00th=[ 768], 50.00th=[ 810], 60.00th=[ 852], | 70.00th=[ 902], 80.00th=[ 953], 90.00th=[ 1028], 95.00th=[ 1099], | 99.00th=[ 1234], 99.50th=[ 1267], 99.90th=[ 1469], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 15, max= 104, per=24.54%, avg=53.00, stdev=19.09, samples=192 iops : min= 3, max= 26, avg=12.83, stdev= 4.81, samples=192 lat (msec) : 10=0.04%, 20=0.04%, 50=0.08%, 100=0.16%, 250=2.50% lat (msec) : 500=43.87%, 750=20.94%, 1000=25.66%, 2000=6.72% cpu : usr=0.10%, sys=0.70%, ctx=4592, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145332: Wed Aug 19 04:24:50 2026 read: IOPS=12, BW=51.5KiB/s (52.7kB/s)(4976KiB/96670msec) slat (usec): min=2838, max=47285, avg=12344.38, stdev=6287.88 clat (msec): min=19, max=666, avg=377.85, stdev=89.01 lat (msec): min=25, max=693, avg=390.20, stdev=90.19 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 249], 10.00th=[ 275], 20.00th=[ 305], | 30.00th=[ 330], 40.00th=[ 351], 50.00th=[ 372], 60.00th=[ 393], | 70.00th=[ 418], 80.00th=[ 456], 90.00th=[ 498], 95.00th=[ 527], | 99.00th=[ 617], 99.50th=[ 625], 99.90th=[ 659], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 7, max= 104, per=24.51%, avg=50.74, stdev=21.63, samples=192 iops : min= 1, max= 26, avg=12.27, stdev= 5.48, samples=192 write: IOPS=13, BW=54.5KiB/s (55.8kB/s)(5264KiB/96670msec); 0 zone resets slat (usec): min=53, max=14545, avg=346.87, stdev=755.74 clat (msec): min=124, max=1474, avg=803.94, stdev=189.43 lat (msec): min=124, max=1474, avg=804.29, stdev=189.45 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 518], 10.00th=[ 575], 20.00th=[ 642], | 30.00th=[ 701], 40.00th=[ 751], 50.00th=[ 810], 60.00th=[ 860], | 70.00th=[ 902], 80.00th=[ 953], 90.00th=[ 1036], 95.00th=[ 1099], | 99.00th=[ 1267], 99.50th=[ 1351], 99.90th=[ 1469], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 119, per=24.81%, avg=53.59, stdev=20.21, samples=192 iops : min= 1, max= 29, avg=12.97, stdev= 5.08, samples=192 lat (msec) : 20=0.04%, 50=0.08%, 100=0.08%, 250=2.66%, 500=43.75% lat (msec) : 750=22.03%, 1000=24.34%, 2000=7.03% cpu : usr=0.09%, sys=0.71%, ctx=4722, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145333: Wed Aug 19 04:24:50 2026 read: IOPS=12, BW=51.7KiB/s (52.9kB/s)(4984KiB/96490msec) slat (usec): min=58, max=46121, avg=12538.28, stdev=6483.16 clat (msec): min=17, max=679, avg=367.91, stdev=92.85 lat (msec): min=25, max=689, avg=380.46, stdev=93.84 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 239], 10.00th=[ 271], 20.00th=[ 305], | 30.00th=[ 326], 40.00th=[ 347], 50.00th=[ 368], 60.00th=[ 388], | 70.00th=[ 409], 80.00th=[ 439], 90.00th=[ 481], 95.00th=[ 514], | 99.00th=[ 625], 99.50th=[ 634], 99.90th=[ 676], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 7, max= 128, per=24.89%, avg=51.51, stdev=24.03, samples=191 iops : min= 1, max= 32, avg=12.46, stdev= 6.08, samples=191 write: IOPS=13, BW=54.5KiB/s (55.8kB/s)(5256KiB/96490msec); 0 zone resets slat (usec): min=58, max=10957, avg=389.12, stdev=906.23 clat (msec): min=183, max=1479, avg=812.93, stdev=189.29 lat (msec): min=183, max=1487, avg=813.32, stdev=189.34 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 514], 10.00th=[ 592], 20.00th=[ 659], | 30.00th=[ 718], 40.00th=[ 768], 50.00th=[ 818], 60.00th=[ 860], | 70.00th=[ 902], 80.00th=[ 961], 90.00th=[ 1045], 95.00th=[ 1116], | 99.00th=[ 1284], 99.50th=[ 1452], 99.90th=[ 1469], 99.95th=[ 1485], | 99.99th=[ 1485] bw ( KiB/s): min= 7, max= 104, per=24.87%, avg=53.73, stdev=17.99, samples=191 iops : min= 1, max= 26, avg=13.01, stdev= 4.53, samples=191 lat (msec) : 20=0.04%, 50=0.31%, 100=0.55%, 250=2.66%, 500=44.41% lat (msec) : 750=19.57%, 1000=25.16%, 2000=7.30% cpu : usr=0.09%, sys=0.71%, ctx=4632, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145334: Wed Aug 19 04:24:50 2026 read: IOPS=13, BW=52.4KiB/s (53.7kB/s)(5060KiB/96497msec) slat (usec): min=3152, max=49184, avg=12792.25, stdev=6546.54 clat (msec): min=13, max=675, avg=377.25, stdev=88.30 lat (msec): min=17, max=693, avg=390.05, stdev=89.23 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 257], 10.00th=[ 279], 20.00th=[ 305], | 30.00th=[ 330], 40.00th=[ 351], 50.00th=[ 372], 60.00th=[ 393], | 70.00th=[ 418], 80.00th=[ 443], 90.00th=[ 489], 95.00th=[ 535], | 99.00th=[ 625], 99.50th=[ 634], 99.90th=[ 659], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 7, max= 104, per=25.25%, avg=52.27, stdev=20.49, samples=191 iops : min= 1, max= 26, avg=12.65, stdev= 5.16, samples=191 write: IOPS=13, BW=53.7KiB/s (54.0kB/s)(5180KiB/96497msec); 0 zone resets slat (usec): min=57, max=9151, avg=336.21, stdev=599.25 clat (msec): min=243, max=1427, avg=808.81, stdev=179.57 lat (msec): min=243, max=1433, avg=809.15, stdev=179.59 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 531], 10.00th=[ 584], 20.00th=[ 659], | 30.00th=[ 718], 40.00th=[ 768], 50.00th=[ 810], 60.00th=[ 852], | 70.00th=[ 894], 80.00th=[ 944], 90.00th=[ 1036], 95.00th=[ 1116], | 99.00th=[ 1250], 99.50th=[ 1301], 99.90th=[ 1385], 99.95th=[ 1435], | 99.99th=[ 1435] bw ( KiB/s): min= 15, max= 109, per=24.42%, avg=52.74, stdev=19.62, samples=192 iops : min= 3, max= 27, avg=12.77, stdev= 4.90, samples=192 lat (msec) : 20=0.04%, 50=0.08%, 100=0.23%, 250=1.88%, 500=45.00% lat (msec) : 750=20.51%, 1000=25.20%, 2000=7.07% cpu : usr=0.10%, sys=0.71%, ctx=4761, majf=0, minf=17 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=208KiB/s (213kB/s), 51.5KiB/s-52.4KiB/s (52.7kB/s-53.7kB/s), io=19.6MiB (20.5MB), run=96490-96670msec WRITE: bw=216KiB/s (221kB/s), 53.7KiB/s-54.5KiB/s (54.0kB/s-55.8kB/s), io=20.4MiB (21.4MB), run=96490-96670msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145353: Wed Aug 19 04:24:59 2026 read: IOPS=2, BW=85.8MiB/s (90.0MB/s)(40.0MiB/466msec) slat (nsec): min=61957k, max=61957k, avg=61956999.00, stdev= 0.00 clat (nsec): min=362180k, max=362180k, avg=362180075.00, stdev= 0.00 lat (nsec): min=424142k, max=424142k, avg=424141789.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 363], 40.00th=[ 363], 50.00th=[ 363], 60.00th=[ 363], | 70.00th=[ 363], 80.00th=[ 363], 90.00th=[ 363], 95.00th=[ 363], | 99.00th=[ 363], 99.50th=[ 363], 99.90th=[ 363], 99.95th=[ 363], | 99.99th=[ 363] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.23%, ctx=46, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=85.8MiB/s (90.0MB/s), 85.8MiB/s-85.8MiB/s (90.0MB/s-90.0MB/s), io=40.0MiB (41.9MB), run=466-466msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout