== sanity test 398c: run fio to test AIO ================= 03:15:03 (1761290103) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.260473 s, 161 MB/s osc.lustre-OST0000-osc-ffff8acf42e83000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=183834: Fri Oct 24 03:15:31 2025 write: IOPS=115, BW=460KiB/s (471kB/s)(10.0MiB/22247msec); 0 zone resets slat (usec): min=27, max=7830, avg=160.59, stdev=381.15 clat (msec): min=16, max=271, avg=138.69, stdev=29.78 lat (msec): min=18, max=272, avg=138.85, stdev=29.78 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 96], 10.00th=[ 108], 20.00th=[ 116], | 30.00th=[ 125], 40.00th=[ 130], 50.00th=[ 138], 60.00th=[ 144], | 70.00th=[ 153], 80.00th=[ 163], 90.00th=[ 178], 95.00th=[ 188], | 99.00th=[ 220], 99.50th=[ 236], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 302, max= 633, per=25.02%, avg=457.43, stdev=71.82, samples=44 iops : min= 75, max= 158, avg=114.00, stdev=18.01, samples=44 lat (msec) : 20=0.04%, 50=0.94%, 100=5.59%, 250=93.32%, 500=0.12% cpu : usr=0.22%, sys=1.12%, ctx=2712, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183835: Fri Oct 24 03:15:31 2025 write: IOPS=114, BW=457KiB/s (468kB/s)(10.0MiB/22407msec); 0 zone resets slat (usec): min=25, max=8613, avg=168.16, stdev=446.12 clat (msec): min=17, max=237, avg=139.33, stdev=26.16 lat (msec): min=18, max=238, avg=139.50, stdev=26.17 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 102], 10.00th=[ 110], 20.00th=[ 120], | 30.00th=[ 125], 40.00th=[ 130], 50.00th=[ 138], 60.00th=[ 144], | 70.00th=[ 153], 80.00th=[ 161], 90.00th=[ 176], 95.00th=[ 186], | 99.00th=[ 209], 99.50th=[ 222], 99.90th=[ 226], 99.95th=[ 226], | 99.99th=[ 239] bw ( KiB/s): min= 328, max= 636, per=24.92%, avg=455.48, stdev=65.66, samples=44 iops : min= 82, max= 159, avg=113.52, stdev=16.47, samples=44 lat (msec) : 20=0.04%, 50=0.08%, 100=4.77%, 250=95.12% cpu : usr=0.18%, sys=1.10%, ctx=2630, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183836: Fri Oct 24 03:15:31 2025 write: IOPS=114, BW=457KiB/s (468kB/s)(10.0MiB/22407msec); 0 zone resets slat (usec): min=26, max=5536, avg=160.45, stdev=389.59 clat (msec): min=15, max=231, avg=139.38, stdev=26.44 lat (msec): min=15, max=231, avg=139.54, stdev=26.44 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 102], 10.00th=[ 110], 20.00th=[ 120], | 30.00th=[ 126], 40.00th=[ 129], 50.00th=[ 136], 60.00th=[ 144], | 70.00th=[ 153], 80.00th=[ 163], 90.00th=[ 176], 95.00th=[ 186], | 99.00th=[ 211], 99.50th=[ 220], 99.90th=[ 230], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 280, max= 629, per=24.96%, avg=456.32, stdev=71.05, samples=44 iops : min= 70, max= 157, avg=113.75, stdev=17.77, samples=44 lat (msec) : 20=0.04%, 50=0.08%, 100=4.77%, 250=95.12% cpu : usr=0.12%, sys=1.37%, ctx=2820, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=183837: Fri Oct 24 03:15:31 2025 write: IOPS=114, BW=458KiB/s (469kB/s)(10.0MiB/22353msec); 0 zone resets slat (usec): min=29, max=6817, avg=169.45, stdev=402.50 clat (msec): min=15, max=278, avg=139.08, stdev=28.23 lat (msec): min=16, max=278, avg=139.25, stdev=28.22 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 97], 10.00th=[ 109], 20.00th=[ 118], | 30.00th=[ 125], 40.00th=[ 130], 50.00th=[ 138], 60.00th=[ 144], | 70.00th=[ 153], 80.00th=[ 161], 90.00th=[ 176], 95.00th=[ 188], | 99.00th=[ 209], 99.50th=[ 220], 99.90th=[ 239], 99.95th=[ 239], | 99.99th=[ 279] bw ( KiB/s): min= 327, max= 573, per=25.05%, avg=457.86, stdev=64.82, samples=44 iops : min= 81, max= 143, avg=114.14, stdev=16.25, samples=44 lat (msec) : 20=0.08%, 50=0.31%, 100=5.82%, 250=93.75%, 500=0.04% cpu : usr=0.23%, sys=1.22%, ctx=2972, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1828KiB/s (1872kB/s), 457KiB/s-460KiB/s (468kB/s-471kB/s), io=40.0MiB (41.9MB), run=22247-22407msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=183850: Fri Oct 24 03:15:59 2025 read: IOPS=57, BW=229KiB/s (235kB/s)(5048KiB/22020msec) slat (usec): min=40, max=120739, avg=9543.66, stdev=6816.66 clat (msec): min=7, max=506, avg=129.76, stdev=43.38 lat (msec): min=12, max=515, avg=139.31, stdev=44.78 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 85], 10.00th=[ 93], 20.00th=[ 105], | 30.00th=[ 113], 40.00th=[ 120], 50.00th=[ 128], 60.00th=[ 134], | 70.00th=[ 142], 80.00th=[ 150], 90.00th=[ 165], 95.00th=[ 178], | 99.00th=[ 236], 99.50th=[ 477], 99.90th=[ 502], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 47, max= 317, per=24.80%, avg=225.93, stdev=55.52, samples=43 iops : min= 11, max= 79, avg=55.95, stdev=13.89, samples=43 write: IOPS=58, BW=236KiB/s (241kB/s)(5192KiB/22020msec); 0 zone resets slat (usec): min=28, max=7447, avg=166.14, stdev=396.17 clat (msec): min=21, max=493, avg=133.84, stdev=34.59 lat (msec): min=21, max=493, avg=134.01, stdev=34.59 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 91], 10.00th=[ 99], 20.00th=[ 110], | 30.00th=[ 117], 40.00th=[ 126], 50.00th=[ 131], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 167], 95.00th=[ 184], | 99.00th=[ 230], 99.50th=[ 268], 99.90th=[ 493], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 87, max= 335, per=24.48%, avg=232.02, stdev=43.98, samples=43 iops : min= 21, max= 83, avg=57.49, stdev=10.98, samples=43 lat (msec) : 10=0.12%, 20=0.12%, 50=0.82%, 100=12.50%, 250=85.66% lat (msec) : 500=0.70%, 750=0.08% cpu : usr=0.29%, sys=2.03%, ctx=4200, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183851: Fri Oct 24 03:15:59 2025 read: IOPS=57, BW=229KiB/s (234kB/s)(4976KiB/21760msec) slat (usec): min=2577, max=43762, avg=9336.35, stdev=3921.97 clat (msec): min=11, max=279, avg=127.99, stdev=29.37 lat (msec): min=19, max=289, avg=137.33, stdev=30.08 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 87], 10.00th=[ 94], 20.00th=[ 104], | 30.00th=[ 112], 40.00th=[ 120], 50.00th=[ 127], 60.00th=[ 133], | 70.00th=[ 142], 80.00th=[ 150], 90.00th=[ 163], 95.00th=[ 174], | 99.00th=[ 232], 99.50th=[ 249], 99.90th=[ 271], 99.95th=[ 279], | 99.99th=[ 279] bw ( KiB/s): min= 135, max= 342, per=24.97%, avg=227.51, stdev=45.72, samples=43 iops : min= 33, max= 85, avg=56.33, stdev=11.44, samples=43 write: IOPS=60, BW=242KiB/s (248kB/s)(5264KiB/21760msec); 0 zone resets slat (usec): min=29, max=6947, avg=177.18, stdev=386.87 clat (msec): min=47, max=261, avg=133.18, stdev=28.90 lat (msec): min=48, max=261, avg=133.36, stdev=28.89 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 93], 10.00th=[ 101], 20.00th=[ 110], | 30.00th=[ 117], 40.00th=[ 124], 50.00th=[ 130], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 167], 95.00th=[ 186], | 99.00th=[ 232], 99.50th=[ 245], 99.90th=[ 251], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 151, max= 375, per=25.52%, avg=241.91, stdev=49.07, samples=43 iops : min= 37, max= 93, avg=59.95, stdev=12.28, samples=43 lat (msec) : 20=0.04%, 50=0.16%, 100=12.54%, 250=86.95%, 500=0.31% cpu : usr=0.22%, sys=2.20%, ctx=4365, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183852: Fri Oct 24 03:15:59 2025 read: IOPS=56, BW=228KiB/s (233kB/s)(4984KiB/21883msec) slat (usec): min=2536, max=50951, avg=9504.32, stdev=4276.81 clat (msec): min=29, max=264, avg=129.19, stdev=29.00 lat (msec): min=36, max=277, avg=138.71, stdev=30.01 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 88], 10.00th=[ 95], 20.00th=[ 105], | 30.00th=[ 113], 40.00th=[ 121], 50.00th=[ 129], 60.00th=[ 136], | 70.00th=[ 142], 80.00th=[ 150], 90.00th=[ 163], 95.00th=[ 178], | 99.00th=[ 224], 99.50th=[ 241], 99.90th=[ 264], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 127, max= 335, per=25.02%, avg=227.98, stdev=46.66, samples=43 iops : min= 31, max= 83, avg=56.44, stdev=11.65, samples=43 write: IOPS=60, BW=240KiB/s (246kB/s)(5256KiB/21883msec); 0 zone resets slat (usec): min=29, max=10589, avg=178.91, stdev=432.77 clat (msec): min=78, max=261, avg=132.96, stdev=27.72 lat (msec): min=78, max=261, avg=133.14, stdev=27.73 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 93], 10.00th=[ 101], 20.00th=[ 110], | 30.00th=[ 117], 40.00th=[ 125], 50.00th=[ 131], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 167], 95.00th=[ 178], | 99.00th=[ 224], 99.50th=[ 239], 99.90th=[ 257], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 132, max= 343, per=25.24%, avg=239.30, stdev=48.32, samples=43 iops : min= 33, max= 85, avg=59.30, stdev=12.04, samples=43 lat (msec) : 50=0.08%, 100=12.58%, 250=87.03%, 500=0.31% cpu : usr=0.32%, sys=1.92%, ctx=4533, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=183853: Fri Oct 24 03:15:59 2025 read: IOPS=58, BW=234KiB/s (239kB/s)(5060KiB/21667msec) slat (usec): min=37, max=29736, avg=9281.10, stdev=3850.59 clat (msec): min=18, max=251, avg=128.25, stdev=28.17 lat (msec): min=20, max=266, avg=137.53, stdev=28.90 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 89], 10.00th=[ 95], 20.00th=[ 106], | 30.00th=[ 112], 40.00th=[ 120], 50.00th=[ 127], 60.00th=[ 134], | 70.00th=[ 142], 80.00th=[ 150], 90.00th=[ 161], 95.00th=[ 178], | 99.00th=[ 224], 99.50th=[ 230], 99.90th=[ 239], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 148, max= 335, per=25.65%, avg=233.71, stdev=39.77, samples=42 iops : min= 37, max= 83, avg=57.90, stdev= 9.93, samples=42 write: IOPS=59, BW=239KiB/s (245kB/s)(5180KiB/21667msec); 0 zone resets slat (usec): min=36, max=11910, avg=194.18, stdev=522.55 clat (msec): min=48, max=266, avg=132.67, stdev=27.63 lat (msec): min=48, max=266, avg=132.87, stdev=27.63 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 93], 10.00th=[ 101], 20.00th=[ 111], | 30.00th=[ 118], 40.00th=[ 124], 50.00th=[ 131], 60.00th=[ 138], | 70.00th=[ 144], 80.00th=[ 155], 90.00th=[ 165], 95.00th=[ 178], | 99.00th=[ 236], 99.50th=[ 241], 99.90th=[ 264], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 151, max= 367, per=25.11%, avg=238.07, stdev=48.20, samples=42 iops : min= 37, max= 91, avg=58.98, stdev=12.00, samples=42 lat (msec) : 20=0.04%, 50=0.08%, 100=12.19%, 250=87.50%, 500=0.20% cpu : usr=0.28%, sys=2.10%, ctx=4633, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=911KiB/s (933kB/s), 228KiB/s-234KiB/s (233kB/s-239kB/s), io=19.6MiB (20.5MB), run=21667-22020msec WRITE: bw=949KiB/s (972kB/s), 236KiB/s-242KiB/s (241kB/s-248kB/s), io=20.4MiB (21.4MB), run=21667-22020msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=183860: Fri Oct 24 03:16:05 2025 read: IOPS=3, BW=159MiB/s (166MB/s)(40.0MiB/252msec) slat (nsec): min=17618k, max=17618k, avg=17618240.00, stdev= 0.00 clat (nsec): min=211543k, max=211543k, avg=211542918.00, stdev= 0.00 lat (nsec): min=229168k, max=229168k, avg=229168321.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 211], 10.00th=[ 211], 20.00th=[ 211], | 30.00th=[ 211], 40.00th=[ 211], 50.00th=[ 211], 60.00th=[ 211], | 70.00th=[ 211], 80.00th=[ 211], 90.00th=[ 211], 95.00th=[ 211], | 99.00th=[ 211], 99.50th=[ 211], 99.90th=[ 211], 99.95th=[ 211], | 99.99th=[ 211] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.58%, ctx=12, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=159MiB/s (166MB/s), 159MiB/s-159MiB/s (166MB/s-166MB/s), io=40.0MiB (41.9MB), run=252-252msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout