== sanity test 398c: run fio to test AIO ================= 23:25:11 (1763353511) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.187594 s, 224 MB/s osc.lustre-OST0000-osc-ffff9f4aea8fb000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=147498: Sun Nov 16 23:26:02 2025 write: IOPS=54, BW=217KiB/s (222kB/s)(10.0MiB/47154msec); 0 zone resets slat (usec): min=31, max=1530, avg=133.99, stdev=87.71 clat (msec): min=35, max=680, avg=294.26, stdev=72.01 lat (msec): min=35, max=680, avg=294.39, stdev=72.01 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 197], 10.00th=[ 220], 20.00th=[ 236], | 30.00th=[ 255], 40.00th=[ 271], 50.00th=[ 288], 60.00th=[ 305], | 70.00th=[ 326], 80.00th=[ 347], 90.00th=[ 384], 95.00th=[ 414], | 99.00th=[ 523], 99.50th=[ 592], 99.90th=[ 667], 99.95th=[ 676], | 99.99th=[ 684] bw ( KiB/s): min= 95, max= 355, per=24.95%, avg=216.55, stdev=49.02, samples=93 iops : min= 23, max= 88, avg=53.95, stdev=12.27, samples=93 lat (msec) : 50=0.04%, 100=0.12%, 250=27.19%, 500=71.60%, 750=1.05% cpu : usr=0.10%, sys=0.63%, ctx=2945, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147499: Sun Nov 16 23:26:02 2025 write: IOPS=54, BW=218KiB/s (223kB/s)(10.0MiB/47075msec); 0 zone resets slat (usec): min=21, max=1472, avg=133.73, stdev=90.97 clat (msec): min=45, max=670, avg=293.91, stdev=74.75 lat (msec): min=45, max=670, avg=294.05, stdev=74.75 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 197], 10.00th=[ 218], 20.00th=[ 236], | 30.00th=[ 253], 40.00th=[ 271], 50.00th=[ 288], 60.00th=[ 305], | 70.00th=[ 321], 80.00th=[ 351], 90.00th=[ 388], 95.00th=[ 414], | 99.00th=[ 542], 99.50th=[ 617], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 87, max= 414, per=25.03%, avg=217.27, stdev=48.26, samples=93 iops : min= 21, max= 103, avg=54.13, stdev=12.12, samples=93 lat (msec) : 50=0.62%, 250=26.91%, 500=71.21%, 750=1.25% cpu : usr=0.12%, sys=0.62%, ctx=2956, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147500: Sun Nov 16 23:26:02 2025 write: IOPS=54, BW=217KiB/s (222kB/s)(10.0MiB/47137msec); 0 zone resets slat (usec): min=31, max=1505, avg=137.20, stdev=92.59 clat (msec): min=82, max=697, avg=294.17, stdev=73.40 lat (msec): min=82, max=697, avg=294.31, stdev=73.41 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 197], 10.00th=[ 215], 20.00th=[ 236], | 30.00th=[ 255], 40.00th=[ 271], 50.00th=[ 288], 60.00th=[ 305], | 70.00th=[ 321], 80.00th=[ 351], 90.00th=[ 388], 95.00th=[ 414], | 99.00th=[ 502], 99.50th=[ 617], 99.90th=[ 676], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 103, max= 414, per=24.96%, avg=216.67, stdev=49.02, samples=93 iops : min= 25, max= 103, avg=53.98, stdev=12.27, samples=93 lat (msec) : 100=0.62%, 250=26.05%, 500=72.30%, 750=1.02% cpu : usr=0.11%, sys=0.66%, ctx=2953, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=147501: Sun Nov 16 23:26:02 2025 write: IOPS=54, BW=217KiB/s (223kB/s)(10.0MiB/47119msec); 0 zone resets slat (usec): min=23, max=2340, avg=131.14, stdev=104.12 clat (msec): min=51, max=680, avg=294.08, stdev=73.64 lat (msec): min=52, max=680, avg=294.22, stdev=73.64 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 197], 10.00th=[ 215], 20.00th=[ 236], | 30.00th=[ 255], 40.00th=[ 275], 50.00th=[ 292], 60.00th=[ 305], | 70.00th=[ 321], 80.00th=[ 347], 90.00th=[ 388], 95.00th=[ 409], | 99.00th=[ 542], 99.50th=[ 617], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 684] bw ( KiB/s): min= 71, max= 414, per=24.99%, avg=216.92, stdev=49.99, samples=93 iops : min= 17, max= 103, avg=54.04, stdev=12.54, samples=93 lat (msec) : 100=0.66%, 250=26.25%, 500=71.95%, 750=1.13% cpu : usr=0.10%, sys=0.64%, ctx=2713, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=869KiB/s (889kB/s), 217KiB/s-218KiB/s (222kB/s-223kB/s), io=40.0MiB (41.9MB), run=47075-47154msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=147517: Sun Nov 16 23:26:36 2025 read: IOPS=40, BW=164KiB/s (168kB/s)(5048KiB/30842msec) slat (usec): min=1379, max=8272, avg=3619.81, stdev=1133.43 clat (msec): min=3, max=210, avg=120.33, stdev=26.57 lat (msec): min=7, max=217, avg=123.96, stdev=26.78 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 84], 10.00th=[ 92], 20.00th=[ 101], | 30.00th=[ 107], 40.00th=[ 114], 50.00th=[ 120], 60.00th=[ 127], | 70.00th=[ 133], 80.00th=[ 142], 90.00th=[ 155], 95.00th=[ 165], | 99.00th=[ 184], 99.50th=[ 190], 99.90th=[ 199], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 80, max= 280, per=25.22%, avg=163.92, stdev=44.23, samples=61 iops : min= 20, max= 70, avg=40.92, stdev=11.09, samples=61 write: IOPS=42, BW=168KiB/s (172kB/s)(5192KiB/30842msec); 0 zone resets slat (usec): min=30, max=1591, avg=134.97, stdev=89.92 clat (msec): min=78, max=435, avg=258.81, stdev=63.33 lat (msec): min=78, max=435, avg=258.94, stdev=63.34 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 150], 10.00th=[ 176], 20.00th=[ 207], | 30.00th=[ 224], 40.00th=[ 243], 50.00th=[ 259], 60.00th=[ 275], | 70.00th=[ 296], 80.00th=[ 317], 90.00th=[ 338], 95.00th=[ 363], | 99.00th=[ 401], 99.50th=[ 414], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 112, max= 280, per=24.77%, avg=167.70, stdev=32.54, samples=61 iops : min= 28, max= 70, avg=41.85, stdev= 8.19, samples=61 lat (msec) : 4=0.04%, 10=0.16%, 20=0.20%, 50=0.12%, 100=9.26% lat (msec) : 250=61.95%, 500=28.28% cpu : usr=0.25%, sys=1.40%, ctx=4725, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147518: Sun Nov 16 23:26:36 2025 read: IOPS=40, BW=161KiB/s (165kB/s)(4976KiB/30855msec) slat (usec): min=1388, max=8274, avg=3663.40, stdev=1094.90 clat (msec): min=5, max=207, avg=119.98, stdev=25.93 lat (msec): min=8, max=212, avg=123.65, stdev=26.11 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 83], 10.00th=[ 91], 20.00th=[ 101], | 30.00th=[ 107], 40.00th=[ 112], 50.00th=[ 120], 60.00th=[ 126], | 70.00th=[ 132], 80.00th=[ 140], 90.00th=[ 155], 95.00th=[ 165], | 99.00th=[ 184], 99.50th=[ 192], 99.90th=[ 205], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 64, max= 304, per=24.73%, avg=160.75, stdev=45.15, samples=61 iops : min= 16, max= 76, avg=40.11, stdev=11.26, samples=61 write: IOPS=42, BW=171KiB/s (175kB/s)(5264KiB/30855msec); 0 zone resets slat (usec): min=34, max=1313, avg=138.20, stdev=84.40 clat (msec): min=51, max=435, avg=257.62, stdev=65.20 lat (msec): min=51, max=435, avg=257.76, stdev=65.20 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 150], 10.00th=[ 176], 20.00th=[ 205], | 30.00th=[ 224], 40.00th=[ 243], 50.00th=[ 259], 60.00th=[ 275], | 70.00th=[ 292], 80.00th=[ 313], 90.00th=[ 338], 95.00th=[ 363], | 99.00th=[ 401], 99.50th=[ 418], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 79, max= 288, per=25.08%, avg=169.77, stdev=33.31, samples=61 iops : min= 19, max= 72, avg=42.38, stdev= 8.41, samples=61 lat (msec) : 10=0.04%, 20=0.08%, 50=0.35%, 100=9.96%, 250=61.21% lat (msec) : 500=28.36% cpu : usr=0.22%, sys=1.41%, ctx=4811, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147519: Sun Nov 16 23:26:36 2025 read: IOPS=40, BW=162KiB/s (166kB/s)(4984KiB/30826msec) slat (usec): min=32, max=8436, avg=3636.33, stdev=1147.73 clat (msec): min=4, max=210, avg=120.05, stdev=25.80 lat (msec): min=8, max=217, avg=123.69, stdev=26.03 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 84], 10.00th=[ 91], 20.00th=[ 101], | 30.00th=[ 107], 40.00th=[ 113], 50.00th=[ 120], 60.00th=[ 126], | 70.00th=[ 133], 80.00th=[ 142], 90.00th=[ 153], 95.00th=[ 163], | 99.00th=[ 184], 99.50th=[ 190], 99.90th=[ 207], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 80, max= 264, per=25.01%, avg=162.56, stdev=41.35, samples=61 iops : min= 20, max= 66, avg=40.57, stdev=10.37, samples=61 write: IOPS=42, BW=171KiB/s (175kB/s)(5256KiB/30826msec); 0 zone resets slat (usec): min=30, max=781, avg=134.17, stdev=70.27 clat (msec): min=38, max=433, avg=257.64, stdev=65.22 lat (msec): min=38, max=433, avg=257.77, stdev=65.22 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 155], 10.00th=[ 178], 20.00th=[ 207], | 30.00th=[ 226], 40.00th=[ 241], 50.00th=[ 255], 60.00th=[ 275], | 70.00th=[ 288], 80.00th=[ 313], 90.00th=[ 342], 95.00th=[ 368], | 99.00th=[ 405], 99.50th=[ 422], 99.90th=[ 430], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 103, max= 264, per=25.05%, avg=169.59, stdev=34.28, samples=61 iops : min= 25, max= 66, avg=42.33, stdev= 8.61, samples=61 lat (msec) : 10=0.08%, 20=0.12%, 50=0.47%, 100=9.61%, 250=61.91% lat (msec) : 500=27.81% cpu : usr=0.25%, sys=1.38%, ctx=4723, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=147520: Sun Nov 16 23:26:36 2025 read: IOPS=41, BW=164KiB/s (168kB/s)(5060KiB/30841msec) slat (usec): min=1403, max=8751, avg=3661.13, stdev=1171.87 clat (msec): min=4, max=220, avg=120.61, stdev=26.82 lat (msec): min=7, max=223, avg=124.27, stdev=27.06 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 83], 10.00th=[ 91], 20.00th=[ 101], | 30.00th=[ 106], 40.00th=[ 113], 50.00th=[ 118], 60.00th=[ 126], | 70.00th=[ 133], 80.00th=[ 142], 90.00th=[ 155], 95.00th=[ 167], | 99.00th=[ 197], 99.50th=[ 201], 99.90th=[ 211], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 88, max= 256, per=25.31%, avg=164.52, stdev=40.27, samples=61 iops : min= 22, max= 64, avg=41.07, stdev=10.10, samples=61 write: IOPS=41, BW=168KiB/s (172kB/s)(5180KiB/30841msec); 0 zone resets slat (usec): min=38, max=889, avg=133.03, stdev=70.01 clat (msec): min=77, max=439, avg=258.75, stdev=64.00 lat (msec): min=77, max=439, avg=258.89, stdev=64.01 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 153], 10.00th=[ 174], 20.00th=[ 207], | 30.00th=[ 226], 40.00th=[ 241], 50.00th=[ 259], 60.00th=[ 275], | 70.00th=[ 296], 80.00th=[ 313], 90.00th=[ 342], 95.00th=[ 363], | 99.00th=[ 405], 99.50th=[ 422], 99.90th=[ 426], 99.95th=[ 439], | 99.99th=[ 439] bw ( KiB/s): min= 79, max= 232, per=24.68%, avg=167.07, stdev=33.16, samples=61 iops : min= 19, max= 58, avg=41.70, stdev= 8.36, samples=61 lat (msec) : 10=0.12%, 20=0.12%, 50=0.16%, 100=9.49%, 250=62.46% lat (msec) : 500=27.66% cpu : usr=0.22%, sys=1.41%, ctx=4944, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=650KiB/s (666kB/s), 161KiB/s-164KiB/s (165kB/s-168kB/s), io=19.6MiB (20.5MB), run=30826-30855msec WRITE: bw=677KiB/s (693kB/s), 168KiB/s-171KiB/s (172kB/s-175kB/s), io=20.4MiB (21.4MB), run=30826-30855msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=147528: Sun Nov 16 23:26:40 2025 read: IOPS=8, BW=333MiB/s (350MB/s)(40.0MiB/120msec) slat (nsec): min=11193k, max=11193k, avg=11193228.00, stdev= 0.00 clat (nsec): min=103239k, max=103239k, avg=103238991.00, stdev= 0.00 lat (nsec): min=114437k, max=114437k, avg=114436616.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 104], 10.00th=[ 104], 20.00th=[ 104], | 30.00th=[ 104], 40.00th=[ 104], 50.00th=[ 104], 60.00th=[ 104], | 70.00th=[ 104], 80.00th=[ 104], 90.00th=[ 104], 95.00th=[ 104], | 99.00th=[ 104], 99.50th=[ 104], 99.90th=[ 104], 99.95th=[ 104], | 99.99th=[ 104] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.56%, ctx=43, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=333MiB/s (350MB/s), 333MiB/s-333MiB/s (350MB/s-350MB/s), io=40.0MiB (41.9MB), run=120-120msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout