== sanity test 398c: run fio to test AIO ================= 17:09:48 (1764108588) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.602134 s, 69.7 MB/s osc.lustre-OST0000-osc-ffff8b17a9df7000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=156048: Tue Nov 25 17:11:42 2025 write: IOPS=24, BW=97.3KiB/s (99.6kB/s)(10.0MiB/105236msec); 0 zone resets slat (usec): min=27, max=13061, avg=138.45, stdev=351.99 clat (msec): min=169, max=1261, avg=656.76, stdev=165.60 lat (msec): min=169, max=1261, avg=656.90, stdev=165.58 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 426], 10.00th=[ 477], 20.00th=[ 531], | 30.00th=[ 567], 40.00th=[ 600], 50.00th=[ 634], 60.00th=[ 684], | 70.00th=[ 743], 80.00th=[ 793], 90.00th=[ 877], 95.00th=[ 944], | 99.00th=[ 1083], 99.50th=[ 1167], 99.90th=[ 1250], 99.95th=[ 1267], | 99.99th=[ 1267] bw ( KiB/s): min= 15, max= 253, per=24.98%, avg=97.18, stdev=31.74, samples=208 iops : min= 3, max= 63, avg=23.83, stdev= 7.92, samples=208 lat (msec) : 250=1.37%, 500=12.30%, 750=58.48%, 1000=25.43%, 2000=2.42% cpu : usr=0.05%, sys=0.32%, ctx=2180, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=156049: Tue Nov 25 17:11:42 2025 write: IOPS=24, BW=97.4KiB/s (99.7kB/s)(10.0MiB/105179msec); 0 zone resets slat (usec): min=26, max=5561, avg=146.79, stdev=317.26 clat (msec): min=169, max=1347, avg=656.30, stdev=168.39 lat (msec): min=169, max=1347, avg=656.45, stdev=168.39 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 418], 10.00th=[ 477], 20.00th=[ 531], | 30.00th=[ 567], 40.00th=[ 600], 50.00th=[ 634], 60.00th=[ 684], | 70.00th=[ 735], 80.00th=[ 793], 90.00th=[ 869], 95.00th=[ 944], | 99.00th=[ 1133], 99.50th=[ 1167], 99.90th=[ 1301], 99.95th=[ 1301], | 99.99th=[ 1351] bw ( KiB/s): min= 7, max= 253, per=25.10%, avg=97.65, stdev=32.04, samples=207 iops : min= 1, max= 63, avg=23.96, stdev= 8.02, samples=207 lat (msec) : 250=1.88%, 500=12.38%, 750=58.52%, 1000=24.22%, 2000=3.01% cpu : usr=0.05%, sys=0.33%, ctx=1978, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=156050: Tue Nov 25 17:11:42 2025 write: IOPS=24, BW=97.3KiB/s (99.6kB/s)(10.0MiB/105237msec); 0 zone resets slat (usec): min=25, max=10976, avg=149.60, stdev=369.98 clat (msec): min=201, max=1296, avg=656.90, stdev=165.26 lat (msec): min=201, max=1297, avg=657.05, stdev=165.26 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 430], 10.00th=[ 472], 20.00th=[ 531], | 30.00th=[ 567], 40.00th=[ 600], 50.00th=[ 634], 60.00th=[ 684], | 70.00th=[ 735], 80.00th=[ 793], 90.00th=[ 869], 95.00th=[ 936], | 99.00th=[ 1116], 99.50th=[ 1183], 99.90th=[ 1284], 99.95th=[ 1301], | 99.99th=[ 1301] bw ( KiB/s): min= 7, max= 253, per=25.35%, avg=98.62, stdev=33.00, samples=205 iops : min= 1, max= 63, avg=24.20, stdev= 8.23, samples=205 lat (msec) : 250=1.45%, 500=12.50%, 750=59.61%, 1000=23.71%, 2000=2.73% cpu : usr=0.06%, sys=0.31%, ctx=2116, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=156051: Tue Nov 25 17:11:42 2025 write: IOPS=24, BW=97.4KiB/s (99.7kB/s)(10.0MiB/105183msec); 0 zone resets slat (usec): min=23, max=15118, avg=146.57, stdev=391.23 clat (msec): min=60, max=1262, avg=656.76, stdev=169.42 lat (msec): min=60, max=1262, avg=656.91, stdev=169.41 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 426], 10.00th=[ 477], 20.00th=[ 535], | 30.00th=[ 567], 40.00th=[ 600], 50.00th=[ 634], 60.00th=[ 676], | 70.00th=[ 735], 80.00th=[ 802], 90.00th=[ 877], 95.00th=[ 953], | 99.00th=[ 1116], 99.50th=[ 1167], 99.90th=[ 1234], 99.95th=[ 1250], | 99.99th=[ 1267] bw ( KiB/s): min= 23, max= 273, per=24.86%, avg=96.71, stdev=31.82, samples=209 iops : min= 5, max= 68, avg=23.72, stdev= 7.94, samples=209 lat (msec) : 100=0.27%, 250=1.60%, 500=11.37%, 750=59.80%, 1000=24.14% lat (msec) : 2000=2.81% cpu : usr=0.05%, sys=0.34%, ctx=1983, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=389KiB/s (399kB/s), 97.3KiB/s-97.4KiB/s (99.6kB/s-99.7kB/s), io=40.0MiB (41.9MB), run=105179-105237msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=156075: Tue Nov 25 17:12:45 2025 read: IOPS=21, BW=86.0KiB/s (88.1kB/s)(5048KiB/58689msec) slat (usec): min=29, max=19537, avg=6550.82, stdev=2728.56 clat (msec): min=19, max=598, avg=238.78, stdev=61.74 lat (msec): min=25, max=606, avg=245.33, stdev=62.42 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 153], 10.00th=[ 174], 20.00th=[ 192], | 30.00th=[ 209], 40.00th=[ 222], 50.00th=[ 234], 60.00th=[ 247], | 70.00th=[ 264], 80.00th=[ 284], 90.00th=[ 313], 95.00th=[ 347], | 99.00th=[ 405], 99.50th=[ 439], 99.90th=[ 592], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 31, max= 152, per=25.24%, avg=85.58, stdev=28.34, samples=116 iops : min= 7, max= 38, avg=21.00, stdev= 7.08, samples=116 write: IOPS=22, BW=88.5KiB/s (90.6kB/s)(5192KiB/58689msec); 0 zone resets slat (usec): min=31, max=2523, avg=155.54, stdev=210.72 clat (msec): min=54, max=1051, avg=484.30, stdev=134.24 lat (msec): min=54, max=1051, avg=484.46, stdev=134.24 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 279], 10.00th=[ 326], 20.00th=[ 380], | 30.00th=[ 414], 40.00th=[ 447], 50.00th=[ 481], 60.00th=[ 510], | 70.00th=[ 542], 80.00th=[ 584], 90.00th=[ 651], 95.00th=[ 709], | 99.00th=[ 885], 99.50th=[ 927], 99.90th=[ 1011], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 23, max= 174, per=24.87%, avg=87.80, stdev=26.66, samples=116 iops : min= 5, max= 43, avg=21.55, stdev= 6.69, samples=116 lat (msec) : 20=0.04%, 50=0.16%, 100=0.86%, 250=31.29%, 500=45.62% lat (msec) : 750=20.23%, 1000=1.72%, 2000=0.08% cpu : usr=0.13%, sys=0.88%, ctx=4776, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=156076: Tue Nov 25 17:12:45 2025 read: IOPS=21, BW=84.2KiB/s (86.2kB/s)(4976KiB/59098msec) slat (usec): min=2156, max=20301, avg=6593.63, stdev=2679.05 clat (msec): min=28, max=445, avg=239.93, stdev=57.00 lat (msec): min=33, max=454, avg=246.53, stdev=57.46 clat percentiles (msec): | 1.00th=[ 116], 5.00th=[ 159], 10.00th=[ 176], 20.00th=[ 194], | 30.00th=[ 211], 40.00th=[ 226], 50.00th=[ 236], 60.00th=[ 249], | 70.00th=[ 264], 80.00th=[ 284], 90.00th=[ 313], 95.00th=[ 347], | 99.00th=[ 397], 99.50th=[ 422], 99.90th=[ 443], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 7, max= 159, per=24.57%, avg=83.29, stdev=29.98, samples=117 iops : min= 1, max= 39, avg=20.43, stdev= 7.50, samples=117 write: IOPS=22, BW=89.1KiB/s (91.2kB/s)(5264KiB/59098msec); 0 zone resets slat (usec): min=28, max=3961, avg=169.20, stdev=268.89 clat (msec): min=45, max=1029, avg=484.52, stdev=132.98 lat (msec): min=45, max=1029, avg=484.70, stdev=132.99 clat percentiles (msec): | 1.00th=[ 188], 5.00th=[ 279], 10.00th=[ 326], 20.00th=[ 380], | 30.00th=[ 418], 40.00th=[ 451], 50.00th=[ 481], 60.00th=[ 510], | 70.00th=[ 542], 80.00th=[ 584], 90.00th=[ 642], 95.00th=[ 709], | 99.00th=[ 860], 99.50th=[ 894], 99.90th=[ 1003], 99.95th=[ 1028], | 99.99th=[ 1028] bw ( KiB/s): min= 23, max= 184, per=25.07%, avg=88.51, stdev=27.68, samples=117 iops : min= 5, max= 46, avg=21.74, stdev= 6.95, samples=117 lat (msec) : 50=0.23%, 100=0.27%, 250=31.09%, 500=46.60%, 750=19.80% lat (msec) : 1000=1.91%, 2000=0.08% cpu : usr=0.10%, sys=0.87%, ctx=4839, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=156077: Tue Nov 25 17:12:45 2025 read: IOPS=21, BW=84.2KiB/s (86.2kB/s)(4984KiB/59179msec) slat (usec): min=2136, max=19760, avg=6629.56, stdev=2710.47 clat (msec): min=4, max=450, avg=238.02, stdev=54.88 lat (msec): min=9, max=461, avg=244.66, stdev=55.41 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 157], 10.00th=[ 176], 20.00th=[ 194], | 30.00th=[ 211], 40.00th=[ 224], 50.00th=[ 236], 60.00th=[ 249], | 70.00th=[ 264], 80.00th=[ 279], 90.00th=[ 305], 95.00th=[ 334], | 99.00th=[ 384], 99.50th=[ 405], 99.90th=[ 443], 99.95th=[ 451], | 99.99th=[ 451] bw ( KiB/s): min= 16, max= 168, per=24.62%, avg=83.45, stdev=31.94, samples=117 iops : min= 4, max= 42, avg=20.47, stdev= 7.98, samples=117 write: IOPS=22, BW=88.8KiB/s (90.9kB/s)(5256KiB/59179msec); 0 zone resets slat (usec): min=30, max=5692, avg=168.48, stdev=300.66 clat (msec): min=83, max=1015, avg=487.69, stdev=134.08 lat (msec): min=83, max=1015, avg=487.86, stdev=134.09 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 279], 10.00th=[ 334], 20.00th=[ 384], | 30.00th=[ 422], 40.00th=[ 447], 50.00th=[ 481], 60.00th=[ 514], | 70.00th=[ 550], 80.00th=[ 592], 90.00th=[ 659], 95.00th=[ 701], | 99.00th=[ 860], 99.50th=[ 894], 99.90th=[ 1011], 99.95th=[ 1020], | 99.99th=[ 1020] bw ( KiB/s): min= 31, max= 159, per=24.82%, avg=87.60, stdev=22.74, samples=117 iops : min= 7, max= 39, avg=21.50, stdev= 5.70, samples=117 lat (msec) : 10=0.08%, 50=0.16%, 100=0.16%, 250=31.48%, 500=45.78% lat (msec) : 750=20.62%, 1000=1.64%, 2000=0.08% cpu : usr=0.15%, sys=0.85%, ctx=4717, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=156078: Tue Nov 25 17:12:45 2025 read: IOPS=21, BW=85.6KiB/s (87.7kB/s)(5060KiB/59098msec) slat (usec): min=1878, max=23728, avg=6587.05, stdev=2659.93 clat (msec): min=34, max=503, avg=240.14, stdev=56.32 lat (msec): min=40, max=509, avg=246.73, stdev=56.83 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 161], 10.00th=[ 176], 20.00th=[ 194], | 30.00th=[ 211], 40.00th=[ 226], 50.00th=[ 239], 60.00th=[ 249], | 70.00th=[ 264], 80.00th=[ 284], 90.00th=[ 313], 95.00th=[ 334], | 99.00th=[ 405], 99.50th=[ 426], 99.90th=[ 456], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 24, max= 167, per=25.13%, avg=85.19, stdev=28.73, samples=117 iops : min= 6, max= 41, avg=20.91, stdev= 7.21, samples=117 write: IOPS=21, BW=87.7KiB/s (89.8kB/s)(5180KiB/59098msec); 0 zone resets slat (usec): min=31, max=4312, avg=170.00, stdev=268.37 clat (msec): min=117, max=1051, avg=487.98, stdev=131.67 lat (msec): min=117, max=1051, avg=488.15, stdev=131.68 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 284], 10.00th=[ 330], 20.00th=[ 384], | 30.00th=[ 418], 40.00th=[ 447], 50.00th=[ 477], 60.00th=[ 514], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 651], 95.00th=[ 718], | 99.00th=[ 852], 99.50th=[ 919], 99.90th=[ 1028], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 31, max= 136, per=24.59%, avg=86.80, stdev=22.98, samples=117 iops : min= 7, max= 34, avg=21.31, stdev= 5.76, samples=117 lat (msec) : 50=0.08%, 100=0.20%, 250=30.70%, 500=46.95%, 750=20.20% lat (msec) : 1000=1.80%, 2000=0.08% cpu : usr=0.13%, sys=0.88%, ctx=5045, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=339KiB/s (347kB/s), 84.2KiB/s-86.0KiB/s (86.2kB/s-88.1kB/s), io=19.6MiB (20.5MB), run=58689-59179msec WRITE: bw=353KiB/s (362kB/s), 87.7KiB/s-89.1KiB/s (89.8kB/s-91.2kB/s), io=20.4MiB (21.4MB), run=58689-59179msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=156090: Tue Nov 25 17:12:50 2025 read: IOPS=5, BW=234MiB/s (245MB/s)(40.0MiB/171msec) slat (nsec): min=13751k, max=13751k, avg=13751068.00, stdev= 0.00 clat (nsec): min=148970k, max=148970k, avg=148969839.00, stdev= 0.00 lat (nsec): min=162725k, max=162725k, avg=162725477.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 150], 10.00th=[ 150], 20.00th=[ 150], | 30.00th=[ 150], 40.00th=[ 150], 50.00th=[ 150], 60.00th=[ 150], | 70.00th=[ 150], 80.00th=[ 150], 90.00th=[ 150], 95.00th=[ 150], | 99.00th=[ 150], 99.50th=[ 150], 99.90th=[ 150], 99.95th=[ 150], | 99.99th=[ 150] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.06%, ctx=40, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=234MiB/s (245MB/s), 234MiB/s-234MiB/s (245MB/s-245MB/s), io=40.0MiB (41.9MB), run=171-171msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout