== sanity test 398c: run fio to test AIO ================= 13:35:46 (1761240946) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.364469 s, 115 MB/s osc.lustre-OST0000-osc-ffff8c1490550000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=184184: Thu Oct 23 13:36:17 2025 write: IOPS=110, BW=443KiB/s (454kB/s)(10.0MiB/23090msec); 0 zone resets slat (usec): min=26, max=89848, avg=199.32, stdev=1843.38 clat (msec): min=21, max=286, avg=142.91, stdev=34.23 lat (msec): min=21, max=286, avg=143.11, stdev=34.17 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 92], 10.00th=[ 104], 20.00th=[ 115], | 30.00th=[ 125], 40.00th=[ 133], 50.00th=[ 140], 60.00th=[ 148], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 188], 95.00th=[ 205], | 99.00th=[ 234], 99.50th=[ 257], 99.90th=[ 288], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 263, max= 624, per=25.03%, avg=443.56, stdev=78.44, samples=45 iops : min= 65, max= 156, avg=110.47, stdev=19.62, samples=45 lat (msec) : 50=0.23%, 100=8.01%, 250=91.25%, 500=0.51% cpu : usr=0.21%, sys=1.27%, ctx=2843, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184185: Thu Oct 23 13:36:17 2025 write: IOPS=110, BW=443KiB/s (454kB/s)(10.0MiB/23108msec); 0 zone resets slat (usec): min=26, max=8743, avg=120.51, stdev=389.42 clat (msec): min=63, max=284, avg=143.03, stdev=33.45 lat (msec): min=63, max=284, avg=143.15, stdev=33.46 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 94], 10.00th=[ 103], 20.00th=[ 115], | 30.00th=[ 125], 40.00th=[ 132], 50.00th=[ 140], 60.00th=[ 148], | 70.00th=[ 159], 80.00th=[ 171], 90.00th=[ 184], 95.00th=[ 209], | 99.00th=[ 228], 99.50th=[ 232], 99.90th=[ 284], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 287, max= 640, per=25.01%, avg=443.18, stdev=84.58, samples=45 iops : min= 71, max= 160, avg=110.40, stdev=21.16, samples=45 lat (msec) : 100=8.71%, 250=91.05%, 500=0.23% cpu : usr=0.15%, sys=1.26%, ctx=1411, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184186: Thu Oct 23 13:36:17 2025 write: IOPS=111, BW=446KiB/s (457kB/s)(10.0MiB/22943msec); 0 zone resets slat (usec): min=28, max=9430, avg=186.66, stdev=582.63 clat (msec): min=11, max=276, avg=143.07, stdev=33.72 lat (msec): min=11, max=276, avg=143.26, stdev=33.72 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 92], 10.00th=[ 104], 20.00th=[ 117], | 30.00th=[ 127], 40.00th=[ 134], 50.00th=[ 140], 60.00th=[ 148], | 70.00th=[ 157], 80.00th=[ 167], 90.00th=[ 190], 95.00th=[ 207], | 99.00th=[ 228], 99.50th=[ 234], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] bw ( KiB/s): min= 287, max= 632, per=25.16%, avg=445.80, stdev=75.03, samples=45 iops : min= 71, max= 158, avg=111.00, stdev=18.81, samples=45 lat (msec) : 20=0.16%, 50=0.39%, 100=7.73%, 250=91.48%, 500=0.23% cpu : usr=0.21%, sys=1.36%, ctx=2925, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=184187: Thu Oct 23 13:36:17 2025 write: IOPS=110, BW=443KiB/s (454kB/s)(10.0MiB/23099msec); 0 zone resets slat (usec): min=23, max=7195, avg=152.14, stdev=428.27 clat (msec): min=15, max=252, avg=142.90, stdev=33.01 lat (msec): min=17, max=252, avg=143.05, stdev=33.01 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 91], 10.00th=[ 103], 20.00th=[ 115], | 30.00th=[ 127], 40.00th=[ 134], 50.00th=[ 142], 60.00th=[ 148], | 70.00th=[ 157], 80.00th=[ 169], 90.00th=[ 186], 95.00th=[ 207], | 99.00th=[ 228], 99.50th=[ 230], 99.90th=[ 251], 99.95th=[ 253], | 99.99th=[ 253] bw ( KiB/s): min= 269, max= 608, per=24.94%, avg=441.96, stdev=72.37, samples=45 iops : min= 67, max= 152, avg=110.04, stdev=18.18, samples=45 lat (msec) : 20=0.04%, 50=0.08%, 100=8.44%, 250=91.33%, 500=0.12% cpu : usr=0.22%, sys=1.07%, ctx=2368, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1773KiB/s (1815kB/s), 443KiB/s-446KiB/s (454kB/s-457kB/s), io=40.0MiB (41.9MB), run=22943-23108msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=184200: Thu Oct 23 13:36:49 2025 read: IOPS=56, BW=227KiB/s (232kB/s)(5048KiB/22241msec) slat (usec): min=2365, max=68356, avg=9477.66, stdev=4662.67 clat (msec): min=41, max=224, avg=130.18, stdev=28.07 lat (msec): min=48, max=283, avg=139.66, stdev=29.45 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 89], 10.00th=[ 96], 20.00th=[ 107], | 30.00th=[ 114], 40.00th=[ 122], 50.00th=[ 128], 60.00th=[ 136], | 70.00th=[ 142], 80.00th=[ 153], 90.00th=[ 169], 95.00th=[ 180], | 99.00th=[ 209], 99.50th=[ 215], 99.90th=[ 224], 99.95th=[ 226], | 99.99th=[ 226] bw ( KiB/s): min= 97, max= 312, per=25.07%, avg=226.14, stdev=48.33, samples=44 iops : min= 24, max= 78, avg=56.11, stdev=12.09, samples=44 write: IOPS=58, BW=233KiB/s (239kB/s)(5192KiB/22241msec); 0 zone resets slat (usec): min=31, max=5048, avg=166.51, stdev=362.13 clat (msec): min=41, max=244, avg=135.36, stdev=27.60 lat (msec): min=41, max=244, avg=135.52, stdev=27.62 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 94], 10.00th=[ 106], 20.00th=[ 113], | 30.00th=[ 121], 40.00th=[ 126], 50.00th=[ 133], 60.00th=[ 140], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 174], 95.00th=[ 186], | 99.00th=[ 211], 99.50th=[ 215], 99.90th=[ 245], 99.95th=[ 245], | 99.99th=[ 245] bw ( KiB/s): min= 97, max= 335, per=24.78%, avg=232.73, stdev=47.33, samples=44 iops : min= 24, max= 83, avg=57.75, stdev=11.94, samples=44 lat (msec) : 50=0.16%, 100=9.96%, 250=89.88% cpu : usr=0.28%, sys=1.94%, ctx=4601, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184201: Thu Oct 23 13:36:49 2025 read: IOPS=56, BW=224KiB/s (229kB/s)(4976KiB/22205msec) slat (usec): min=2615, max=62872, avg=9229.72, stdev=4429.87 clat (msec): min=19, max=257, avg=130.44, stdev=29.86 lat (msec): min=30, max=264, avg=139.69, stdev=30.78 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 89], 10.00th=[ 97], 20.00th=[ 106], | 30.00th=[ 114], 40.00th=[ 122], 50.00th=[ 127], 60.00th=[ 134], | 70.00th=[ 144], 80.00th=[ 155], 90.00th=[ 169], 95.00th=[ 184], | 99.00th=[ 215], 99.50th=[ 245], 99.90th=[ 255], 99.95th=[ 257], | 99.99th=[ 257] bw ( KiB/s): min= 97, max= 336, per=24.69%, avg=222.73, stdev=46.52, samples=44 iops : min= 24, max= 84, avg=55.25, stdev=11.67, samples=44 write: IOPS=59, BW=237KiB/s (243kB/s)(5264KiB/22205msec); 0 zone resets slat (usec): min=29, max=5098, avg=180.83, stdev=401.67 clat (msec): min=69, max=261, avg=135.72, stdev=27.22 lat (msec): min=69, max=261, avg=135.90, stdev=27.22 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 97], 10.00th=[ 105], 20.00th=[ 114], | 30.00th=[ 122], 40.00th=[ 128], 50.00th=[ 133], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 174], 95.00th=[ 186], | 99.00th=[ 213], 99.50th=[ 234], 99.90th=[ 245], 99.95th=[ 262], | 99.99th=[ 262] bw ( KiB/s): min= 135, max= 327, per=25.21%, avg=236.73, stdev=46.92, samples=44 iops : min= 33, max= 81, avg=58.77, stdev=11.80, samples=44 lat (msec) : 20=0.04%, 50=0.04%, 100=10.20%, 250=89.61%, 500=0.12% cpu : usr=0.28%, sys=2.04%, ctx=4729, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184202: Thu Oct 23 13:36:49 2025 read: IOPS=56, BW=224KiB/s (230kB/s)(4984KiB/22203msec) slat (usec): min=2400, max=68183, avg=9367.92, stdev=4466.23 clat (msec): min=57, max=251, avg=130.76, stdev=28.50 lat (msec): min=67, max=284, avg=140.13, stdev=29.68 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 89], 10.00th=[ 97], 20.00th=[ 108], | 30.00th=[ 114], 40.00th=[ 122], 50.00th=[ 129], 60.00th=[ 136], | 70.00th=[ 144], 80.00th=[ 153], 90.00th=[ 167], 95.00th=[ 182], | 99.00th=[ 218], 99.50th=[ 228], 99.90th=[ 245], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 118, max= 312, per=24.87%, avg=224.32, stdev=43.50, samples=44 iops : min= 29, max= 78, avg=55.64, stdev=10.94, samples=44 write: IOPS=59, BW=237KiB/s (242kB/s)(5256KiB/22203msec); 0 zone resets slat (usec): min=33, max=10153, avg=168.25, stdev=428.25 clat (msec): min=69, max=237, avg=135.50, stdev=27.32 lat (msec): min=69, max=237, avg=135.67, stdev=27.33 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 96], 10.00th=[ 104], 20.00th=[ 114], | 30.00th=[ 120], 40.00th=[ 125], 50.00th=[ 132], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 157], 90.00th=[ 171], 95.00th=[ 186], | 99.00th=[ 211], 99.50th=[ 215], 99.90th=[ 232], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 151, max= 349, per=25.05%, avg=235.20, stdev=51.04, samples=44 iops : min= 37, max= 87, avg=58.39, stdev=12.79, samples=44 lat (msec) : 100=10.00%, 250=89.96%, 500=0.04% cpu : usr=0.28%, sys=2.07%, ctx=4619, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=184203: Thu Oct 23 13:36:49 2025 read: IOPS=57, BW=229KiB/s (235kB/s)(5060KiB/22087msec) slat (usec): min=35, max=61755, avg=9247.97, stdev=4652.98 clat (msec): min=59, max=288, avg=130.76, stdev=28.29 lat (msec): min=70, max=305, avg=140.02, stdev=29.45 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 90], 10.00th=[ 99], 20.00th=[ 108], | 30.00th=[ 116], 40.00th=[ 122], 50.00th=[ 128], 60.00th=[ 134], | 70.00th=[ 142], 80.00th=[ 150], 90.00th=[ 169], 95.00th=[ 182], | 99.00th=[ 211], 99.50th=[ 236], 99.90th=[ 255], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 143, max= 327, per=25.32%, avg=228.42, stdev=46.65, samples=43 iops : min= 35, max= 81, avg=56.65, stdev=11.74, samples=43 write: IOPS=58, BW=235KiB/s (240kB/s)(5180KiB/22087msec); 0 zone resets slat (usec): min=32, max=14884, avg=182.84, stdev=621.74 clat (msec): min=55, max=236, avg=135.36, stdev=26.63 lat (msec): min=56, max=237, avg=135.55, stdev=26.62 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 96], 10.00th=[ 105], 20.00th=[ 114], | 30.00th=[ 122], 40.00th=[ 127], 50.00th=[ 133], 60.00th=[ 138], | 70.00th=[ 146], 80.00th=[ 155], 90.00th=[ 171], 95.00th=[ 186], | 99.00th=[ 211], 99.50th=[ 218], 99.90th=[ 230], 99.95th=[ 239], | 99.99th=[ 239] bw ( KiB/s): min= 167, max= 312, per=24.82%, avg=233.02, stdev=40.88, samples=43 iops : min= 41, max= 78, avg=57.81, stdev=10.29, samples=43 lat (msec) : 100=9.30%, 250=90.62%, 500=0.08% cpu : usr=0.30%, sys=2.04%, ctx=4745, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=902KiB/s (924kB/s), 224KiB/s-229KiB/s (229kB/s-235kB/s), io=19.6MiB (20.5MB), run=22087-22241msec WRITE: bw=939KiB/s (962kB/s), 233KiB/s-237KiB/s (239kB/s-243kB/s), io=20.4MiB (21.4MB), run=22087-22241msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=184210: Thu Oct 23 13:36:57 2025 read: IOPS=3, BW=127MiB/s (133MB/s)(40.0MiB/315msec) slat (nsec): min=14018k, max=14018k, avg=14017527.00, stdev= 0.00 clat (nsec): min=273220k, max=273220k, avg=273219997.00, stdev= 0.00 lat (nsec): min=287241k, max=287241k, avg=287241272.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 275], 10.00th=[ 275], 20.00th=[ 275], | 30.00th=[ 275], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 275], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.50%, ctx=12, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=40.0MiB (41.9MB), run=315-315msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout