== sanity test 398c: run fio to test AIO ================= 23:40:20 (1781235620) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.431262 s, 97.3 MB/s osc.lustre-OST0000-osc-ffffa0b7de980000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180354: Thu Jun 11 23:40:58 2026 write: IOPS=92, BW=372KiB/s (381kB/s)(10.0MiB/27528msec); 0 zone resets slat (usec): min=37, max=15189, avg=255.26, stdev=787.44 clat (msec): min=30, max=404, avg=171.04, stdev=53.57 lat (msec): min=31, max=404, avg=171.30, stdev=53.58 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 102], 10.00th=[ 117], 20.00th=[ 130], | 30.00th=[ 138], 40.00th=[ 148], 50.00th=[ 159], 60.00th=[ 174], | 70.00th=[ 194], 80.00th=[ 218], 90.00th=[ 249], 95.00th=[ 271], | 99.00th=[ 309], 99.50th=[ 401], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 166, max= 555, per=24.76%, avg=368.11, stdev=91.24, samples=54 iops : min= 41, max= 138, avg=91.65, stdev=22.82, samples=54 lat (msec) : 50=0.08%, 100=4.69%, 250=86.48%, 500=8.75% cpu : usr=0.19%, sys=1.10%, ctx=2663, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180355: Thu Jun 11 23:40:58 2026 write: IOPS=93, BW=374KiB/s (383kB/s)(10.0MiB/27355msec); 0 zone resets slat (usec): min=39, max=15633, avg=217.60, stdev=666.33 clat (msec): min=20, max=403, avg=170.54, stdev=54.88 lat (msec): min=20, max=403, avg=170.76, stdev=54.89 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 94], 10.00th=[ 116], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 150], 50.00th=[ 161], 60.00th=[ 174], | 70.00th=[ 192], 80.00th=[ 211], 90.00th=[ 249], 95.00th=[ 275], | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 405] bw ( KiB/s): min= 198, max= 780, per=25.03%, avg=372.20, stdev=105.99, samples=54 iops : min= 49, max= 195, avg=92.67, stdev=26.51, samples=54 lat (msec) : 50=0.82%, 100=5.35%, 250=83.98%, 500=9.84% cpu : usr=0.15%, sys=0.98%, ctx=2248, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180356: Thu Jun 11 23:40:58 2026 write: IOPS=93, BW=373KiB/s (382kB/s)(10.0MiB/27432msec); 0 zone resets slat (usec): min=37, max=20263, avg=254.65, stdev=931.61 clat (msec): min=19, max=347, avg=170.60, stdev=52.56 lat (msec): min=20, max=347, avg=170.85, stdev=52.59 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 104], 10.00th=[ 117], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 148], 50.00th=[ 161], 60.00th=[ 174], | 70.00th=[ 190], 80.00th=[ 220], 90.00th=[ 251], 95.00th=[ 271], | 99.00th=[ 305], 99.50th=[ 309], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] bw ( KiB/s): min= 190, max= 555, per=24.92%, avg=370.61, stdev=92.15, samples=54 iops : min= 47, max= 138, avg=92.28, stdev=22.98, samples=54 lat (msec) : 20=0.04%, 50=0.20%, 100=3.87%, 250=85.47%, 500=10.43% cpu : usr=0.15%, sys=1.24%, ctx=2094, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180357: Thu Jun 11 23:40:58 2026 write: IOPS=93, BW=373KiB/s (382kB/s)(10.0MiB/27466msec); 0 zone resets slat (usec): min=38, max=13119, avg=276.01, stdev=797.88 clat (msec): min=15, max=404, avg=170.67, stdev=51.61 lat (msec): min=16, max=404, avg=170.95, stdev=51.62 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 104], 10.00th=[ 115], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 150], 50.00th=[ 159], 60.00th=[ 174], | 70.00th=[ 199], 80.00th=[ 218], 90.00th=[ 241], 95.00th=[ 268], | 99.00th=[ 300], 99.50th=[ 330], 99.90th=[ 388], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 238, max= 603, per=24.85%, avg=369.50, stdev=92.23, samples=54 iops : min= 59, max= 150, avg=92.00, stdev=23.03, samples=54 lat (msec) : 20=0.04%, 50=0.08%, 100=4.06%, 250=88.01%, 500=7.81% cpu : usr=0.23%, sys=1.08%, ctx=2775, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1488KiB/s (1524kB/s), 372KiB/s-374KiB/s (381kB/s-383kB/s), io=40.0MiB (41.9MB), run=27355-27528msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180371: Thu Jun 11 23:41:38 2026 read: IOPS=42, BW=170KiB/s (174kB/s)(5048KiB/29625msec) slat (usec): min=2393, max=66842, avg=13191.97, stdev=8276.67 clat (msec): min=9, max=353, avg=173.27, stdev=51.36 lat (msec): min=12, max=386, avg=186.47, stdev=54.27 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 94], 10.00th=[ 109], 20.00th=[ 128], | 30.00th=[ 144], 40.00th=[ 159], 50.00th=[ 171], 60.00th=[ 184], | 70.00th=[ 199], 80.00th=[ 215], 90.00th=[ 241], 95.00th=[ 262], | 99.00th=[ 309], 99.50th=[ 321], 99.90th=[ 347], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 50, max= 296, per=25.03%, avg=169.45, stdev=53.46, samples=58 iops : min= 12, max= 74, avg=41.97, stdev=13.37, samples=58 write: IOPS=43, BW=175KiB/s (179kB/s)(5192KiB/29625msec); 0 zone resets slat (usec): min=47, max=11609, avg=303.30, stdev=770.22 clat (msec): min=76, max=377, avg=180.30, stdev=52.27 lat (msec): min=76, max=377, avg=180.61, stdev=52.27 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 101], 10.00th=[ 114], 20.00th=[ 136], | 30.00th=[ 150], 40.00th=[ 163], 50.00th=[ 178], 60.00th=[ 192], | 70.00th=[ 205], 80.00th=[ 222], 90.00th=[ 249], 95.00th=[ 271], | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 376], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 43, max= 328, per=24.85%, avg=175.17, stdev=55.18, samples=58 iops : min= 10, max= 82, avg=43.40, stdev=13.80, samples=58 lat (msec) : 10=0.04%, 100=5.78%, 250=85.39%, 500=8.79% cpu : usr=0.26%, sys=2.05%, ctx=4359, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180372: Thu Jun 11 23:41:38 2026 read: IOPS=41, BW=168KiB/s (172kB/s)(4976KiB/29629msec) slat (usec): min=2725, max=67212, avg=13371.17, stdev=8372.96 clat (msec): min=33, max=353, avg=175.14, stdev=53.02 lat (msec): min=37, max=361, avg=188.52, stdev=55.86 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 94], 10.00th=[ 109], 20.00th=[ 128], | 30.00th=[ 144], 40.00th=[ 161], 50.00th=[ 171], 60.00th=[ 184], | 70.00th=[ 201], 80.00th=[ 220], 90.00th=[ 251], 95.00th=[ 268], | 99.00th=[ 309], 99.50th=[ 317], 99.90th=[ 330], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 42, max= 280, per=24.64%, avg=166.83, stdev=48.88, samples=58 iops : min= 10, max= 70, avg=41.29, stdev=12.22, samples=58 write: IOPS=44, BW=178KiB/s (182kB/s)(5264KiB/29629msec); 0 zone resets slat (usec): min=39, max=9716, avg=295.97, stdev=724.08 clat (msec): min=62, max=382, avg=179.46, stdev=52.28 lat (msec): min=62, max=382, avg=179.76, stdev=52.31 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 103], 10.00th=[ 113], 20.00th=[ 134], | 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 176], 60.00th=[ 190], | 70.00th=[ 205], 80.00th=[ 226], 90.00th=[ 249], 95.00th=[ 266], | 99.00th=[ 321], 99.50th=[ 342], 99.90th=[ 376], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 92, max= 328, per=25.16%, avg=177.38, stdev=56.59, samples=58 iops : min= 23, max= 82, avg=43.95, stdev=14.14, samples=58 lat (msec) : 50=0.04%, 100=5.35%, 250=85.04%, 500=9.57% cpu : usr=0.27%, sys=1.92%, ctx=4289, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180373: Thu Jun 11 23:41:38 2026 read: IOPS=42, BW=168KiB/s (172kB/s)(4984KiB/29604msec) slat (usec): min=2901, max=63935, avg=13159.10, stdev=8170.58 clat (msec): min=11, max=412, avg=175.16, stdev=53.13 lat (msec): min=30, max=421, avg=188.32, stdev=55.69 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 96], 10.00th=[ 109], 20.00th=[ 129], | 30.00th=[ 148], 40.00th=[ 159], 50.00th=[ 171], 60.00th=[ 186], | 70.00th=[ 201], 80.00th=[ 218], 90.00th=[ 243], 95.00th=[ 266], | 99.00th=[ 334], 99.50th=[ 342], 99.90th=[ 401], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 57, max= 288, per=24.83%, avg=168.10, stdev=49.21, samples=58 iops : min= 14, max= 72, avg=41.62, stdev=12.31, samples=58 write: IOPS=44, BW=178KiB/s (182kB/s)(5256KiB/29604msec); 0 zone resets slat (usec): min=42, max=49798, avg=277.62, stdev=1504.52 clat (msec): min=65, max=374, avg=179.43, stdev=52.59 lat (msec): min=65, max=374, avg=179.71, stdev=52.62 clat percentiles (msec): | 1.00th=[ 87], 5.00th=[ 103], 10.00th=[ 114], 20.00th=[ 133], | 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 174], 60.00th=[ 190], | 70.00th=[ 207], 80.00th=[ 224], 90.00th=[ 249], 95.00th=[ 275], | 99.00th=[ 321], 99.50th=[ 326], 99.90th=[ 372], 99.95th=[ 376], | 99.99th=[ 376] bw ( KiB/s): min= 71, max= 351, per=25.07%, avg=176.78, stdev=58.01, samples=58 iops : min= 17, max= 87, avg=43.79, stdev=14.51, samples=58 lat (msec) : 20=0.04%, 100=5.00%, 250=86.05%, 500=8.91% cpu : usr=0.23%, sys=1.96%, ctx=4502, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180374: Thu Jun 11 23:41:38 2026 read: IOPS=42, BW=171KiB/s (175kB/s)(5060KiB/29624msec) slat (usec): min=171, max=79195, avg=13210.94, stdev=8309.49 clat (msec): min=11, max=399, avg=174.85, stdev=52.53 lat (msec): min=19, max=410, avg=188.07, stdev=55.31 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 95], 10.00th=[ 107], 20.00th=[ 130], | 30.00th=[ 146], 40.00th=[ 159], 50.00th=[ 171], 60.00th=[ 184], | 70.00th=[ 201], 80.00th=[ 220], 90.00th=[ 247], 95.00th=[ 266], | 99.00th=[ 313], 99.50th=[ 338], 99.90th=[ 372], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 80, max= 336, per=25.11%, avg=170.02, stdev=50.44, samples=58 iops : min= 20, max= 84, avg=42.14, stdev=12.61, samples=58 write: IOPS=43, BW=175KiB/s (179kB/s)(5180KiB/29624msec); 0 zone resets slat (usec): min=38, max=40288, avg=311.44, stdev=1326.61 clat (msec): min=73, max=403, avg=181.40, stdev=53.90 lat (msec): min=74, max=403, avg=181.71, stdev=53.90 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 100], 10.00th=[ 112], 20.00th=[ 132], | 30.00th=[ 150], 40.00th=[ 163], 50.00th=[ 180], 60.00th=[ 194], | 70.00th=[ 207], 80.00th=[ 228], 90.00th=[ 253], 95.00th=[ 275], | 99.00th=[ 321], 99.50th=[ 338], 99.90th=[ 372], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 88, max= 296, per=24.65%, avg=173.81, stdev=53.22, samples=58 iops : min= 22, max= 74, avg=43.07, stdev=13.26, samples=58 lat (msec) : 20=0.04%, 100=5.94%, 250=84.30%, 500=9.73% cpu : usr=0.27%, sys=2.08%, ctx=4254, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=677KiB/s (694kB/s), 168KiB/s-171KiB/s (172kB/s-175kB/s), io=19.6MiB (20.5MB), run=29604-29629msec WRITE: bw=705KiB/s (722kB/s), 175KiB/s-178KiB/s (179kB/s-182kB/s), io=20.4MiB (21.4MB), run=29604-29629msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180382: Thu Jun 11 23:41:47 2026 read: IOPS=1, BW=78.3MiB/s (82.1MB/s)(40.0MiB/511msec) slat (nsec): min=58460k, max=58460k, avg=58459606.00, stdev= 0.00 clat (nsec): min=423323k, max=423323k, avg=423322632.00, stdev= 0.00 lat (nsec): min=481788k, max=481788k, avg=481787761.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 422], 10.00th=[ 422], 20.00th=[ 422], | 30.00th=[ 422], 40.00th=[ 422], 50.00th=[ 422], 60.00th=[ 422], | 70.00th=[ 422], 80.00th=[ 422], 90.00th=[ 422], 95.00th=[ 422], | 99.00th=[ 422], 99.50th=[ 422], 99.90th=[ 422], 99.95th=[ 422], | 99.99th=[ 422] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.57%, ctx=5, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=78.3MiB/s (82.1MB/s), 78.3MiB/s-78.3MiB/s (82.1MB/s-82.1MB/s), io=40.0MiB (41.9MB), run=511-511msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout