== sanity test 398c: run fio to test AIO ================= 07:43:03 (1788867783) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.206219 s, 203 MB/s osc.lustre-OST0000-osc-ffff971a2a041000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=155693: Tue Sep 8 07:44:24 2026 write: IOPS=33, BW=134KiB/s (137kB/s)(10.0MiB/76702msec); 0 zone resets slat (usec): min=24, max=6842, avg=166.23, stdev=345.54 clat (msec): min=126, max=779, avg=478.84, stdev=103.91 lat (msec): min=126, max=780, avg=479.01, stdev=103.92 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 305], 10.00th=[ 338], 20.00th=[ 388], | 30.00th=[ 426], 40.00th=[ 456], 50.00th=[ 485], 60.00th=[ 506], | 70.00th=[ 535], 80.00th=[ 567], 90.00th=[ 609], 95.00th=[ 634], | 99.00th=[ 701], 99.50th=[ 726], 99.90th=[ 776], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 72, max= 256, per=24.94%, avg=132.91, stdev=30.81, samples=152 iops : min= 18, max= 64, avg=32.95, stdev= 7.71, samples=152 lat (msec) : 250=0.62%, 500=56.25%, 750=42.89%, 1000=0.23% cpu : usr=0.06%, sys=0.38%, ctx=2863, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155694: Tue Sep 8 07:44:24 2026 write: IOPS=33, BW=134KiB/s (137kB/s)(10.0MiB/76614msec); 0 zone resets slat (usec): min=21, max=6763, avg=157.03, stdev=311.17 clat (msec): min=83, max=768, avg=478.31, stdev=104.05 lat (msec): min=83, max=768, avg=478.47, stdev=104.07 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 305], 10.00th=[ 347], 20.00th=[ 393], | 30.00th=[ 426], 40.00th=[ 456], 50.00th=[ 481], 60.00th=[ 506], | 70.00th=[ 535], 80.00th=[ 567], 90.00th=[ 617], 95.00th=[ 642], | 99.00th=[ 693], 99.50th=[ 726], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 48, max= 256, per=24.99%, avg=133.22, stdev=33.03, samples=152 iops : min= 12, max= 64, avg=33.03, stdev= 8.30, samples=152 lat (msec) : 100=0.66%, 250=0.39%, 500=57.66%, 750=41.05%, 1000=0.23% cpu : usr=0.05%, sys=0.37%, ctx=2717, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155695: Tue Sep 8 07:44:24 2026 write: IOPS=33, BW=134KiB/s (137kB/s)(10.0MiB/76510msec); 0 zone resets slat (usec): min=22, max=6216, avg=158.90, stdev=299.93 clat (msec): min=112, max=777, avg=477.79, stdev=102.73 lat (msec): min=113, max=777, avg=477.95, stdev=102.74 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 305], 10.00th=[ 347], 20.00th=[ 393], | 30.00th=[ 426], 40.00th=[ 456], 50.00th=[ 481], 60.00th=[ 502], | 70.00th=[ 535], 80.00th=[ 567], 90.00th=[ 609], 95.00th=[ 642], | 99.00th=[ 693], 99.50th=[ 726], 99.90th=[ 751], 99.95th=[ 768], | 99.99th=[ 776] bw ( KiB/s): min= 48, max= 254, per=25.00%, avg=133.22, stdev=31.89, samples=152 iops : min= 12, max= 63, avg=33.03, stdev= 8.02, samples=152 lat (msec) : 250=1.33%, 500=57.34%, 750=41.13%, 1000=0.20% cpu : usr=0.06%, sys=0.38%, ctx=2485, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155696: Tue Sep 8 07:44:24 2026 write: IOPS=33, BW=133KiB/s (137kB/s)(10.0MiB/76746msec); 0 zone resets slat (usec): min=22, max=6578, avg=167.25, stdev=328.23 clat (msec): min=108, max=765, avg=479.08, stdev=99.14 lat (msec): min=108, max=766, avg=479.25, stdev=99.14 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 309], 10.00th=[ 351], 20.00th=[ 393], | 30.00th=[ 426], 40.00th=[ 456], 50.00th=[ 485], 60.00th=[ 510], | 70.00th=[ 535], 80.00th=[ 567], 90.00th=[ 609], 95.00th=[ 634], | 99.00th=[ 684], 99.50th=[ 709], 99.90th=[ 751], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 63, max= 256, per=24.94%, avg=132.91, stdev=32.11, samples=152 iops : min= 15, max= 64, avg=32.95, stdev= 8.06, samples=152 lat (msec) : 250=0.70%, 500=55.51%, 750=43.67%, 1000=0.12% cpu : usr=0.06%, sys=0.39%, ctx=2890, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=534KiB/s (547kB/s), 133KiB/s-134KiB/s (137kB/s-137kB/s), io=40.0MiB (41.9MB), run=76510-76746msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=155717: Tue Sep 8 07:45:24 2026 read: IOPS=22, BW=89.5KiB/s (91.7kB/s)(5048KiB/56394msec) slat (usec): min=39, max=24143, avg=6658.28, stdev=3015.98 clat (msec): min=6, max=504, avg=220.67, stdev=53.76 lat (msec): min=7, max=509, avg=227.33, stdev=54.52 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 148], 10.00th=[ 165], 20.00th=[ 182], | 30.00th=[ 197], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 230], | 70.00th=[ 245], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 380], 99.50th=[ 393], 99.90th=[ 460], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 40, max= 167, per=25.12%, avg=88.92, stdev=30.53, samples=112 iops : min= 10, max= 41, avg=21.87, stdev= 7.64, samples=112 write: IOPS=23, BW=92.1KiB/s (94.3kB/s)(5192KiB/56394msec); 0 zone resets slat (usec): min=31, max=6945, avg=199.98, stdev=372.37 clat (msec): min=54, max=770, avg=473.39, stdev=114.79 lat (msec): min=54, max=770, avg=473.59, stdev=114.81 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 284], 10.00th=[ 321], 20.00th=[ 376], | 30.00th=[ 418], 40.00th=[ 451], 50.00th=[ 477], 60.00th=[ 506], | 70.00th=[ 535], 80.00th=[ 575], 90.00th=[ 617], 95.00th=[ 659], | 99.00th=[ 735], 99.50th=[ 743], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 40, max= 160, per=24.74%, avg=91.29, stdev=25.75, samples=112 iops : min= 10, max= 40, avg=22.46, stdev= 6.51, samples=112 lat (msec) : 10=0.16%, 20=0.12%, 50=0.16%, 100=0.66%, 250=36.72% lat (msec) : 500=41.41%, 750=20.55%, 1000=0.23% cpu : usr=0.11%, sys=0.81%, ctx=5343, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155718: Tue Sep 8 07:45:24 2026 read: IOPS=21, BW=87.9KiB/s (90.0kB/s)(4976KiB/56588msec) slat (usec): min=2053, max=23727, avg=6645.50, stdev=2923.66 clat (msec): min=23, max=412, avg=220.15, stdev=50.01 lat (msec): min=27, max=418, avg=226.80, stdev=50.69 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 148], 10.00th=[ 165], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 205], 50.00th=[ 218], 60.00th=[ 230], | 70.00th=[ 245], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 359], 99.50th=[ 380], 99.90th=[ 401], 99.95th=[ 414], | 99.99th=[ 414] bw ( KiB/s): min= 23, max= 167, per=24.59%, avg=87.05, stdev=31.79, samples=112 iops : min= 5, max= 41, avg=21.41, stdev= 7.98, samples=112 write: IOPS=23, BW=93.0KiB/s (95.3kB/s)(5264KiB/56588msec); 0 zone resets slat (usec): min=24, max=6213, avg=180.96, stdev=311.98 clat (msec): min=112, max=798, avg=472.62, stdev=112.01 lat (msec): min=112, max=798, avg=472.80, stdev=112.00 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 288], 10.00th=[ 326], 20.00th=[ 376], | 30.00th=[ 418], 40.00th=[ 447], 50.00th=[ 472], 60.00th=[ 502], | 70.00th=[ 535], 80.00th=[ 575], 90.00th=[ 609], 95.00th=[ 659], | 99.00th=[ 735], 99.50th=[ 760], 99.90th=[ 793], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 32, max= 160, per=25.00%, avg=92.25, stdev=24.87, samples=112 iops : min= 8, max= 40, avg=22.71, stdev= 6.28, samples=112 lat (msec) : 50=0.23%, 100=0.20%, 250=36.56%, 500=42.54%, 750=20.20% lat (msec) : 1000=0.27% cpu : usr=0.14%, sys=0.80%, ctx=4995, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155719: Tue Sep 8 07:45:24 2026 read: IOPS=22, BW=88.2KiB/s (90.3kB/s)(4984KiB/56537msec) slat (usec): min=1747, max=23227, avg=6718.42, stdev=3017.78 clat (msec): min=16, max=405, avg=222.26, stdev=50.67 lat (msec): min=22, max=416, avg=228.99, stdev=51.31 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 150], 10.00th=[ 165], 20.00th=[ 184], | 30.00th=[ 194], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 232], | 70.00th=[ 247], 80.00th=[ 262], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 368], 99.50th=[ 384], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 23, max= 183, per=24.85%, avg=87.97, stdev=32.59, samples=112 iops : min= 5, max= 45, avg=21.63, stdev= 8.16, samples=112 write: IOPS=23, BW=92.0KiB/s (95.2kB/s)(5256KiB/56537msec); 0 zone resets slat (usec): min=23, max=3948, avg=179.85, stdev=289.50 clat (msec): min=152, max=791, avg=470.11, stdev=111.75 lat (msec): min=152, max=791, avg=470.29, stdev=111.76 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 284], 10.00th=[ 321], 20.00th=[ 376], | 30.00th=[ 414], 40.00th=[ 443], 50.00th=[ 468], 60.00th=[ 498], | 70.00th=[ 535], 80.00th=[ 575], 90.00th=[ 617], 95.00th=[ 642], | 99.00th=[ 726], 99.50th=[ 751], 99.90th=[ 776], 99.95th=[ 793], | 99.99th=[ 793] bw ( KiB/s): min= 24, max= 160, per=24.96%, avg=92.09, stdev=23.24, samples=112 iops : min= 6, max= 40, avg=22.67, stdev= 5.84, samples=112 lat (msec) : 20=0.04%, 50=0.23%, 100=0.16%, 250=35.66%, 500=43.52% lat (msec) : 750=20.08%, 1000=0.31% cpu : usr=0.11%, sys=0.83%, ctx=5136, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155720: Tue Sep 8 07:45:24 2026 read: IOPS=22, BW=89.5KiB/s (91.6kB/s)(5060KiB/56537msec) slat (usec): min=1967, max=32124, avg=6525.09, stdev=2968.16 clat (msec): min=9, max=400, avg=221.49, stdev=49.09 lat (msec): min=13, max=409, avg=228.02, stdev=49.65 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 150], 10.00th=[ 165], 20.00th=[ 184], | 30.00th=[ 194], 40.00th=[ 207], 50.00th=[ 218], 60.00th=[ 230], | 70.00th=[ 245], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 355], 99.50th=[ 372], 99.90th=[ 397], 99.95th=[ 401], | 99.99th=[ 401] bw ( KiB/s): min= 7, max= 168, per=25.19%, avg=89.18, stdev=29.18, samples=112 iops : min= 1, max= 42, avg=21.94, stdev= 7.34, samples=112 write: IOPS=22, BW=91.6KiB/s (93.8kB/s)(5180KiB/56537msec); 0 zone resets slat (usec): min=26, max=5350, avg=196.39, stdev=373.57 clat (msec): min=145, max=800, avg=474.44, stdev=110.50 lat (msec): min=145, max=801, avg=474.64, stdev=110.51 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 292], 10.00th=[ 326], 20.00th=[ 380], | 30.00th=[ 422], 40.00th=[ 451], 50.00th=[ 472], 60.00th=[ 502], | 70.00th=[ 531], 80.00th=[ 567], 90.00th=[ 609], 95.00th=[ 659], | 99.00th=[ 743], 99.50th=[ 751], 99.90th=[ 760], 99.95th=[ 802], | 99.99th=[ 802] bw ( KiB/s): min= 40, max= 168, per=24.65%, avg=90.97, stdev=21.72, samples=112 iops : min= 10, max= 42, avg=22.38, stdev= 5.49, samples=112 lat (msec) : 10=0.04%, 20=0.04%, 50=0.16%, 100=0.08%, 250=37.77% lat (msec) : 500=41.33%, 750=20.23%, 1000=0.35% cpu : usr=0.11%, sys=0.83%, ctx=5339, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=355KiB/s (363kB/s), 87.9KiB/s-89.5KiB/s (90.0kB/s-91.7kB/s), io=19.6MiB (20.5MB), run=56394-56588msec WRITE: bw=369KiB/s (378kB/s), 91.6KiB/s-93.0KiB/s (93.8kB/s-95.3kB/s), io=20.4MiB (21.4MB), run=56394-56588msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=155732: Tue Sep 8 07:45:29 2026 read: IOPS=5, BW=217MiB/s (228MB/s)(40.0MiB/184msec) slat (nsec): min=12996k, max=12996k, avg=12996015.00, stdev= 0.00 clat (nsec): min=158293k, max=158293k, avg=158292544.00, stdev= 0.00 lat (nsec): min=171292k, max=171292k, avg=171292136.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 159], 10.00th=[ 159], 20.00th=[ 159], | 30.00th=[ 159], 40.00th=[ 159], 50.00th=[ 159], 60.00th=[ 159], | 70.00th=[ 159], 80.00th=[ 159], 90.00th=[ 159], 95.00th=[ 159], | 99.00th=[ 159], 99.50th=[ 159], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.10%, ctx=5, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=217MiB/s (228MB/s), 217MiB/s-217MiB/s (228MB/s-228MB/s), io=40.0MiB (41.9MB), run=184-184msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout