== sanity test 398c: run fio to test AIO ================= 03:24:07 (1780385047) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.399189 s, 105 MB/s osc.lustre-OST0000-osc-ffff95532f31e800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146617: Tue Jun 2 03:26:04 2026 write: IOPS=23, BW=92.0KiB/s (95.2kB/s)(10.0MiB/110115msec); 0 zone resets slat (usec): min=37, max=7985, avg=329.91, stdev=653.21 clat (msec): min=138, max=1148, avg=687.15, stdev=156.91 lat (msec): min=138, max=1149, avg=687.48, stdev=156.94 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 409], 10.00th=[ 502], 20.00th=[ 567], | 30.00th=[ 617], 40.00th=[ 667], 50.00th=[ 709], 60.00th=[ 735], | 70.00th=[ 768], 80.00th=[ 802], 90.00th=[ 869], 95.00th=[ 927], | 99.00th=[ 1028], 99.50th=[ 1070], 99.90th=[ 1133], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 16, max= 255, per=24.85%, avg=92.21, stdev=29.15, samples=219 iops : min= 4, max= 63, avg=22.64, stdev= 7.34, samples=219 lat (msec) : 250=1.56%, 500=8.40%, 750=54.80%, 1000=33.59%, 2000=1.64% cpu : usr=0.06%, sys=0.38%, ctx=2993, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146618: Tue Jun 2 03:26:04 2026 write: IOPS=23, BW=92.0KiB/s (95.2kB/s)(10.0MiB/110117msec); 0 zone resets slat (usec): min=42, max=12713, avg=310.94, stdev=640.50 clat (msec): min=62, max=1249, avg=687.59, stdev=160.05 lat (msec): min=63, max=1249, avg=687.91, stdev=160.07 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 414], 10.00th=[ 502], 20.00th=[ 567], | 30.00th=[ 625], 40.00th=[ 667], 50.00th=[ 701], 60.00th=[ 735], | 70.00th=[ 768], 80.00th=[ 810], 90.00th=[ 869], 95.00th=[ 919], | 99.00th=[ 1045], 99.50th=[ 1083], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 271, per=24.84%, avg=92.14, stdev=29.95, samples=219 iops : min= 1, max= 67, avg=22.62, stdev= 7.51, samples=219 lat (msec) : 100=0.16%, 250=1.56%, 500=8.20%, 750=55.27%, 1000=32.42% lat (msec) : 2000=2.38% cpu : usr=0.06%, sys=0.39%, ctx=2836, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146619: Tue Jun 2 03:26:04 2026 write: IOPS=23, BW=92.0KiB/s (95.2kB/s)(10.0MiB/110116msec); 0 zone resets slat (usec): min=40, max=9346, avg=311.86, stdev=664.69 clat (msec): min=91, max=1144, avg=687.55, stdev=156.14 lat (msec): min=92, max=1145, avg=687.86, stdev=156.16 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 418], 10.00th=[ 506], 20.00th=[ 567], | 30.00th=[ 625], 40.00th=[ 659], 50.00th=[ 701], 60.00th=[ 743], | 70.00th=[ 768], 80.00th=[ 802], 90.00th=[ 869], 95.00th=[ 927], | 99.00th=[ 1045], 99.50th=[ 1070], 99.90th=[ 1116], 99.95th=[ 1116], | 99.99th=[ 1150] bw ( KiB/s): min= 31, max= 247, per=24.85%, avg=92.20, stdev=29.73, samples=219 iops : min= 7, max= 61, avg=22.63, stdev= 7.48, samples=219 lat (msec) : 100=0.04%, 250=1.41%, 500=8.05%, 750=54.65%, 1000=33.67% lat (msec) : 2000=2.19% cpu : usr=0.05%, sys=0.40%, ctx=2727, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146620: Tue Jun 2 03:26:04 2026 write: IOPS=23, BW=92.0KiB/s (95.2kB/s)(10.0MiB/110114msec); 0 zone resets slat (usec): min=40, max=10247, avg=332.48, stdev=659.67 clat (msec): min=140, max=1151, avg=687.30, stdev=156.89 lat (msec): min=140, max=1151, avg=687.63, stdev=156.90 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 414], 10.00th=[ 506], 20.00th=[ 567], | 30.00th=[ 625], 40.00th=[ 667], 50.00th=[ 701], 60.00th=[ 735], | 70.00th=[ 768], 80.00th=[ 802], 90.00th=[ 860], 95.00th=[ 944], | 99.00th=[ 1036], 99.50th=[ 1083], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 23, max= 255, per=24.85%, avg=92.19, stdev=31.88, samples=219 iops : min= 5, max= 63, avg=22.64, stdev= 8.00, samples=219 lat (msec) : 250=1.52%, 500=7.66%, 750=56.25%, 1000=32.81%, 2000=1.76% cpu : usr=0.06%, sys=0.39%, ctx=2939, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=372KiB/s (381kB/s), 92.0KiB/s-92.0KiB/s (95.2kB/s-95.2kB/s), io=40.0MiB (41.9MB), run=110114-110117msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146645: Tue Jun 2 03:27:18 2026 read: IOPS=18, BW=73.7KiB/s (75.5kB/s)(5048KiB/68476msec) slat (usec): min=2232, max=31890, avg=8724.09, stdev=3912.98 clat (msec): min=7, max=570, avg=268.00, stdev=61.95 lat (msec): min=14, max=574, avg=276.74, stdev=62.62 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 188], 10.00th=[ 203], 20.00th=[ 222], | 30.00th=[ 234], 40.00th=[ 249], 50.00th=[ 264], 60.00th=[ 279], | 70.00th=[ 296], 80.00th=[ 313], 90.00th=[ 342], 95.00th=[ 376], | 99.00th=[ 430], 99.50th=[ 460], 99.90th=[ 567], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 15, max= 143, per=25.11%, avg=73.33, stdev=24.70, samples=136 iops : min= 3, max= 35, avg=18.01, stdev= 6.21, samples=136 write: IOPS=18, BW=75.8KiB/s (77.6kB/s)(5192KiB/68476msec); 0 zone resets slat (usec): min=57, max=8138, avg=357.63, stdev=622.11 clat (msec): min=211, max=1084, avg=571.66, stdev=140.00 lat (msec): min=212, max=1085, avg=572.03, stdev=140.01 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 363], 10.00th=[ 401], 20.00th=[ 447], | 30.00th=[ 485], 40.00th=[ 527], 50.00th=[ 567], 60.00th=[ 600], | 70.00th=[ 642], 80.00th=[ 684], 90.00th=[ 751], 95.00th=[ 835], | 99.00th=[ 936], 99.50th=[ 969], 99.90th=[ 1070], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 160, per=24.82%, avg=75.44, stdev=22.62, samples=136 iops : min= 1, max= 40, avg=18.54, stdev= 5.72, samples=136 lat (msec) : 10=0.12%, 20=0.08%, 50=0.20%, 100=0.04%, 250=19.61% lat (msec) : 500=46.05%, 750=28.87%, 1000=4.80%, 2000=0.23% cpu : usr=0.11%, sys=0.90%, ctx=5372, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146646: Tue Jun 2 03:27:18 2026 read: IOPS=18, BW=72.5KiB/s (74.2kB/s)(4976KiB/68642msec) slat (usec): min=60, max=38873, avg=8498.50, stdev=4063.82 clat (msec): min=8, max=595, avg=270.70, stdev=64.56 lat (msec): min=15, max=598, avg=279.20, stdev=65.24 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 184], 10.00th=[ 201], 20.00th=[ 222], | 30.00th=[ 236], 40.00th=[ 251], 50.00th=[ 264], 60.00th=[ 279], | 70.00th=[ 296], 80.00th=[ 317], 90.00th=[ 351], 95.00th=[ 384], | 99.00th=[ 472], 99.50th=[ 498], 99.90th=[ 575], 99.95th=[ 592], | 99.99th=[ 592] bw ( KiB/s): min= 15, max= 160, per=24.73%, avg=72.20, stdev=27.78, samples=136 iops : min= 3, max= 40, avg=17.73, stdev= 6.94, samples=136 write: IOPS=19, BW=76.7KiB/s (78.5kB/s)(5264KiB/68642msec); 0 zone resets slat (usec): min=60, max=6360, avg=306.13, stdev=433.77 clat (msec): min=121, max=1069, avg=569.75, stdev=145.28 lat (msec): min=121, max=1069, avg=570.06, stdev=145.30 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 359], 10.00th=[ 397], 20.00th=[ 443], | 30.00th=[ 493], 40.00th=[ 527], 50.00th=[ 567], 60.00th=[ 600], | 70.00th=[ 634], 80.00th=[ 676], 90.00th=[ 751], 95.00th=[ 852], | 99.00th=[ 936], 99.50th=[ 961], 99.90th=[ 1036], 99.95th=[ 1070], | 99.99th=[ 1070] bw ( KiB/s): min= 31, max= 152, per=25.13%, avg=76.39, stdev=22.91, samples=136 iops : min= 7, max= 38, avg=18.78, stdev= 5.82, samples=136 lat (msec) : 10=0.04%, 20=0.04%, 50=0.04%, 100=0.16%, 250=19.65% lat (msec) : 500=45.39%, 750=29.34%, 1000=5.23%, 2000=0.12% cpu : usr=0.11%, sys=0.93%, ctx=5195, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146647: Tue Jun 2 03:27:18 2026 read: IOPS=18, BW=72.5KiB/s (74.3kB/s)(4984KiB/68719msec) slat (usec): min=2563, max=32771, avg=8692.07, stdev=4102.10 clat (msec): min=6, max=567, avg=267.51, stdev=58.74 lat (msec): min=9, max=572, avg=276.20, stdev=59.44 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 184], 10.00th=[ 199], 20.00th=[ 220], | 30.00th=[ 234], 40.00th=[ 251], 50.00th=[ 264], 60.00th=[ 279], | 70.00th=[ 296], 80.00th=[ 313], 90.00th=[ 342], 95.00th=[ 372], | 99.00th=[ 418], 99.50th=[ 447], 99.90th=[ 506], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 15, max= 152, per=25.07%, avg=73.20, stdev=29.88, samples=134 iops : min= 3, max= 38, avg=17.99, stdev= 7.50, samples=134 write: IOPS=19, BW=76.5KiB/s (78.3kB/s)(5256KiB/68719msec); 0 zone resets slat (usec): min=57, max=10437, avg=345.23, stdev=631.38 clat (msec): min=147, max=1088, avg=572.10, stdev=147.07 lat (msec): min=148, max=1089, avg=572.45, stdev=147.07 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 355], 10.00th=[ 393], 20.00th=[ 443], | 30.00th=[ 485], 40.00th=[ 527], 50.00th=[ 567], 60.00th=[ 609], | 70.00th=[ 642], 80.00th=[ 684], 90.00th=[ 760], 95.00th=[ 860], | 99.00th=[ 936], 99.50th=[ 986], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 23, max= 143, per=24.87%, avg=75.60, stdev=23.18, samples=136 iops : min= 5, max= 35, avg=18.58, stdev= 5.86, samples=136 lat (msec) : 10=0.12%, 100=0.08%, 250=19.57%, 500=46.25%, 750=28.67% lat (msec) : 1000=5.08%, 2000=0.23% cpu : usr=0.13%, sys=0.92%, ctx=5101, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146648: Tue Jun 2 03:27:18 2026 read: IOPS=18, BW=73.7KiB/s (75.4kB/s)(5060KiB/68686msec) slat (usec): min=2557, max=31990, avg=8586.76, stdev=3875.45 clat (msec): min=44, max=644, avg=266.90, stdev=61.79 lat (msec): min=50, max=648, avg=275.50, stdev=62.38 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 184], 10.00th=[ 201], 20.00th=[ 218], | 30.00th=[ 234], 40.00th=[ 247], 50.00th=[ 259], 60.00th=[ 275], | 70.00th=[ 292], 80.00th=[ 309], 90.00th=[ 347], 95.00th=[ 376], | 99.00th=[ 460], 99.50th=[ 514], 99.90th=[ 558], 99.95th=[ 642], | 99.99th=[ 642] bw ( KiB/s): min= 7, max= 160, per=25.22%, avg=73.63, stdev=27.39, samples=136 iops : min= 1, max= 40, avg=18.09, stdev= 6.92, samples=136 write: IOPS=18, BW=75.4KiB/s (77.2kB/s)(5180KiB/68686msec); 0 zone resets slat (usec): min=58, max=11597, avg=337.29, stdev=584.41 clat (msec): min=122, max=1050, avg=576.52, stdev=144.99 lat (msec): min=123, max=1051, avg=576.86, stdev=144.98 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 363], 10.00th=[ 401], 20.00th=[ 451], | 30.00th=[ 489], 40.00th=[ 527], 50.00th=[ 575], 60.00th=[ 609], | 70.00th=[ 642], 80.00th=[ 684], 90.00th=[ 760], 95.00th=[ 844], | 99.00th=[ 969], 99.50th=[ 1011], 99.90th=[ 1045], 99.95th=[ 1053], | 99.99th=[ 1053] bw ( KiB/s): min= 31, max= 128, per=24.69%, avg=75.05, stdev=21.40, samples=136 iops : min= 7, max= 32, avg=18.44, stdev= 5.38, samples=136 lat (msec) : 50=0.04%, 100=0.20%, 250=20.70%, 500=45.55%, 750=27.89% lat (msec) : 1000=5.31%, 2000=0.31% cpu : usr=0.11%, sys=0.89%, ctx=5279, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=292KiB/s (299kB/s), 72.5KiB/s-73.7KiB/s (74.2kB/s-75.5kB/s), io=19.6MiB (20.5MB), run=68476-68719msec WRITE: bw=304KiB/s (311kB/s), 75.4KiB/s-76.7KiB/s (77.2kB/s-78.5kB/s), io=20.4MiB (21.4MB), run=68476-68719msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146662: Tue Jun 2 03:27:25 2026 read: IOPS=2, BW=118MiB/s (124MB/s)(40.0MiB/339msec) slat (nsec): min=33758k, max=33758k, avg=33757625.00, stdev= 0.00 clat (nsec): min=272013k, max=272013k, avg=272012890.00, stdev= 0.00 lat (nsec): min=305775k, max=305775k, avg=305775007.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.66%, ctx=13, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=118MiB/s (124MB/s), 118MiB/s-118MiB/s (124MB/s-124MB/s), io=40.0MiB (41.9MB), run=339-339msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout