== sanity test 398c: run fio to test AIO ================= 10:46:45 (1781621205) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.221821 s, 189 MB/s osc.lustre-OST0000-osc-ffff972f50888800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=187163: Tue Jun 16 10:47:07 2026 write: IOPS=160, BW=641KiB/s (656kB/s)(10.0MiB/15975msec); 0 zone resets slat (usec): min=38, max=7000, avg=229.32, stdev=395.73 clat (msec): min=7, max=162, avg=99.01, stdev=20.01 lat (msec): min=7, max=162, avg=99.25, stdev=20.02 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 72], 10.00th=[ 77], 20.00th=[ 83], | 30.00th=[ 87], 40.00th=[ 91], 50.00th=[ 96], 60.00th=[ 102], | 70.00th=[ 109], 80.00th=[ 116], 90.00th=[ 127], 95.00th=[ 136], | 99.00th=[ 157], 99.50th=[ 161], 99.90th=[ 163], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 502, max= 772, per=24.97%, avg=640.16, stdev=75.44, samples=31 iops : min= 125, max= 193, avg=159.74, stdev=18.85, samples=31 lat (msec) : 10=0.04%, 20=0.08%, 50=0.04%, 100=58.09%, 250=41.76% cpu : usr=0.26%, sys=2.07%, ctx=2668, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=187164: Tue Jun 16 10:47:07 2026 write: IOPS=160, BW=642KiB/s (657kB/s)(10.0MiB/15953msec); 0 zone resets slat (usec): min=36, max=9941, avg=247.56, stdev=452.32 clat (msec): min=22, max=178, avg=98.81, stdev=20.00 lat (msec): min=22, max=178, avg=99.06, stdev=19.99 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 71], 10.00th=[ 77], 20.00th=[ 83], | 30.00th=[ 87], 40.00th=[ 91], 50.00th=[ 95], 60.00th=[ 102], | 70.00th=[ 109], 80.00th=[ 116], 90.00th=[ 126], 95.00th=[ 136], | 99.00th=[ 155], 99.50th=[ 161], 99.90th=[ 167], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 447, max= 768, per=25.01%, avg=641.19, stdev=80.58, samples=31 iops : min= 111, max= 192, avg=160.00, stdev=20.19, samples=31 lat (msec) : 50=0.12%, 100=58.32%, 250=41.56% cpu : usr=0.35%, sys=2.06%, ctx=2688, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=187165: Tue Jun 16 10:47:07 2026 write: IOPS=160, BW=643KiB/s (659kB/s)(10.0MiB/15921msec); 0 zone resets slat (usec): min=37, max=6985, avg=254.21, stdev=444.27 clat (msec): min=9, max=178, avg=98.71, stdev=20.58 lat (msec): min=9, max=178, avg=98.97, stdev=20.60 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 71], 10.00th=[ 75], 20.00th=[ 81], | 30.00th=[ 87], 40.00th=[ 92], 50.00th=[ 96], 60.00th=[ 102], | 70.00th=[ 109], 80.00th=[ 116], 90.00th=[ 126], 95.00th=[ 134], | 99.00th=[ 159], 99.50th=[ 163], 99.90th=[ 178], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min= 518, max= 764, per=25.07%, avg=642.74, stdev=64.90, samples=31 iops : min= 129, max= 191, avg=160.39, stdev=16.21, samples=31 lat (msec) : 10=0.04%, 50=0.08%, 100=57.23%, 250=42.66% cpu : usr=0.35%, sys=2.26%, ctx=3007, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=187166: Tue Jun 16 10:47:07 2026 write: IOPS=162, BW=648KiB/s (664kB/s)(10.0MiB/15796msec); 0 zone resets slat (usec): min=35, max=8771, avg=269.89, stdev=484.25 clat (msec): min=11, max=201, avg=98.32, stdev=21.17 lat (msec): min=12, max=201, avg=98.59, stdev=21.17 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 70], 10.00th=[ 75], 20.00th=[ 82], | 30.00th=[ 87], 40.00th=[ 91], 50.00th=[ 96], 60.00th=[ 102], | 70.00th=[ 108], 80.00th=[ 115], 90.00th=[ 126], 95.00th=[ 138], | 99.00th=[ 159], 99.50th=[ 163], 99.90th=[ 171], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 495, max= 810, per=25.20%, avg=646.23, stdev=68.56, samples=31 iops : min= 123, max= 202, avg=161.23, stdev=17.15, samples=31 lat (msec) : 20=0.16%, 50=0.66%, 100=57.11%, 250=42.07% cpu : usr=0.25%, sys=2.45%, ctx=2899, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2564KiB/s (2626kB/s), 641KiB/s-648KiB/s (656kB/s-664kB/s), io=40.0MiB (41.9MB), run=15796-15975msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=187178: Tue Jun 16 10:47:30 2026 read: IOPS=70, BW=281KiB/s (288kB/s)(5048KiB/17944msec) slat (usec): min=49, max=40672, avg=7943.96, stdev=4355.06 clat (msec): min=44, max=227, avg=106.98, stdev=25.87 lat (msec): min=54, max=238, avg=114.93, stdev=26.86 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 73], 10.00th=[ 79], 20.00th=[ 85], | 30.00th=[ 91], 40.00th=[ 97], 50.00th=[ 104], 60.00th=[ 110], | 70.00th=[ 118], 80.00th=[ 128], 90.00th=[ 142], 95.00th=[ 153], | 99.00th=[ 190], 99.50th=[ 197], 99.90th=[ 226], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 186, max= 352, per=25.31%, avg=280.97, stdev=42.09, samples=35 iops : min= 46, max= 88, avg=69.89, stdev=10.57, samples=35 write: IOPS=72, BW=289KiB/s (296kB/s)(5192KiB/17944msec); 0 zone resets slat (usec): min=38, max=8946, avg=298.91, stdev=572.70 clat (msec): min=53, max=221, avg=108.79, stdev=24.42 lat (msec): min=54, max=221, avg=109.09, stdev=24.39 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 78], 10.00th=[ 83], 20.00th=[ 90], | 30.00th=[ 94], 40.00th=[ 99], 50.00th=[ 104], 60.00th=[ 110], | 70.00th=[ 118], 80.00th=[ 129], 90.00th=[ 144], 95.00th=[ 155], | 99.00th=[ 188], 99.50th=[ 199], 99.90th=[ 220], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 174, max= 381, per=25.08%, avg=289.89, stdev=56.83, samples=35 iops : min= 43, max= 95, avg=72.14, stdev=14.24, samples=35 lat (msec) : 50=0.04%, 100=43.91%, 250=56.05% cpu : usr=0.46%, sys=2.94%, ctx=4717, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=187179: Tue Jun 16 10:47:30 2026 read: IOPS=68, BW=275KiB/s (282kB/s)(4976KiB/18070msec) slat (usec): min=1848, max=56580, avg=8047.96, stdev=4284.46 clat (msec): min=17, max=214, avg=106.26, stdev=25.39 lat (msec): min=21, max=237, avg=114.31, stdev=26.68 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 73], 10.00th=[ 78], 20.00th=[ 85], | 30.00th=[ 91], 40.00th=[ 96], 50.00th=[ 103], 60.00th=[ 110], | 70.00th=[ 117], 80.00th=[ 129], 90.00th=[ 142], 95.00th=[ 150], | 99.00th=[ 176], 99.50th=[ 186], 99.90th=[ 207], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 103, max= 383, per=24.83%, avg=275.60, stdev=56.66, samples=35 iops : min= 25, max= 95, avg=68.54, stdev=14.21, samples=35 write: IOPS=72, BW=291KiB/s (298kB/s)(5264KiB/18070msec); 0 zone resets slat (usec): min=46, max=11492, avg=310.45, stdev=687.19 clat (msec): min=59, max=228, avg=109.70, stdev=24.01 lat (msec): min=59, max=228, avg=110.01, stdev=24.03 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 79], 10.00th=[ 83], 20.00th=[ 90], | 30.00th=[ 95], 40.00th=[ 101], 50.00th=[ 105], 60.00th=[ 112], | 70.00th=[ 122], 80.00th=[ 130], 90.00th=[ 142], 95.00th=[ 153], | 99.00th=[ 186], 99.50th=[ 190], 99.90th=[ 220], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 161, max= 420, per=25.22%, avg=291.57, stdev=60.85, samples=35 iops : min= 40, max= 105, avg=72.57, stdev=15.25, samples=35 lat (msec) : 20=0.04%, 100=43.83%, 250=56.13% cpu : usr=0.31%, sys=2.94%, ctx=4772, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=187180: Tue Jun 16 10:47:30 2026 read: IOPS=68, BW=276KiB/s (282kB/s)(4984KiB/18071msec) slat (usec): min=1844, max=52090, avg=7918.35, stdev=4160.98 clat (msec): min=54, max=223, avg=106.12, stdev=25.02 lat (msec): min=58, max=231, avg=114.05, stdev=26.18 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 73], 10.00th=[ 79], 20.00th=[ 85], | 30.00th=[ 90], 40.00th=[ 96], 50.00th=[ 103], 60.00th=[ 109], | 70.00th=[ 117], 80.00th=[ 127], 90.00th=[ 140], 95.00th=[ 157], | 99.00th=[ 171], 99.50th=[ 180], 99.90th=[ 213], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 96, max= 407, per=24.95%, avg=276.94, stdev=58.87, samples=35 iops : min= 24, max= 101, avg=68.86, stdev=14.65, samples=35 write: IOPS=72, BW=291KiB/s (298kB/s)(5256KiB/18071msec); 0 zone resets slat (usec): min=39, max=11975, avg=263.53, stdev=489.24 clat (msec): min=34, max=212, avg=109.54, stdev=24.91 lat (msec): min=34, max=212, avg=109.81, stdev=24.91 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 77], 10.00th=[ 83], 20.00th=[ 89], | 30.00th=[ 94], 40.00th=[ 99], 50.00th=[ 105], 60.00th=[ 111], | 70.00th=[ 120], 80.00th=[ 131], 90.00th=[ 144], 95.00th=[ 155], | 99.00th=[ 186], 99.50th=[ 194], 99.90th=[ 209], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 135, max= 407, per=25.13%, avg=290.49, stdev=66.55, samples=35 iops : min= 33, max= 101, avg=72.26, stdev=16.68, samples=35 lat (msec) : 50=0.04%, 100=44.92%, 250=55.04% cpu : usr=0.40%, sys=2.89%, ctx=4680, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=187181: Tue Jun 16 10:47:30 2026 read: IOPS=70, BW=281KiB/s (288kB/s)(5060KiB/17993msec) slat (usec): min=90, max=44530, avg=7712.33, stdev=4084.06 clat (msec): min=8, max=212, avg=105.77, stdev=25.93 lat (msec): min=9, max=219, avg=113.49, stdev=26.95 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 74], 10.00th=[ 78], 20.00th=[ 84], | 30.00th=[ 90], 40.00th=[ 95], 50.00th=[ 103], 60.00th=[ 108], | 70.00th=[ 115], 80.00th=[ 126], 90.00th=[ 142], 95.00th=[ 157], | 99.00th=[ 178], 99.50th=[ 186], 99.90th=[ 209], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 151, max= 384, per=25.39%, avg=281.86, stdev=57.88, samples=35 iops : min= 37, max= 96, avg=70.11, stdev=14.52, samples=35 write: IOPS=71, BW=288KiB/s (295kB/s)(5180KiB/17993msec); 0 zone resets slat (usec): min=38, max=8457, avg=267.93, stdev=451.18 clat (msec): min=35, max=212, avg=109.79, stdev=25.20 lat (msec): min=36, max=212, avg=110.06, stdev=25.19 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 78], 10.00th=[ 83], 20.00th=[ 89], | 30.00th=[ 95], 40.00th=[ 100], 50.00th=[ 105], 60.00th=[ 112], | 70.00th=[ 121], 80.00th=[ 131], 90.00th=[ 146], 95.00th=[ 157], | 99.00th=[ 184], 99.50th=[ 194], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 175, max= 395, per=24.92%, avg=288.09, stdev=51.84, samples=35 iops : min= 43, max= 98, avg=71.66, stdev=12.95, samples=35 lat (msec) : 10=0.04%, 20=0.04%, 50=0.08%, 100=44.26%, 250=55.59% cpu : usr=0.39%, sys=2.86%, ctx=4608, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1111KiB/s (1137kB/s), 275KiB/s-281KiB/s (282kB/s-288kB/s), io=19.6MiB (20.5MB), run=17944-18071msec WRITE: bw=1156KiB/s (1184kB/s), 288KiB/s-291KiB/s (295kB/s-298kB/s), io=20.4MiB (21.4MB), run=17944-18071msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=187188: Tue Jun 16 10:47:36 2026 read: IOPS=3, BW=129MiB/s (135MB/s)(40.0MiB/310msec) slat (nsec): min=18235k, max=18235k, avg=18235184.00, stdev= 0.00 clat (nsec): min=276511k, max=276511k, avg=276511439.00, stdev= 0.00 lat (nsec): min=294772k, max=294772k, avg=294772273.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 275], 10.00th=[ 275], 20.00th=[ 275], | 30.00th=[ 275], 40.00th=[ 275], 50.00th=[ 275], 60.00th=[ 275], | 70.00th=[ 275], 80.00th=[ 275], 90.00th=[ 275], 95.00th=[ 275], | 99.00th=[ 275], 99.50th=[ 275], 99.90th=[ 275], 99.95th=[ 275], | 99.99th=[ 275] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.88%, ctx=5, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=40.0MiB (41.9MB), run=310-310msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout