== sanity test 398c: run fio to test AIO ================= 13:51:19 (1784051479) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.4376 s, 95.8 MB/s osc.lustre-OST0000-osc-ffff9b8f6cc10000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=181116: Tue Jul 14 13:51:58 2026 write: IOPS=81, BW=327KiB/s (335kB/s)(10.0MiB/31277msec); 0 zone resets slat (usec): min=51, max=17520, avg=370.97, stdev=1107.33 clat (msec): min=30, max=378, avg=194.41, stdev=49.15 lat (msec): min=31, max=378, avg=194.79, stdev=49.15 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 122], 10.00th=[ 140], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 178], 50.00th=[ 190], 60.00th=[ 205], | 70.00th=[ 215], 80.00th=[ 239], 90.00th=[ 259], 95.00th=[ 275], | 99.00th=[ 321], 99.50th=[ 334], 99.90th=[ 376], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 191, max= 510, per=24.83%, avg=325.00, stdev=71.84, samples=62 iops : min= 47, max= 127, avg=80.79, stdev=17.94, samples=62 lat (msec) : 50=0.08%, 100=2.73%, 250=83.12%, 500=14.06% cpu : usr=0.14%, sys=1.26%, ctx=2336, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181117: Tue Jul 14 13:51:58 2026 write: IOPS=81, BW=327KiB/s (335kB/s)(10.0MiB/31275msec); 0 zone resets slat (usec): min=51, max=15494, avg=404.82, stdev=1196.78 clat (msec): min=37, max=379, avg=194.02, stdev=48.58 lat (msec): min=37, max=379, avg=194.43, stdev=48.58 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 120], 10.00th=[ 138], 20.00th=[ 155], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 192], 60.00th=[ 205], | 70.00th=[ 215], 80.00th=[ 239], 90.00th=[ 259], 95.00th=[ 271], | 99.00th=[ 317], 99.50th=[ 347], 99.90th=[ 368], 99.95th=[ 376], | 99.99th=[ 380] bw ( KiB/s): min= 167, max= 510, per=24.83%, avg=325.05, stdev=72.86, samples=62 iops : min= 41, max= 127, avg=80.79, stdev=18.24, samples=62 lat (msec) : 50=0.16%, 100=3.05%, 250=82.70%, 500=14.10% cpu : usr=0.18%, sys=1.34%, ctx=2552, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181118: Tue Jul 14 13:51:58 2026 write: IOPS=81, BW=328KiB/s (335kB/s)(10.0MiB/31263msec); 0 zone resets slat (usec): min=49, max=13125, avg=376.30, stdev=943.62 clat (msec): min=26, max=379, avg=194.09, stdev=48.22 lat (msec): min=26, max=379, avg=194.47, stdev=48.22 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 123], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 165], 40.00th=[ 178], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 218], 80.00th=[ 236], 90.00th=[ 262], 95.00th=[ 271], | 99.00th=[ 313], 99.50th=[ 334], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 183, max= 510, per=24.77%, avg=324.28, stdev=70.03, samples=61 iops : min= 45, max= 127, avg=80.61, stdev=17.51, samples=61 lat (msec) : 50=0.04%, 100=2.27%, 250=83.91%, 500=13.79% cpu : usr=0.20%, sys=1.23%, ctx=2574, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181119: Tue Jul 14 13:51:58 2026 write: IOPS=82, BW=329KiB/s (337kB/s)(10.0MiB/31100msec); 0 zone resets slat (usec): min=50, max=29251, avg=398.04, stdev=1130.31 clat (msec): min=12, max=373, avg=193.84, stdev=48.63 lat (msec): min=12, max=373, avg=194.24, stdev=48.63 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 122], 10.00th=[ 136], 20.00th=[ 157], | 30.00th=[ 167], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 203], | 70.00th=[ 218], 80.00th=[ 236], 90.00th=[ 259], 95.00th=[ 271], | 99.00th=[ 326], 99.50th=[ 334], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 372] bw ( KiB/s): min= 191, max= 510, per=25.04%, avg=327.80, stdev=74.03, samples=61 iops : min= 47, max= 127, avg=81.48, stdev=18.53, samples=61 lat (msec) : 20=0.08%, 50=0.12%, 100=2.38%, 250=84.22%, 500=13.20% cpu : usr=0.21%, sys=1.25%, ctx=2676, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1310KiB/s (1341kB/s), 327KiB/s-329KiB/s (335kB/s-337kB/s), io=40.0MiB (41.9MB), run=31100-31277msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=181134: Tue Jul 14 13:52:39 2026 read: IOPS=40, BW=161KiB/s (165kB/s)(5048KiB/31297msec) slat (usec): min=3181, max=52134, avg=14552.64, stdev=7847.52 clat (msec): min=32, max=384, avg=183.32, stdev=47.17 lat (msec): min=43, max=425, avg=197.88, stdev=49.30 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 121], 10.00th=[ 132], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 203], 80.00th=[ 222], 90.00th=[ 249], 95.00th=[ 271], | 99.00th=[ 326], 99.50th=[ 359], 99.90th=[ 376], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 51, max= 272, per=25.01%, avg=160.29, stdev=42.05, samples=62 iops : min= 12, max= 68, avg=39.53, stdev=10.60, samples=62 write: IOPS=41, BW=166KiB/s (170kB/s)(5192KiB/31297msec); 0 zone resets slat (usec): min=56, max=17151, avg=390.01, stdev=1018.16 clat (msec): min=95, max=393, avg=189.04, stdev=45.16 lat (msec): min=95, max=393, avg=189.43, stdev=45.16 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 132], 10.00th=[ 140], 20.00th=[ 153], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 190], | 70.00th=[ 207], 80.00th=[ 224], 90.00th=[ 253], 95.00th=[ 275], | 99.00th=[ 326], 99.50th=[ 351], 99.90th=[ 376], 99.95th=[ 393], | 99.99th=[ 393] bw ( KiB/s): min= 36, max= 248, per=24.82%, avg=165.53, stdev=40.76, samples=62 iops : min= 9, max= 62, avg=40.87, stdev=10.21, samples=62 lat (msec) : 50=0.08%, 100=0.47%, 250=89.53%, 500=9.92% cpu : usr=0.26%, sys=2.10%, ctx=4219, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181135: Tue Jul 14 13:52:39 2026 read: IOPS=39, BW=159KiB/s (163kB/s)(4976KiB/31220msec) slat (usec): min=2913, max=54335, avg=14524.93, stdev=7754.22 clat (msec): min=75, max=368, avg=182.51, stdev=45.76 lat (msec): min=91, max=414, avg=197.04, stdev=48.07 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 121], 10.00th=[ 132], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 218], 90.00th=[ 247], 95.00th=[ 271], | 99.00th=[ 317], 99.50th=[ 347], 99.90th=[ 363], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 36, max= 232, per=24.71%, avg=158.39, stdev=41.42, samples=61 iops : min= 9, max= 58, avg=39.05, stdev=10.39, samples=61 write: IOPS=42, BW=169KiB/s (173kB/s)(5264KiB/31220msec); 0 zone resets slat (usec): min=57, max=15997, avg=379.43, stdev=889.97 clat (msec): min=46, max=435, avg=190.31, stdev=48.30 lat (msec): min=46, max=435, avg=190.70, stdev=48.28 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 131], 10.00th=[ 142], 20.00th=[ 153], | 30.00th=[ 163], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 205], 80.00th=[ 222], 90.00th=[ 257], 95.00th=[ 284], | 99.00th=[ 342], 99.50th=[ 384], 99.90th=[ 435], 99.95th=[ 435], | 99.99th=[ 435] bw ( KiB/s): min= 79, max= 255, per=25.18%, avg=167.92, stdev=44.41, samples=61 iops : min= 19, max= 63, avg=41.46, stdev=11.19, samples=61 lat (msec) : 50=0.04%, 100=0.51%, 250=88.95%, 500=10.51% cpu : usr=0.26%, sys=2.25%, ctx=4267, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181136: Tue Jul 14 13:52:39 2026 read: IOPS=40, BW=161KiB/s (165kB/s)(4984KiB/30919msec) slat (usec): min=89, max=69742, avg=14425.08, stdev=7982.99 clat (msec): min=39, max=380, avg=183.03, stdev=47.84 lat (msec): min=54, max=407, avg=197.46, stdev=50.12 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 120], 10.00th=[ 130], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 203], 80.00th=[ 220], 90.00th=[ 249], 95.00th=[ 275], | 99.00th=[ 321], 99.50th=[ 351], 99.90th=[ 376], 99.95th=[ 380], | 99.99th=[ 380] bw ( KiB/s): min= 79, max= 263, per=25.18%, avg=161.41, stdev=41.66, samples=61 iops : min= 19, max= 65, avg=39.80, stdev=10.48, samples=61 write: IOPS=42, BW=170KiB/s (174kB/s)(5256KiB/30919msec); 0 zone resets slat (usec): min=54, max=19090, avg=442.48, stdev=1163.85 clat (msec): min=95, max=388, avg=188.02, stdev=44.91 lat (msec): min=95, max=388, avg=188.47, stdev=44.93 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 130], 10.00th=[ 140], 20.00th=[ 153], | 30.00th=[ 161], 40.00th=[ 171], 50.00th=[ 180], 60.00th=[ 190], | 70.00th=[ 205], 80.00th=[ 224], 90.00th=[ 249], 95.00th=[ 275], | 99.00th=[ 334], 99.50th=[ 351], 99.90th=[ 372], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 55, max= 262, per=25.36%, avg=169.18, stdev=43.36, samples=61 iops : min= 13, max= 65, avg=41.75, stdev=10.92, samples=61 lat (msec) : 50=0.04%, 100=0.62%, 250=89.88%, 500=9.45% cpu : usr=0.29%, sys=2.13%, ctx=4296, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181137: Tue Jul 14 13:52:39 2026 read: IOPS=40, BW=162KiB/s (166kB/s)(5060KiB/31275msec) slat (usec): min=2787, max=57641, avg=14335.87, stdev=7623.29 clat (msec): min=41, max=386, avg=183.83, stdev=46.38 lat (msec): min=70, max=428, avg=198.17, stdev=48.41 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 122], 10.00th=[ 132], 20.00th=[ 146], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 222], 90.00th=[ 251], 95.00th=[ 271], | 99.00th=[ 317], 99.50th=[ 334], 99.90th=[ 380], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 51, max= 271, per=25.11%, avg=160.95, stdev=42.27, samples=62 iops : min= 12, max= 67, avg=39.69, stdev=10.61, samples=62 write: IOPS=41, BW=166KiB/s (170kB/s)(5180KiB/31275msec); 0 zone resets slat (usec): min=56, max=6001, avg=372.26, stdev=660.81 clat (msec): min=41, max=410, avg=189.57, stdev=45.90 lat (msec): min=41, max=410, avg=189.94, stdev=45.85 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 130], 10.00th=[ 138], 20.00th=[ 150], | 30.00th=[ 163], 40.00th=[ 171], 50.00th=[ 184], 60.00th=[ 194], | 70.00th=[ 209], 80.00th=[ 224], 90.00th=[ 253], 95.00th=[ 279], | 99.00th=[ 326], 99.50th=[ 338], 99.90th=[ 393], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 73, max= 264, per=24.71%, avg=164.84, stdev=40.71, samples=62 iops : min= 18, max= 66, avg=40.69, stdev=10.26, samples=62 lat (msec) : 50=0.08%, 100=0.39%, 250=89.34%, 500=10.20% cpu : usr=0.27%, sys=2.24%, ctx=4265, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=641KiB/s (657kB/s), 159KiB/s-162KiB/s (163kB/s-166kB/s), io=19.6MiB (20.5MB), run=30919-31297msec WRITE: bw=668KiB/s (684kB/s), 166KiB/s-170KiB/s (170kB/s-174kB/s), io=20.4MiB (21.4MB), run=30919-31297msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=181145: Tue Jul 14 13:52:49 2026 read: IOPS=2, BW=93.9MiB/s (98.5MB/s)(40.0MiB/426msec) slat (nsec): min=34541k, max=34541k, avg=34540851.00, stdev= 0.00 clat (nsec): min=372504k, max=372504k, avg=372503906.00, stdev= 0.00 lat (nsec): min=407049k, max=407049k, avg=407048961.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 372], 10.00th=[ 372], 20.00th=[ 372], | 30.00th=[ 372], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.06%, ctx=4, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=93.9MiB/s (98.5MB/s), 93.9MiB/s-93.9MiB/s (98.5MB/s-98.5MB/s), io=40.0MiB (41.9MB), run=426-426msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout