== sanity test 398c: run fio to test AIO ================= 10:35:42 (1787582142) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.402478 s, 104 MB/s osc.lustre-OST0000-osc-ffff9259c4d1d800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=180135: Mon Aug 24 10:36:21 2026 write: IOPS=87, BW=349KiB/s (358kB/s)(10.0MiB/29305msec); 0 zone resets slat (usec): min=27, max=13609, avg=248.67, stdev=729.77 clat (msec): min=27, max=357, avg=182.71, stdev=41.16 lat (msec): min=28, max=357, avg=182.96, stdev=41.13 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 123], 10.00th=[ 136], 20.00th=[ 150], | 30.00th=[ 159], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 218], 90.00th=[ 241], 95.00th=[ 257], | 99.00th=[ 279], 99.50th=[ 288], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 359] bw ( KiB/s): min= 245, max= 534, per=25.02%, avg=347.48, stdev=63.43, samples=58 iops : min= 61, max= 133, avg=86.38, stdev=15.87, samples=58 lat (msec) : 50=0.16%, 100=1.48%, 250=90.70%, 500=7.66% cpu : usr=0.23%, sys=1.00%, ctx=2339, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180136: Mon Aug 24 10:36:21 2026 write: IOPS=87, BW=348KiB/s (356kB/s)(10.0MiB/29425msec); 0 zone resets slat (usec): min=28, max=10235, avg=201.82, stdev=615.54 clat (msec): min=89, max=368, avg=182.89, stdev=40.91 lat (msec): min=89, max=368, avg=183.10, stdev=40.90 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 125], 10.00th=[ 136], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 188], | 70.00th=[ 203], 80.00th=[ 220], 90.00th=[ 241], 95.00th=[ 257], | 99.00th=[ 288], 99.50th=[ 292], 99.90th=[ 300], 99.95th=[ 313], | 99.99th=[ 368] bw ( KiB/s): min= 253, max= 502, per=25.01%, avg=347.45, stdev=61.16, samples=58 iops : min= 63, max= 125, avg=86.38, stdev=15.29, samples=58 lat (msec) : 100=1.37%, 250=91.60%, 500=7.03% cpu : usr=0.21%, sys=0.84%, ctx=1366, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180137: Mon Aug 24 10:36:21 2026 write: IOPS=86, BW=347KiB/s (356kB/s)(10.0MiB/29478msec); 0 zone resets slat (usec): min=29, max=15147, avg=232.95, stdev=699.02 clat (msec): min=21, max=304, avg=183.30, stdev=40.23 lat (msec): min=21, max=304, avg=183.53, stdev=40.21 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 118], 10.00th=[ 138], 20.00th=[ 153], | 30.00th=[ 163], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 218], 90.00th=[ 232], 95.00th=[ 255], | 99.00th=[ 292], 99.50th=[ 300], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 253, max= 479, per=24.95%, avg=346.62, stdev=56.40, samples=58 iops : min= 63, max= 119, avg=86.17, stdev=14.06, samples=58 lat (msec) : 50=0.08%, 100=1.99%, 250=92.27%, 500=5.66% cpu : usr=0.22%, sys=0.97%, ctx=2551, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=180138: Mon Aug 24 10:36:21 2026 write: IOPS=86, BW=347KiB/s (356kB/s)(10.0MiB/29486msec); 0 zone resets slat (usec): min=28, max=11227, avg=282.57, stdev=869.74 clat (msec): min=24, max=343, avg=183.14, stdev=40.17 lat (msec): min=25, max=344, avg=183.42, stdev=40.16 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 124], 10.00th=[ 138], 20.00th=[ 153], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 215], 90.00th=[ 239], 95.00th=[ 255], | 99.00th=[ 275], 99.50th=[ 288], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 347] bw ( KiB/s): min= 213, max= 502, per=24.91%, avg=346.05, stdev=63.19, samples=58 iops : min= 53, max= 125, avg=86.03, stdev=15.79, samples=58 lat (msec) : 50=0.08%, 100=2.03%, 250=91.68%, 500=6.21% cpu : usr=0.22%, sys=1.14%, ctx=2586, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1389KiB/s (1422kB/s), 347KiB/s-349KiB/s (356kB/s-358kB/s), io=40.0MiB (41.9MB), run=29305-29486msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=180152: Mon Aug 24 10:37:05 2026 read: IOPS=36, BW=145KiB/s (148kB/s)(5048KiB/34894msec) slat (usec): min=29, max=59123, avg=14883.42, stdev=8175.94 clat (msec): min=34, max=404, avg=208.31, stdev=57.58 lat (msec): min=40, max=425, avg=223.20, stdev=59.97 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 123], 10.00th=[ 140], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 192], 50.00th=[ 207], 60.00th=[ 220], | 70.00th=[ 234], 80.00th=[ 253], 90.00th=[ 275], 95.00th=[ 317], | 99.00th=[ 372], 99.50th=[ 380], 99.90th=[ 393], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 80, max= 256, per=25.24%, avg=143.86, stdev=35.75, samples=69 iops : min= 20, max= 64, avg=35.48, stdev= 9.00, samples=69 write: IOPS=37, BW=149KiB/s (152kB/s)(5192KiB/34894msec); 0 zone resets slat (usec): min=27, max=16589, avg=280.57, stdev=958.68 clat (msec): min=40, max=418, avg=212.35, stdev=55.04 lat (msec): min=40, max=419, avg=212.63, stdev=55.02 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 132], 10.00th=[ 146], 20.00th=[ 167], | 30.00th=[ 184], 40.00th=[ 199], 50.00th=[ 211], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 255], 90.00th=[ 279], 95.00th=[ 313], | 99.00th=[ 359], 99.50th=[ 380], 99.90th=[ 405], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 71, max= 256, per=25.00%, avg=148.28, stdev=39.36, samples=69 iops : min= 17, max= 64, avg=36.61, stdev= 9.86, samples=69 lat (msec) : 50=0.23%, 100=1.29%, 250=76.29%, 500=22.19% cpu : usr=0.24%, sys=1.69%, ctx=4069, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180153: Mon Aug 24 10:37:05 2026 read: IOPS=35, BW=141KiB/s (145kB/s)(4976KiB/35181msec) slat (msec): min=2, max=164, avg=15.08, stdev= 8.92 clat (msec): min=26, max=458, avg=209.38, stdev=59.38 lat (msec): min=36, max=473, avg=224.47, stdev=61.65 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 126], 10.00th=[ 142], 20.00th=[ 161], | 30.00th=[ 176], 40.00th=[ 192], 50.00th=[ 205], 60.00th=[ 220], | 70.00th=[ 234], 80.00th=[ 253], 90.00th=[ 284], 95.00th=[ 326], | 99.00th=[ 376], 99.50th=[ 401], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 63, max= 264, per=24.62%, avg=140.32, stdev=38.01, samples=69 iops : min= 15, max= 66, avg=34.59, stdev= 9.54, samples=69 write: IOPS=37, BW=150KiB/s (153kB/s)(5264KiB/35181msec); 0 zone resets slat (usec): min=31, max=13297, avg=238.53, stdev=741.99 clat (msec): min=62, max=410, avg=214.12, stdev=53.55 lat (msec): min=62, max=410, avg=214.36, stdev=53.57 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 136], 10.00th=[ 153], 20.00th=[ 167], | 30.00th=[ 182], 40.00th=[ 197], 50.00th=[ 213], 60.00th=[ 224], | 70.00th=[ 241], 80.00th=[ 257], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 359], 99.50th=[ 380], 99.90th=[ 388], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 64, max= 231, per=25.19%, avg=149.41, stdev=37.52, samples=69 iops : min= 16, max= 57, avg=36.88, stdev= 9.41, samples=69 lat (msec) : 50=0.16%, 100=0.98%, 250=76.21%, 500=22.66% cpu : usr=0.23%, sys=1.60%, ctx=4007, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180154: Mon Aug 24 10:37:05 2026 read: IOPS=35, BW=142KiB/s (145kB/s)(4984KiB/35123msec) slat (usec): min=3052, max=66251, avg=14989.95, stdev=8120.01 clat (msec): min=37, max=405, avg=204.96, stdev=55.07 lat (msec): min=47, max=425, avg=219.97, stdev=57.25 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 126], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 171], 40.00th=[ 186], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 232], 80.00th=[ 249], 90.00th=[ 275], 95.00th=[ 305], | 99.00th=[ 359], 99.50th=[ 368], 99.90th=[ 401], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 55, max= 261, per=24.89%, avg=141.86, stdev=44.50, samples=69 iops : min= 13, max= 65, avg=35.00, stdev=11.20, samples=69 write: IOPS=37, BW=150KiB/s (153kB/s)(5256KiB/35123msec); 0 zone resets slat (usec): min=31, max=13736, avg=293.67, stdev=827.61 clat (msec): min=24, max=382, avg=217.48, stdev=55.42 lat (msec): min=27, max=382, avg=217.77, stdev=55.42 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 133], 10.00th=[ 150], 20.00th=[ 169], | 30.00th=[ 186], 40.00th=[ 199], 50.00th=[ 213], 60.00th=[ 230], | 70.00th=[ 245], 80.00th=[ 264], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 355], 99.50th=[ 372], 99.90th=[ 380], 99.95th=[ 384], | 99.99th=[ 384] bw ( KiB/s): min= 63, max= 256, per=25.07%, avg=148.68, stdev=37.13, samples=69 iops : min= 15, max= 64, avg=36.70, stdev= 9.33, samples=69 lat (msec) : 50=0.08%, 100=0.74%, 250=75.94%, 500=23.24% cpu : usr=0.27%, sys=1.60%, ctx=4585, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=180155: Mon Aug 24 10:37:05 2026 read: IOPS=35, BW=144KiB/s (147kB/s)(5060KiB/35181msec) slat (usec): min=2935, max=66217, avg=14965.18, stdev=8210.90 clat (msec): min=32, max=407, avg=208.46, stdev=55.71 lat (msec): min=53, max=423, avg=223.43, stdev=57.42 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 127], 10.00th=[ 140], 20.00th=[ 161], | 30.00th=[ 176], 40.00th=[ 190], 50.00th=[ 205], 60.00th=[ 220], | 70.00th=[ 234], 80.00th=[ 251], 90.00th=[ 279], 95.00th=[ 313], | 99.00th=[ 372], 99.50th=[ 380], 99.90th=[ 397], 99.95th=[ 409], | 99.99th=[ 409] bw ( KiB/s): min= 71, max= 224, per=25.05%, avg=142.80, stdev=32.74, samples=69 iops : min= 17, max= 56, avg=35.23, stdev= 8.26, samples=69 write: IOPS=36, BW=147KiB/s (151kB/s)(5180KiB/35181msec); 0 zone resets slat (usec): min=31, max=13653, avg=278.03, stdev=822.09 clat (msec): min=76, max=395, avg=214.69, stdev=52.53 lat (msec): min=76, max=395, avg=214.97, stdev=52.54 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 134], 10.00th=[ 148], 20.00th=[ 169], | 30.00th=[ 186], 40.00th=[ 201], 50.00th=[ 215], 60.00th=[ 228], | 70.00th=[ 241], 80.00th=[ 257], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 351], 99.50th=[ 359], 99.90th=[ 376], 99.95th=[ 397], | 99.99th=[ 397] bw ( KiB/s): min= 55, max= 256, per=24.60%, avg=145.88, stdev=41.38, samples=69 iops : min= 13, max= 64, avg=36.00, stdev=10.38, samples=69 lat (msec) : 50=0.04%, 100=0.78%, 250=77.38%, 500=21.80% cpu : usr=0.22%, sys=1.59%, ctx=4027, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=570KiB/s (584kB/s), 141KiB/s-145KiB/s (145kB/s-148kB/s), io=19.6MiB (20.5MB), run=34894-35181msec WRITE: bw=594KiB/s (608kB/s), 147KiB/s-150KiB/s (151kB/s-153kB/s), io=20.4MiB (21.4MB), run=34894-35181msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=180165: Mon Aug 24 10:37:12 2026 read: IOPS=2, BW=100MiB/s (105MB/s)(40.0MiB/399msec) slat (nsec): min=28358k, max=28358k, avg=28358165.00, stdev= 0.00 clat (nsec): min=344725k, max=344725k, avg=344725089.00, stdev= 0.00 lat (nsec): min=373089k, max=373089k, avg=373089125.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 347], 10.00th=[ 347], 20.00th=[ 347], | 30.00th=[ 347], 40.00th=[ 347], 50.00th=[ 347], 60.00th=[ 347], | 70.00th=[ 347], 80.00th=[ 347], 90.00th=[ 347], 95.00th=[ 347], | 99.00th=[ 347], 99.50th=[ 347], 99.90th=[ 347], 99.95th=[ 347], | 99.99th=[ 347] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.27%, ctx=6, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=40.0MiB (41.9MB), run=399-399msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout