== sanity test 398c: run fio to test AIO ================= 20:31:41 (1786840301) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.121925 s, 344 MB/s osc.lustre-OST0000-osc-ffff941cc08be800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=155345: Sat Aug 15 20:32:14 2026 write: IOPS=87, BW=350KiB/s (359kB/s)(10.0MiB/29240msec); 0 zone resets slat (usec): min=34, max=1031, avg=215.00, stdev=116.79 clat (msec): min=55, max=299, avg=182.34, stdev=34.49 lat (msec): min=56, max=300, avg=182.56, stdev=34.49 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 126], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 165], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 203], 80.00th=[ 211], 90.00th=[ 224], 95.00th=[ 236], | 99.00th=[ 257], 99.50th=[ 271], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 280, max= 511, per=24.92%, avg=348.81, stdev=48.94, samples=58 iops : min= 70, max= 127, avg=87.19, stdev=12.19, samples=58 lat (msec) : 100=0.74%, 250=97.34%, 500=1.91% cpu : usr=0.10%, sys=1.55%, ctx=2725, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155346: Sat Aug 15 20:32:14 2026 write: IOPS=87, BW=351KiB/s (359kB/s)(10.0MiB/29211msec); 0 zone resets slat (usec): min=36, max=1526, avg=207.07, stdev=113.45 clat (msec): min=37, max=318, avg=182.17, stdev=35.99 lat (msec): min=37, max=318, avg=182.38, stdev=36.00 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 125], 10.00th=[ 138], 20.00th=[ 153], | 30.00th=[ 165], 40.00th=[ 176], 50.00th=[ 184], 60.00th=[ 192], | 70.00th=[ 203], 80.00th=[ 211], 90.00th=[ 224], 95.00th=[ 234], | 99.00th=[ 271], 99.50th=[ 284], 99.90th=[ 305], 99.95th=[ 309], | 99.99th=[ 317] bw ( KiB/s): min= 256, max= 511, per=24.92%, avg=348.81, stdev=50.76, samples=58 iops : min= 64, max= 127, avg=87.19, stdev=12.65, samples=58 lat (msec) : 50=0.62%, 100=0.86%, 250=96.52%, 500=1.99% cpu : usr=0.14%, sys=1.47%, ctx=2643, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155347: Sat Aug 15 20:32:14 2026 write: IOPS=87, BW=351KiB/s (359kB/s)(10.0MiB/29181msec); 0 zone resets slat (usec): min=41, max=1152, avg=215.97, stdev=113.40 clat (msec): min=31, max=318, avg=182.03, stdev=35.96 lat (msec): min=31, max=318, avg=182.24, stdev=35.97 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 125], 10.00th=[ 136], 20.00th=[ 153], | 30.00th=[ 165], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 203], 80.00th=[ 211], 90.00th=[ 224], 95.00th=[ 234], | 99.00th=[ 259], 99.50th=[ 288], 99.90th=[ 309], 99.95th=[ 317], | 99.99th=[ 317] bw ( KiB/s): min= 256, max= 512, per=24.92%, avg=348.81, stdev=55.27, samples=58 iops : min= 64, max= 128, avg=87.19, stdev=13.78, samples=58 lat (msec) : 50=0.62%, 100=1.33%, 250=96.60%, 500=1.45% cpu : usr=0.14%, sys=1.51%, ctx=2775, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=155348: Sat Aug 15 20:32:14 2026 write: IOPS=87, BW=350KiB/s (359kB/s)(10.0MiB/29240msec); 0 zone resets slat (usec): min=47, max=814, avg=215.56, stdev=106.27 clat (msec): min=17, max=299, avg=182.30, stdev=34.20 lat (msec): min=17, max=300, avg=182.52, stdev=34.20 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 129], 10.00th=[ 138], 20.00th=[ 153], | 30.00th=[ 165], 40.00th=[ 176], 50.00th=[ 186], 60.00th=[ 194], | 70.00th=[ 203], 80.00th=[ 211], 90.00th=[ 224], 95.00th=[ 234], | 99.00th=[ 255], 99.50th=[ 271], 99.90th=[ 292], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 264, max= 517, per=24.91%, avg=348.78, stdev=48.67, samples=58 iops : min= 66, max= 129, avg=87.19, stdev=12.15, samples=58 lat (msec) : 20=0.04%, 100=1.33%, 250=97.34%, 500=1.29% cpu : usr=0.16%, sys=1.52%, ctx=2747, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1401KiB/s (1434kB/s), 350KiB/s-351KiB/s (359kB/s-359kB/s), io=40.0MiB (41.9MB), run=29181-29240msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=155362: Sat Aug 15 20:32:37 2026 read: IOPS=60, BW=243KiB/s (249kB/s)(5048KiB/20765msec) slat (usec): min=1135, max=7621, avg=2699.20, stdev=987.62 clat (msec): min=3, max=211, avg=81.38, stdev=16.99 lat (msec): min=6, max=214, avg=84.08, stdev=17.13 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 69], | 30.00th=[ 74], 40.00th=[ 78], 50.00th=[ 82], 60.00th=[ 86], | 70.00th=[ 90], 80.00th=[ 94], 90.00th=[ 101], 95.00th=[ 106], | 99.00th=[ 122], 99.50th=[ 129], 99.90th=[ 197], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 152, max= 336, per=25.22%, avg=243.66, stdev=51.21, samples=41 iops : min= 38, max= 84, avg=60.88, stdev=12.79, samples=41 write: IOPS=62, BW=250KiB/s (256kB/s)(5192KiB/20765msec); 0 zone resets slat (usec): min=54, max=1897, avg=199.43, stdev=112.76 clat (msec): min=68, max=312, avg=173.44, stdev=36.62 lat (msec): min=68, max=312, avg=173.64, stdev=36.61 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 114], 10.00th=[ 125], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 203], 90.00th=[ 218], 95.00th=[ 232], | 99.00th=[ 271], 99.50th=[ 296], 99.90th=[ 313], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 184, max= 320, per=24.82%, avg=249.71, stdev=31.87, samples=41 iops : min= 46, max= 80, avg=62.39, stdev= 7.97, samples=41 lat (msec) : 4=0.04%, 10=0.04%, 20=0.31%, 50=0.35%, 100=44.92% lat (msec) : 250=53.40%, 500=0.94% cpu : usr=0.15%, sys=2.32%, ctx=4891, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155363: Sat Aug 15 20:32:37 2026 read: IOPS=59, BW=240KiB/s (245kB/s)(4976KiB/20765msec) slat (usec): min=34, max=7162, avg=2738.50, stdev=976.10 clat (msec): min=6, max=156, avg=81.67, stdev=16.20 lat (msec): min=6, max=159, avg=84.41, stdev=16.36 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 59], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 74], 40.00th=[ 79], 50.00th=[ 82], 60.00th=[ 87], | 70.00th=[ 90], 80.00th=[ 95], 90.00th=[ 101], 95.00th=[ 107], | 99.00th=[ 121], 99.50th=[ 124], 99.90th=[ 127], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 136, max= 328, per=24.73%, avg=238.93, stdev=50.49, samples=41 iops : min= 34, max= 82, avg=59.71, stdev=12.60, samples=41 write: IOPS=63, BW=254KiB/s (260kB/s)(5264KiB/20765msec); 0 zone resets slat (usec): min=36, max=683, avg=191.07, stdev=97.95 clat (msec): min=25, max=328, avg=172.31, stdev=38.18 lat (msec): min=25, max=328, avg=172.50, stdev=38.19 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 113], 10.00th=[ 124], 20.00th=[ 144], | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 203], 90.00th=[ 218], 95.00th=[ 230], | 99.00th=[ 255], 99.50th=[ 275], 99.90th=[ 326], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 168, max= 328, per=25.09%, avg=252.44, stdev=36.86, samples=41 iops : min= 42, max= 82, avg=63.07, stdev= 9.20, samples=41 lat (msec) : 10=0.20%, 20=0.16%, 50=1.05%, 100=43.59%, 250=54.30% lat (msec) : 500=0.70% cpu : usr=0.21%, sys=2.26%, ctx=4764, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155364: Sat Aug 15 20:32:37 2026 read: IOPS=60, BW=240KiB/s (246kB/s)(4984KiB/20765msec) slat (usec): min=1078, max=8192, avg=2763.26, stdev=974.89 clat (msec): min=3, max=210, avg=81.44, stdev=16.22 lat (msec): min=5, max=214, avg=84.20, stdev=16.39 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 58], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 74], 40.00th=[ 79], 50.00th=[ 82], 60.00th=[ 86], | 70.00th=[ 89], 80.00th=[ 94], 90.00th=[ 100], 95.00th=[ 106], | 99.00th=[ 121], 99.50th=[ 125], 99.90th=[ 174], 99.95th=[ 211], | 99.99th=[ 211] bw ( KiB/s): min= 160, max= 376, per=24.95%, avg=241.00, stdev=50.30, samples=41 iops : min= 40, max= 94, avg=60.22, stdev=12.58, samples=41 write: IOPS=63, BW=253KiB/s (259kB/s)(5256KiB/20765msec); 0 zone resets slat (usec): min=43, max=1082, avg=199.42, stdev=107.26 clat (msec): min=40, max=294, avg=172.36, stdev=36.60 lat (msec): min=40, max=294, avg=172.56, stdev=36.60 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 113], 10.00th=[ 125], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 192], 80.00th=[ 203], 90.00th=[ 220], 95.00th=[ 232], | 99.00th=[ 255], 99.50th=[ 264], 99.90th=[ 292], 99.95th=[ 296], | 99.99th=[ 296] bw ( KiB/s): min= 176, max= 344, per=25.05%, avg=251.98, stdev=36.66, samples=41 iops : min= 44, max= 86, avg=62.98, stdev= 9.17, samples=41 lat (msec) : 4=0.08%, 10=0.08%, 20=0.23%, 50=0.31%, 100=44.69% lat (msec) : 250=53.95%, 500=0.66% cpu : usr=0.24%, sys=2.23%, ctx=4839, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=155365: Sat Aug 15 20:32:37 2026 read: IOPS=61, BW=245KiB/s (251kB/s)(5060KiB/20644msec) slat (usec): min=1046, max=7779, avg=2721.07, stdev=975.48 clat (msec): min=3, max=198, avg=81.62, stdev=16.15 lat (msec): min=5, max=201, avg=84.34, stdev=16.29 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 59], 10.00th=[ 63], 20.00th=[ 69], | 30.00th=[ 74], 40.00th=[ 79], 50.00th=[ 82], 60.00th=[ 86], | 70.00th=[ 89], 80.00th=[ 94], 90.00th=[ 101], 95.00th=[ 106], | 99.00th=[ 122], 99.50th=[ 127], 99.90th=[ 194], 99.95th=[ 199], | 99.99th=[ 199] bw ( KiB/s): min= 160, max= 336, per=25.57%, avg=247.00, stdev=43.20, samples=40 iops : min= 40, max= 84, avg=61.73, stdev=10.82, samples=40 write: IOPS=62, BW=251KiB/s (257kB/s)(5180KiB/20644msec); 0 zone resets slat (usec): min=55, max=1481, avg=194.94, stdev=107.95 clat (msec): min=43, max=339, avg=171.99, stdev=37.37 lat (msec): min=43, max=339, avg=172.18, stdev=37.37 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 113], 10.00th=[ 126], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 190], 80.00th=[ 201], 90.00th=[ 218], 95.00th=[ 230], | 99.00th=[ 268], 99.50th=[ 313], 99.90th=[ 338], 99.95th=[ 338], | 99.99th=[ 338] bw ( KiB/s): min= 184, max= 344, per=24.97%, avg=251.15, stdev=35.12, samples=40 iops : min= 46, max= 86, avg=62.75, stdev= 8.75, samples=40 lat (msec) : 4=0.04%, 10=0.04%, 20=0.16%, 50=0.35%, 100=44.84% lat (msec) : 250=53.91%, 500=0.66% cpu : usr=0.19%, sys=2.26%, ctx=4927, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=966KiB/s (990kB/s), 240KiB/s-245KiB/s (245kB/s-251kB/s), io=19.6MiB (20.5MB), run=20644-20765msec WRITE: bw=1006KiB/s (1030kB/s), 250KiB/s-254KiB/s (256kB/s-260kB/s), io=20.4MiB (21.4MB), run=20644-20765msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=155372: Sat Aug 15 20:32:41 2026 read: IOPS=8, BW=354MiB/s (371MB/s)(40.0MiB/113msec) slat (nsec): min=9852.0k, max=9852.0k, avg=9852994.00, stdev= 0.00 clat (nsec): min=98066k, max=98066k, avg=98065846.00, stdev= 0.00 lat (nsec): min=107922k, max=107922k, avg=107922140.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[98042], 5.00th=[98042], 10.00th=[98042], 20.00th=[98042], | 30.00th=[98042], 40.00th=[98042], 50.00th=[98042], 60.00th=[98042], | 70.00th=[98042], 80.00th=[98042], 90.00th=[98042], 95.00th=[98042], | 99.00th=[98042], 99.50th=[98042], 99.90th=[98042], 99.95th=[98042], | 99.99th=[98042] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=8.04%, ctx=43, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=354MiB/s (371MB/s), 354MiB/s-354MiB/s (371MB/s-371MB/s), io=40.0MiB (41.9MB), run=113-113msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout