== sanity test 398c: run fio to test AIO ================= 09:45:48 (1788529548) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.499204 s, 84.0 MB/s osc.lustre-OST0000-osc-ffff9f75d2c63800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=181152: Fri Sep 4 09:46:32 2026 write: IOPS=77, BW=312KiB/s (319kB/s)(10.0MiB/32856msec); 0 zone resets slat (usec): min=26, max=17965, avg=229.54, stdev=923.48 clat (msec): min=29, max=496, avg=204.82, stdev=60.11 lat (msec): min=29, max=496, avg=205.06, stdev=60.07 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 111], 10.00th=[ 138], 20.00th=[ 157], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 201], 60.00th=[ 213], | 70.00th=[ 232], 80.00th=[ 253], 90.00th=[ 279], 95.00th=[ 300], | 99.00th=[ 376], 99.50th=[ 439], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 498] bw ( KiB/s): min= 148, max= 572, per=25.07%, avg=310.66, stdev=78.96, samples=65 iops : min= 37, max= 143, avg=77.25, stdev=19.81, samples=65 lat (msec) : 50=0.12%, 100=3.32%, 250=76.13%, 500=20.43% cpu : usr=0.16%, sys=0.94%, ctx=2617, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181153: Fri Sep 4 09:46:32 2026 write: IOPS=77, BW=310KiB/s (318kB/s)(10.0MiB/33025msec); 0 zone resets slat (usec): min=22, max=30574, avg=233.88, stdev=1034.80 clat (msec): min=24, max=467, avg=205.55, stdev=58.94 lat (msec): min=24, max=467, avg=205.79, stdev=58.92 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 120], 10.00th=[ 140], 20.00th=[ 153], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 215], | 70.00th=[ 230], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 305], | 99.00th=[ 363], 99.50th=[ 443], 99.90th=[ 468], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 125, max= 568, per=25.04%, avg=310.28, stdev=80.19, samples=65 iops : min= 31, max= 142, avg=77.15, stdev=20.14, samples=65 lat (msec) : 50=0.20%, 100=1.48%, 250=76.17%, 500=22.15% cpu : usr=0.16%, sys=0.78%, ctx=2490, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181154: Fri Sep 4 09:46:32 2026 write: IOPS=77, BW=310KiB/s (318kB/s)(10.0MiB/32990msec); 0 zone resets slat (usec): min=28, max=20333, avg=172.21, stdev=718.29 clat (msec): min=64, max=447, avg=204.99, stdev=57.05 lat (msec): min=64, max=447, avg=205.17, stdev=57.05 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 117], 10.00th=[ 138], 20.00th=[ 159], | 30.00th=[ 174], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 232], 80.00th=[ 251], 90.00th=[ 279], 95.00th=[ 309], | 99.00th=[ 347], 99.50th=[ 405], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 167, max= 512, per=24.96%, avg=309.29, stdev=74.89, samples=65 iops : min= 41, max= 128, avg=76.89, stdev=18.78, samples=65 lat (msec) : 100=1.21%, 250=78.59%, 500=20.20% cpu : usr=0.13%, sys=0.63%, ctx=1441, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=181155: Fri Sep 4 09:46:32 2026 write: IOPS=77, BW=310KiB/s (317kB/s)(10.0MiB/33039msec); 0 zone resets slat (usec): min=28, max=21840, avg=201.39, stdev=898.45 clat (msec): min=23, max=448, avg=205.12, stdev=57.73 lat (msec): min=23, max=449, avg=205.32, stdev=57.72 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 115], 10.00th=[ 136], 20.00th=[ 155], | 30.00th=[ 176], 40.00th=[ 190], 50.00th=[ 203], 60.00th=[ 213], | 70.00th=[ 230], 80.00th=[ 251], 90.00th=[ 279], 95.00th=[ 309], | 99.00th=[ 368], 99.50th=[ 388], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 451] bw ( KiB/s): min= 195, max= 512, per=24.94%, avg=309.06, stdev=71.70, samples=65 iops : min= 48, max= 128, avg=76.85, stdev=17.98, samples=65 lat (msec) : 50=0.08%, 100=2.27%, 250=77.23%, 500=20.43% cpu : usr=0.15%, sys=0.74%, ctx=1913, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1240KiB/s (1270kB/s), 310KiB/s-312KiB/s (317kB/s-319kB/s), io=40.0MiB (41.9MB), run=32856-33039msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=181170: Fri Sep 4 09:47:21 2026 read: IOPS=32, BW=132KiB/s (135kB/s)(5048KiB/38329msec) slat (usec): min=2725, max=72389, avg=16399.11, stdev=9587.99 clat (msec): min=27, max=585, avg=228.45, stdev=63.99 lat (msec): min=42, max=597, avg=244.86, stdev=65.90 clat percentiles (msec): | 1.00th=[ 111], 5.00th=[ 136], 10.00th=[ 153], 20.00th=[ 176], | 30.00th=[ 192], 40.00th=[ 209], 50.00th=[ 222], 60.00th=[ 239], | 70.00th=[ 255], 80.00th=[ 275], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 430], 99.50th=[ 468], 99.90th=[ 575], 99.95th=[ 584], | 99.99th=[ 584] bw ( KiB/s): min= 41, max= 199, per=25.25%, avg=131.80, stdev=29.04, samples=75 iops : min= 10, max= 49, avg=32.44, stdev= 7.25, samples=75 write: IOPS=33, BW=135KiB/s (139kB/s)(5192KiB/38329msec); 0 zone resets slat (usec): min=30, max=22946, avg=296.66, stdev=1205.13 clat (msec): min=44, max=574, avg=230.90, stdev=61.37 lat (msec): min=44, max=574, avg=231.20, stdev=61.29 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 140], 10.00th=[ 155], 20.00th=[ 178], | 30.00th=[ 201], 40.00th=[ 211], 50.00th=[ 228], 60.00th=[ 241], | 70.00th=[ 257], 80.00th=[ 275], 90.00th=[ 305], 95.00th=[ 334], | 99.00th=[ 414], 99.50th=[ 443], 99.90th=[ 518], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 34, max= 238, per=24.99%, avg=135.92, stdev=38.55, samples=75 iops : min= 8, max= 59, avg=33.49, stdev= 9.63, samples=75 lat (msec) : 50=0.16%, 100=0.16%, 250=65.70%, 500=33.79%, 750=0.20% cpu : usr=0.25%, sys=1.48%, ctx=4475, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181171: Fri Sep 4 09:47:21 2026 read: IOPS=32, BW=131KiB/s (134kB/s)(4976KiB/37991msec) slat (usec): min=30, max=83912, avg=16063.76, stdev=9712.03 clat (msec): min=33, max=496, avg=225.94, stdev=61.06 lat (msec): min=48, max=523, avg=242.01, stdev=63.26 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 138], 10.00th=[ 155], 20.00th=[ 174], | 30.00th=[ 192], 40.00th=[ 205], 50.00th=[ 220], 60.00th=[ 236], | 70.00th=[ 255], 80.00th=[ 275], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 430], 99.50th=[ 447], 99.90th=[ 481], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 47, max= 230, per=24.95%, avg=130.21, stdev=37.20, samples=75 iops : min= 11, max= 57, avg=32.05, stdev= 9.30, samples=75 write: IOPS=34, BW=139KiB/s (142kB/s)(5264KiB/37991msec); 0 zone resets slat (usec): min=27, max=46717, avg=265.37, stdev=1474.26 clat (msec): min=96, max=496, avg=232.21, stdev=60.21 lat (msec): min=96, max=496, avg=232.47, stdev=60.14 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 144], 10.00th=[ 159], 20.00th=[ 180], | 30.00th=[ 199], 40.00th=[ 213], 50.00th=[ 226], 60.00th=[ 243], | 70.00th=[ 259], 80.00th=[ 279], 90.00th=[ 309], 95.00th=[ 334], | 99.00th=[ 401], 99.50th=[ 422], 99.90th=[ 451], 99.95th=[ 498], | 99.99th=[ 498] bw ( KiB/s): min= 79, max= 277, per=25.41%, avg=138.25, stdev=36.60, samples=75 iops : min= 19, max= 69, avg=34.07, stdev= 9.14, samples=75 lat (msec) : 50=0.04%, 100=0.12%, 250=65.74%, 500=34.10% cpu : usr=0.20%, sys=1.38%, ctx=3906, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181172: Fri Sep 4 09:47:21 2026 read: IOPS=32, BW=130KiB/s (133kB/s)(4984KiB/38351msec) slat (usec): min=2646, max=77414, avg=16129.99, stdev=9277.72 clat (msec): min=79, max=491, avg=226.57, stdev=61.54 lat (msec): min=93, max=531, avg=242.70, stdev=64.06 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 138], 10.00th=[ 153], 20.00th=[ 176], | 30.00th=[ 192], 40.00th=[ 207], 50.00th=[ 222], 60.00th=[ 239], | 70.00th=[ 253], 80.00th=[ 275], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 422], 99.50th=[ 443], 99.90th=[ 485], 99.95th=[ 493], | 99.99th=[ 493] bw ( KiB/s): min= 40, max= 214, per=24.98%, avg=130.39, stdev=37.65, samples=75 iops : min= 10, max= 53, avg=32.11, stdev= 9.40, samples=75 write: IOPS=34, BW=137KiB/s (140kB/s)(5256KiB/38351msec); 0 zone resets slat (usec): min=30, max=30546, avg=272.05, stdev=1254.26 clat (msec): min=105, max=562, avg=233.77, stdev=60.59 lat (msec): min=105, max=562, avg=234.04, stdev=60.61 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 148], 10.00th=[ 161], 20.00th=[ 184], | 30.00th=[ 201], 40.00th=[ 213], 50.00th=[ 228], 60.00th=[ 245], | 70.00th=[ 262], 80.00th=[ 284], 90.00th=[ 309], 95.00th=[ 334], | 99.00th=[ 409], 99.50th=[ 468], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 47, max= 236, per=25.19%, avg=137.01, stdev=35.91, samples=75 iops : min= 11, max= 59, avg=33.75, stdev= 9.03, samples=75 lat (msec) : 100=0.35%, 250=65.62%, 500=33.87%, 750=0.16% cpu : usr=0.23%, sys=1.48%, ctx=4345, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=181173: Fri Sep 4 09:47:21 2026 read: IOPS=32, BW=132KiB/s (135kB/s)(5060KiB/38401msec) slat (msec): min=2, max=253, avg=16.41, stdev=11.71 clat (msec): min=49, max=479, avg=225.35, stdev=61.71 lat (msec): min=66, max=504, avg=241.76, stdev=63.86 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 136], 10.00th=[ 150], 20.00th=[ 171], | 30.00th=[ 190], 40.00th=[ 207], 50.00th=[ 222], 60.00th=[ 239], | 70.00th=[ 255], 80.00th=[ 275], 90.00th=[ 305], 95.00th=[ 326], | 99.00th=[ 409], 99.50th=[ 447], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 27, max= 246, per=25.10%, avg=131.04, stdev=37.95, samples=76 iops : min= 6, max= 61, avg=32.22, stdev= 9.53, samples=76 write: IOPS=33, BW=135KiB/s (138kB/s)(5180KiB/38401msec); 0 zone resets slat (usec): min=25, max=21895, avg=260.25, stdev=1119.70 clat (msec): min=97, max=458, avg=234.89, stdev=62.56 lat (msec): min=97, max=458, avg=235.15, stdev=62.54 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 144], 10.00th=[ 161], 20.00th=[ 182], | 30.00th=[ 197], 40.00th=[ 213], 50.00th=[ 228], 60.00th=[ 247], | 70.00th=[ 268], 80.00th=[ 284], 90.00th=[ 317], 95.00th=[ 347], | 99.00th=[ 422], 99.50th=[ 435], 99.90th=[ 460], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 48, max= 230, per=24.67%, avg=134.22, stdev=33.43, samples=76 iops : min= 12, max= 57, avg=33.08, stdev= 8.35, samples=76 lat (msec) : 50=0.04%, 100=0.31%, 250=64.18%, 500=35.47% cpu : usr=0.23%, sys=1.57%, ctx=4242, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=523KiB/s (535kB/s), 130KiB/s-132KiB/s (133kB/s-135kB/s), io=19.6MiB (20.5MB), run=37991-38401msec WRITE: bw=544KiB/s (557kB/s), 135KiB/s-139KiB/s (138kB/s-142kB/s), io=20.4MiB (21.4MB), run=37991-38401msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=181182: Fri Sep 4 09:47:31 2026 read: IOPS=1, BW=67.2MiB/s (70.5MB/s)(40.0MiB/595msec) slat (nsec): min=30802k, max=30802k, avg=30801582.00, stdev= 0.00 clat (nsec): min=550398k, max=550398k, avg=550398334.00, stdev= 0.00 lat (nsec): min=581204k, max=581204k, avg=581204126.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 550], 10.00th=[ 550], 20.00th=[ 550], | 30.00th=[ 550], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.52%, ctx=6, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=67.2MiB/s (70.5MB/s), 67.2MiB/s-67.2MiB/s (70.5MB/s-70.5MB/s), io=40.0MiB (41.9MB), run=595-595msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout