== sanity test 398c: run fio to test AIO ================= 17:23:02 (1786483382) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.18189 s, 231 MB/s osc.lustre-OST0000-osc-ffff9d273117a800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=148887: Tue Aug 11 17:24:13 2026 write: IOPS=37, BW=151KiB/s (155kB/s)(10.0MiB/67594msec); 0 zone resets slat (usec): min=38, max=3138, avg=231.75, stdev=201.91 clat (msec): min=21, max=724, avg=421.91, stdev=96.76 lat (msec): min=21, max=725, avg=422.14, stdev=96.76 clat percentiles (msec): | 1.00th=[ 140], 5.00th=[ 247], 10.00th=[ 305], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 430], 60.00th=[ 451], | 70.00th=[ 468], 80.00th=[ 493], 90.00th=[ 531], 95.00th=[ 567], | 99.00th=[ 659], 99.50th=[ 684], 99.90th=[ 718], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 64, max= 374, per=24.92%, avg=150.74, stdev=38.57, samples=134 iops : min= 16, max= 93, avg=37.51, stdev= 9.63, samples=134 lat (msec) : 50=0.04%, 100=0.39%, 250=5.12%, 500=76.48%, 750=17.97% cpu : usr=0.07%, sys=0.60%, ctx=3029, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148888: Tue Aug 11 17:24:13 2026 write: IOPS=37, BW=152KiB/s (155kB/s)(10.0MiB/67568msec); 0 zone resets slat (usec): min=37, max=3901, avg=236.51, stdev=241.36 clat (msec): min=47, max=721, avg=421.87, stdev=98.36 lat (msec): min=47, max=721, avg=422.11, stdev=98.37 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 245], 10.00th=[ 305], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 430], 60.00th=[ 447], | 70.00th=[ 468], 80.00th=[ 493], 90.00th=[ 542], 95.00th=[ 575], | 99.00th=[ 659], 99.50th=[ 676], 99.90th=[ 718], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 71, max= 413, per=24.93%, avg=150.84, stdev=40.39, samples=134 iops : min= 17, max= 103, avg=37.54, stdev=10.08, samples=134 lat (msec) : 50=0.62%, 250=4.88%, 500=76.25%, 750=18.24% cpu : usr=0.07%, sys=0.57%, ctx=3143, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148889: Tue Aug 11 17:24:13 2026 write: IOPS=37, BW=151KiB/s (155kB/s)(10.0MiB/67594msec); 0 zone resets slat (usec): min=36, max=2775, avg=226.96, stdev=217.81 clat (msec): min=54, max=725, avg=421.95, stdev=97.96 lat (msec): min=54, max=725, avg=422.18, stdev=97.97 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 243], 10.00th=[ 309], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 430], 60.00th=[ 447], | 70.00th=[ 468], 80.00th=[ 498], 90.00th=[ 535], 95.00th=[ 575], | 99.00th=[ 667], 99.50th=[ 676], 99.90th=[ 718], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 48, max= 413, per=24.90%, avg=150.66, stdev=43.28, samples=134 iops : min= 12, max= 103, avg=37.50, stdev=10.81, samples=134 lat (msec) : 100=0.62%, 250=5.16%, 500=75.98%, 750=18.24% cpu : usr=0.08%, sys=0.56%, ctx=3057, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=148890: Tue Aug 11 17:24:13 2026 write: IOPS=37, BW=151KiB/s (155kB/s)(10.0MiB/67614msec); 0 zone resets slat (usec): min=37, max=2500, avg=239.16, stdev=224.83 clat (msec): min=122, max=721, avg=422.02, stdev=95.27 lat (msec): min=122, max=721, avg=422.26, stdev=95.29 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 249], 10.00th=[ 309], 20.00th=[ 355], | 30.00th=[ 384], 40.00th=[ 409], 50.00th=[ 426], 60.00th=[ 447], | 70.00th=[ 468], 80.00th=[ 493], 90.00th=[ 535], 95.00th=[ 575], | 99.00th=[ 659], 99.50th=[ 676], 99.90th=[ 709], 99.95th=[ 718], | 99.99th=[ 718] bw ( KiB/s): min= 71, max= 310, per=24.91%, avg=150.68, stdev=35.56, samples=134 iops : min= 17, max= 77, avg=37.50, stdev= 8.91, samples=134 lat (msec) : 250=5.08%, 500=76.99%, 750=17.93% cpu : usr=0.09%, sys=0.58%, ctx=3079, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=606KiB/s (620kB/s), 151KiB/s-152KiB/s (155kB/s-155kB/s), io=40.0MiB (41.9MB), run=67568-67614msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=148909: Tue Aug 11 17:25:03 2026 read: IOPS=27, BW=110KiB/s (112kB/s)(5048KiB/46093msec) slat (usec): min=2073, max=17997, avg=5576.01, stdev=2164.81 clat (msec): min=10, max=575, avg=184.89, stdev=44.14 lat (msec): min=15, max=580, avg=190.47, stdev=44.39 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 125], 10.00th=[ 138], 20.00th=[ 153], | 30.00th=[ 165], 40.00th=[ 174], 50.00th=[ 182], 60.00th=[ 194], | 70.00th=[ 205], 80.00th=[ 215], 90.00th=[ 232], 95.00th=[ 247], | 99.00th=[ 284], 99.50th=[ 430], 99.90th=[ 523], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 40, max= 200, per=25.25%, avg=109.59, stdev=35.48, samples=91 iops : min= 10, max= 50, avg=27.13, stdev= 8.81, samples=91 write: IOPS=28, BW=113KiB/s (115kB/s)(5192KiB/46093msec); 0 zone resets slat (usec): min=45, max=2460, avg=254.09, stdev=228.19 clat (msec): min=128, max=710, avg=381.58, stdev=93.81 lat (msec): min=128, max=710, avg=381.84, stdev=93.82 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 236], 10.00th=[ 262], 20.00th=[ 305], | 30.00th=[ 326], 40.00th=[ 355], 50.00th=[ 376], 60.00th=[ 401], | 70.00th=[ 426], 80.00th=[ 456], 90.00th=[ 502], 95.00th=[ 542], | 99.00th=[ 642], 99.50th=[ 667], 99.90th=[ 684], 99.95th=[ 709], | 99.99th=[ 709] bw ( KiB/s): min= 32, max= 176, per=24.94%, avg=112.74, stdev=27.65, samples=91 iops : min= 8, max= 44, avg=27.92, stdev= 6.98, samples=91 lat (msec) : 20=0.08%, 50=0.23%, 100=0.51%, 250=49.80%, 500=43.87% lat (msec) : 750=5.51% cpu : usr=0.14%, sys=1.19%, ctx=5191, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148910: Tue Aug 11 17:25:03 2026 read: IOPS=26, BW=108KiB/s (110kB/s)(4976KiB/46195msec) slat (usec): min=1745, max=15462, avg=5551.63, stdev=2021.62 clat (msec): min=5, max=464, avg=182.90, stdev=40.76 lat (msec): min=11, max=466, avg=188.46, stdev=41.03 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 127], 10.00th=[ 140], 20.00th=[ 153], | 30.00th=[ 163], 40.00th=[ 171], 50.00th=[ 180], 60.00th=[ 190], | 70.00th=[ 203], 80.00th=[ 213], 90.00th=[ 232], 95.00th=[ 247], | 99.00th=[ 288], 99.50th=[ 292], 99.90th=[ 405], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 15, max= 192, per=24.82%, avg=107.74, stdev=36.46, samples=91 iops : min= 3, max= 48, avg=26.67, stdev= 9.14, samples=91 write: IOPS=28, BW=114KiB/s (117kB/s)(5264KiB/46195msec); 0 zone resets slat (usec): min=41, max=3388, avg=275.15, stdev=271.43 clat (msec): min=82, max=731, avg=382.33, stdev=96.65 lat (msec): min=82, max=732, avg=382.61, stdev=96.65 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 245], 10.00th=[ 264], 20.00th=[ 300], | 30.00th=[ 326], 40.00th=[ 355], 50.00th=[ 380], 60.00th=[ 401], | 70.00th=[ 422], 80.00th=[ 460], 90.00th=[ 506], 95.00th=[ 550], | 99.00th=[ 642], 99.50th=[ 676], 99.90th=[ 735], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min= 40, max= 176, per=25.20%, avg=113.89, stdev=28.03, samples=91 iops : min= 10, max= 44, avg=28.21, stdev= 7.05, samples=91 lat (msec) : 10=0.08%, 20=0.04%, 50=0.20%, 100=0.23%, 250=49.14% lat (msec) : 500=44.61%, 750=5.70% cpu : usr=0.16%, sys=1.20%, ctx=5244, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148911: Tue Aug 11 17:25:03 2026 read: IOPS=26, BW=108KiB/s (111kB/s)(4984KiB/46167msec) slat (usec): min=35, max=16658, avg=5609.83, stdev=2110.41 clat (msec): min=8, max=536, avg=184.42, stdev=40.69 lat (msec): min=8, max=542, avg=190.03, stdev=41.07 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 126], 10.00th=[ 142], 20.00th=[ 155], | 30.00th=[ 165], 40.00th=[ 174], 50.00th=[ 184], 60.00th=[ 194], | 70.00th=[ 203], 80.00th=[ 213], 90.00th=[ 232], 95.00th=[ 249], | 99.00th=[ 279], 99.50th=[ 284], 99.90th=[ 489], 99.95th=[ 535], | 99.99th=[ 535] bw ( KiB/s): min= 40, max= 192, per=24.98%, avg=108.42, stdev=35.78, samples=91 iops : min= 10, max= 48, avg=26.84, stdev= 9.00, samples=91 write: IOPS=28, BW=114KiB/s (117kB/s)(5256KiB/46167msec); 0 zone resets slat (usec): min=35, max=2567, avg=255.60, stdev=242.31 clat (msec): min=92, max=682, avg=381.19, stdev=98.19 lat (msec): min=92, max=682, avg=381.45, stdev=98.18 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 232], 10.00th=[ 264], 20.00th=[ 300], | 30.00th=[ 334], 40.00th=[ 355], 50.00th=[ 376], 60.00th=[ 401], | 70.00th=[ 426], 80.00th=[ 456], 90.00th=[ 510], 95.00th=[ 558], | 99.00th=[ 651], 99.50th=[ 659], 99.90th=[ 667], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 48, max= 187, per=25.09%, avg=113.43, stdev=25.81, samples=91 iops : min= 12, max= 46, avg=28.09, stdev= 6.46, samples=91 lat (msec) : 10=0.12%, 20=0.16%, 50=0.08%, 100=0.51%, 250=49.10% lat (msec) : 500=44.30%, 750=5.74% cpu : usr=0.15%, sys=1.23%, ctx=5191, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=148912: Tue Aug 11 17:25:03 2026 read: IOPS=27, BW=109KiB/s (112kB/s)(5060KiB/46217msec) slat (usec): min=1976, max=18691, avg=5493.48, stdev=2039.89 clat (msec): min=8, max=313, avg=184.03, stdev=38.66 lat (msec): min=15, max=317, avg=189.53, stdev=39.00 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 129], 10.00th=[ 140], 20.00th=[ 153], | 30.00th=[ 165], 40.00th=[ 174], 50.00th=[ 182], 60.00th=[ 192], | 70.00th=[ 205], 80.00th=[ 218], 90.00th=[ 232], 95.00th=[ 249], | 99.00th=[ 275], 99.50th=[ 284], 99.90th=[ 309], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 16, max= 207, per=25.26%, avg=109.64, stdev=35.27, samples=91 iops : min= 4, max= 51, avg=27.14, stdev= 8.85, samples=91 write: IOPS=28, BW=112KiB/s (115kB/s)(5180KiB/46217msec); 0 zone resets slat (usec): min=53, max=2704, avg=259.48, stdev=254.07 clat (msec): min=132, max=680, avg=384.66, stdev=95.16 lat (msec): min=132, max=680, avg=384.92, stdev=95.14 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 236], 10.00th=[ 266], 20.00th=[ 305], | 30.00th=[ 338], 40.00th=[ 363], 50.00th=[ 380], 60.00th=[ 401], | 70.00th=[ 430], 80.00th=[ 460], 90.00th=[ 510], 95.00th=[ 550], | 99.00th=[ 651], 99.50th=[ 667], 99.90th=[ 676], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 64, max= 159, per=24.60%, avg=111.21, stdev=20.76, samples=91 iops : min= 16, max= 39, avg=27.54, stdev= 5.20, samples=91 lat (msec) : 10=0.08%, 20=0.04%, 50=0.23%, 100=0.43%, 250=49.96% lat (msec) : 500=43.40%, 750=5.86% cpu : usr=0.12%, sys=1.23%, ctx=5168, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=434KiB/s (445kB/s), 108KiB/s-110KiB/s (110kB/s-112kB/s), io=19.6MiB (20.5MB), run=46093-46217msec WRITE: bw=452KiB/s (463kB/s), 112KiB/s-114KiB/s (115kB/s-117kB/s), io=20.4MiB (21.4MB), run=46093-46217msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=148922: Tue Aug 11 17:25:07 2026 read: IOPS=6, BW=258MiB/s (271MB/s)(40.0MiB/155msec) slat (nsec): min=13815k, max=13815k, avg=13815162.00, stdev= 0.00 clat (nsec): min=134478k, max=134478k, avg=134477557.00, stdev= 0.00 lat (nsec): min=148298k, max=148298k, avg=148297500.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 136], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136], | 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 136], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 136], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 136], 99.50th=[ 136], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=7.14%, ctx=43, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=258MiB/s (271MB/s), 258MiB/s-258MiB/s (271MB/s-271MB/s), io=40.0MiB (41.9MB), run=155-155msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout