== sanity test 398c: run fio to test AIO ================= 14:40:48 (1785955248) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.55576 s, 75.5 MB/s osc.lustre-OST0000-osc-ffff99e45981a000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179200: Wed Aug 5 14:41:43 2026 write: IOPS=64, BW=256KiB/s (263kB/s)(10.0MiB/39941msec); 0 zone resets slat (usec): min=50, max=29002, avg=248.02, stdev=974.70 clat (msec): min=96, max=479, avg=248.63, stdev=69.23 lat (msec): min=96, max=479, avg=248.88, stdev=69.24 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 128], 10.00th=[ 157], 20.00th=[ 188], | 30.00th=[ 211], 40.00th=[ 232], 50.00th=[ 249], 60.00th=[ 268], | 70.00th=[ 284], 80.00th=[ 309], 90.00th=[ 342], 95.00th=[ 363], | 99.00th=[ 397], 99.50th=[ 422], 99.90th=[ 481], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 125, max= 504, per=24.93%, avg=255.25, stdev=68.90, samples=79 iops : min= 31, max= 126, avg=63.42, stdev=17.27, samples=79 lat (msec) : 100=0.62%, 250=50.51%, 500=48.87% cpu : usr=0.13%, sys=0.71%, ctx=1942, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179201: Wed Aug 5 14:41:43 2026 write: IOPS=64, BW=256KiB/s (262kB/s)(10.0MiB/39992msec); 0 zone resets slat (usec): min=49, max=26767, avg=317.50, stdev=1324.47 clat (msec): min=23, max=417, avg=248.62, stdev=68.22 lat (msec): min=24, max=418, avg=248.94, stdev=68.27 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 128], 10.00th=[ 157], 20.00th=[ 188], | 30.00th=[ 213], 40.00th=[ 232], 50.00th=[ 249], 60.00th=[ 268], | 70.00th=[ 288], 80.00th=[ 309], 90.00th=[ 338], 95.00th=[ 368], | 99.00th=[ 393], 99.50th=[ 405], 99.90th=[ 418], 99.95th=[ 418], | 99.99th=[ 418] bw ( KiB/s): min= 127, max= 504, per=24.83%, avg=254.24, stdev=70.98, samples=79 iops : min= 31, max= 126, avg=63.18, stdev=17.77, samples=79 lat (msec) : 50=0.12%, 100=0.08%, 250=50.90%, 500=48.91% cpu : usr=0.14%, sys=0.85%, ctx=2212, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179202: Wed Aug 5 14:41:43 2026 write: IOPS=64, BW=257KiB/s (264kB/s)(10.0MiB/39792msec); 0 zone resets slat (usec): min=49, max=21986, avg=286.08, stdev=1072.98 clat (msec): min=10, max=463, avg=248.16, stdev=70.85 lat (msec): min=10, max=463, avg=248.45, stdev=70.84 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 125], 10.00th=[ 153], 20.00th=[ 188], | 30.00th=[ 211], 40.00th=[ 230], 50.00th=[ 249], 60.00th=[ 268], | 70.00th=[ 288], 80.00th=[ 313], 90.00th=[ 338], 95.00th=[ 368], | 99.00th=[ 397], 99.50th=[ 422], 99.90th=[ 464], 99.95th=[ 464], | 99.99th=[ 464] bw ( KiB/s): min= 143, max= 535, per=24.94%, avg=255.41, stdev=72.05, samples=78 iops : min= 35, max= 133, avg=63.45, stdev=18.01, samples=78 lat (msec) : 20=0.04%, 50=0.12%, 100=0.98%, 250=49.30%, 500=49.57% cpu : usr=0.13%, sys=0.85%, ctx=2574, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179203: Wed Aug 5 14:41:43 2026 write: IOPS=64, BW=256KiB/s (262kB/s)(10.0MiB/39968msec); 0 zone resets slat (usec): min=36, max=12958, avg=190.26, stdev=581.82 clat (msec): min=34, max=504, avg=248.72, stdev=69.58 lat (msec): min=43, max=504, avg=248.92, stdev=69.55 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 131], 10.00th=[ 155], 20.00th=[ 190], | 30.00th=[ 209], 40.00th=[ 228], 50.00th=[ 249], 60.00th=[ 268], | 70.00th=[ 288], 80.00th=[ 313], 90.00th=[ 338], 95.00th=[ 368], | 99.00th=[ 397], 99.50th=[ 468], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] bw ( KiB/s): min= 126, max= 497, per=24.92%, avg=255.18, stdev=72.22, samples=79 iops : min= 31, max= 124, avg=63.41, stdev=18.07, samples=79 lat (msec) : 50=0.08%, 100=0.39%, 250=49.96%, 500=49.30%, 750=0.27% cpu : usr=0.09%, sys=0.66%, ctx=1779, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1024KiB/s (1049kB/s), 256KiB/s-257KiB/s (262kB/s-264kB/s), io=40.0MiB (41.9MB), run=39792-39992msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179219: Wed Aug 5 14:42:51 2026 read: IOPS=23, BW=92.4KiB/s (94.6kB/s)(5048KiB/54636msec) slat (usec): min=81, max=120754, avg=25254.63, stdev=16820.94 clat (msec): min=30, max=683, avg=323.63, stdev=95.23 lat (msec): min=43, max=725, avg=348.89, stdev=99.59 clat percentiles (msec): | 1.00th=[ 163], 5.00th=[ 192], 10.00th=[ 213], 20.00th=[ 243], | 30.00th=[ 266], 40.00th=[ 288], 50.00th=[ 313], 60.00th=[ 334], | 70.00th=[ 359], 80.00th=[ 401], 90.00th=[ 456], 95.00th=[ 498], | 99.00th=[ 600], 99.50th=[ 634], 99.90th=[ 676], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 24, max= 171, per=24.98%, avg=91.66, stdev=28.69, samples=107 iops : min= 6, max= 42, avg=22.39, stdev= 7.18, samples=107 write: IOPS=23, BW=95.0KiB/s (97.3kB/s)(5192KiB/54636msec); 0 zone resets slat (usec): min=52, max=19034, avg=224.31, stdev=704.15 clat (msec): min=42, max=696, avg=331.17, stdev=95.99 lat (msec): min=42, max=696, avg=331.40, stdev=96.03 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 201], 10.00th=[ 222], 20.00th=[ 253], | 30.00th=[ 275], 40.00th=[ 296], 50.00th=[ 317], 60.00th=[ 342], | 70.00th=[ 368], 80.00th=[ 405], 90.00th=[ 468], 95.00th=[ 514], | 99.00th=[ 609], 99.50th=[ 634], 99.90th=[ 684], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 23, max= 183, per=24.77%, avg=94.64, stdev=32.95, samples=107 iops : min= 5, max= 45, avg=23.14, stdev= 8.28, samples=107 lat (msec) : 50=0.08%, 100=0.08%, 250=20.74%, 500=73.63%, 750=5.47% cpu : usr=0.17%, sys=1.10%, ctx=3701, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179220: Wed Aug 5 14:42:51 2026 read: IOPS=22, BW=91.2KiB/s (93.3kB/s)(4976KiB/54589msec) slat (msec): min=2, max=137, avg=24.84, stdev=16.43 clat (msec): min=20, max=695, avg=317.78, stdev=89.21 lat (msec): min=28, max=710, avg=342.63, stdev=92.84 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 197], 10.00th=[ 215], 20.00th=[ 243], | 30.00th=[ 262], 40.00th=[ 284], 50.00th=[ 305], 60.00th=[ 330], | 70.00th=[ 351], 80.00th=[ 384], 90.00th=[ 443], 95.00th=[ 493], | 99.00th=[ 558], 99.50th=[ 584], 99.90th=[ 609], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 31, max= 174, per=24.61%, avg=90.31, stdev=30.04, samples=107 iops : min= 7, max= 43, avg=22.09, stdev= 7.55, samples=107 write: IOPS=24, BW=96.4KiB/s (98.7kB/s)(5264KiB/54589msec); 0 zone resets slat (usec): min=50, max=29240, avg=283.83, stdev=1166.70 clat (msec): min=82, max=669, avg=336.64, stdev=96.00 lat (msec): min=82, max=669, avg=336.92, stdev=96.01 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 203], 10.00th=[ 226], 20.00th=[ 257], | 30.00th=[ 279], 40.00th=[ 300], 50.00th=[ 321], 60.00th=[ 342], | 70.00th=[ 376], 80.00th=[ 422], 90.00th=[ 477], 95.00th=[ 518], | 99.00th=[ 592], 99.50th=[ 617], 99.90th=[ 667], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 47, max= 179, per=25.18%, avg=96.20, stdev=28.18, samples=107 iops : min= 11, max= 44, avg=23.52, stdev= 7.08, samples=107 lat (msec) : 50=0.04%, 100=0.08%, 250=20.23%, 500=73.91%, 750=5.74% cpu : usr=0.19%, sys=1.15%, ctx=3928, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179221: Wed Aug 5 14:42:51 2026 read: IOPS=23, BW=92.1KiB/s (94.3kB/s)(4984KiB/54138msec) slat (usec): min=54, max=117180, avg=24727.75, stdev=15680.19 clat (msec): min=26, max=667, avg=318.28, stdev=94.48 lat (msec): min=26, max=711, avg=343.01, stdev=98.41 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 188], 10.00th=[ 209], 20.00th=[ 241], | 30.00th=[ 262], 40.00th=[ 284], 50.00th=[ 305], 60.00th=[ 330], | 70.00th=[ 355], 80.00th=[ 397], 90.00th=[ 451], 95.00th=[ 493], | 99.00th=[ 567], 99.50th=[ 617], 99.90th=[ 659], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 32, max= 190, per=25.00%, avg=91.75, stdev=32.00, samples=106 iops : min= 8, max= 47, avg=22.42, stdev= 8.07, samples=106 write: IOPS=24, BW=97.1KiB/s (99.4kB/s)(5256KiB/54138msec); 0 zone resets slat (usec): min=51, max=29480, avg=296.44, stdev=1317.74 clat (msec): min=153, max=697, avg=333.04, stdev=94.37 lat (msec): min=153, max=697, avg=333.34, stdev=94.39 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 203], 10.00th=[ 224], 20.00th=[ 255], | 30.00th=[ 279], 40.00th=[ 296], 50.00th=[ 317], 60.00th=[ 334], | 70.00th=[ 368], 80.00th=[ 409], 90.00th=[ 468], 95.00th=[ 514], | 99.00th=[ 609], 99.50th=[ 634], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 31, max= 183, per=25.17%, avg=96.16, stdev=31.24, samples=106 iops : min= 7, max= 45, avg=23.56, stdev= 7.84, samples=106 lat (msec) : 50=0.08%, 100=0.08%, 250=20.98%, 500=73.12%, 750=5.74% cpu : usr=0.13%, sys=1.17%, ctx=3919, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179222: Wed Aug 5 14:42:51 2026 read: IOPS=23, BW=92.8KiB/s (95.1kB/s)(5060KiB/54510msec) slat (msec): min=3, max=117, avg=25.23, stdev=16.43 clat (msec): min=54, max=666, avg=320.88, stdev=94.90 lat (msec): min=89, max=686, avg=346.11, stdev=99.17 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 194], 10.00th=[ 215], 20.00th=[ 239], | 30.00th=[ 259], 40.00th=[ 284], 50.00th=[ 309], 60.00th=[ 330], | 70.00th=[ 359], 80.00th=[ 393], 90.00th=[ 460], 95.00th=[ 506], | 99.00th=[ 592], 99.50th=[ 600], 99.90th=[ 642], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 32, max= 182, per=25.18%, avg=92.42, stdev=28.34, samples=107 iops : min= 8, max= 45, avg=22.61, stdev= 7.14, samples=107 write: IOPS=23, BW=95.0KiB/s (97.3kB/s)(5180KiB/54510msec); 0 zone resets slat (usec): min=48, max=67575, avg=270.25, stdev=1972.12 clat (msec): min=157, max=693, avg=332.29, stdev=94.39 lat (msec): min=157, max=693, avg=332.57, stdev=94.47 clat percentiles (msec): | 1.00th=[ 176], 5.00th=[ 199], 10.00th=[ 226], 20.00th=[ 255], | 30.00th=[ 275], 40.00th=[ 296], 50.00th=[ 317], 60.00th=[ 338], | 70.00th=[ 368], 80.00th=[ 414], 90.00th=[ 472], 95.00th=[ 510], | 99.00th=[ 592], 99.50th=[ 617], 99.90th=[ 659], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 24, max= 179, per=24.75%, avg=94.55, stdev=32.34, samples=107 iops : min= 6, max= 44, avg=23.11, stdev= 8.09, samples=107 lat (msec) : 100=0.04%, 250=21.60%, 500=72.54%, 750=5.82% cpu : usr=0.13%, sys=1.09%, ctx=3224, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=367KiB/s (376kB/s), 91.2KiB/s-92.8KiB/s (93.3kB/s-95.1kB/s), io=19.6MiB (20.5MB), run=54138-54636msec WRITE: bw=382KiB/s (392kB/s), 95.0KiB/s-97.1KiB/s (97.3kB/s-99.4kB/s), io=20.4MiB (21.4MB), run=54138-54636msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179235: Wed Aug 5 14:43:06 2026 read: IOPS=1, BW=57.0MiB/s (60.8MB/s)(40.0MiB/690msec) slat (nsec): min=26857k, max=26857k, avg=26857373.00, stdev= 0.00 clat (nsec): min=624529k, max=624529k, avg=624528786.00, stdev= 0.00 lat (nsec): min=651390k, max=651390k, avg=651390419.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 625], 5.00th=[ 625], 10.00th=[ 625], 20.00th=[ 625], | 30.00th=[ 625], 40.00th=[ 625], 50.00th=[ 625], 60.00th=[ 625], | 70.00th=[ 625], 80.00th=[ 625], 90.00th=[ 625], 95.00th=[ 625], | 99.00th=[ 625], 99.50th=[ 625], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.89%, ctx=3, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=57.0MiB/s (60.8MB/s), 57.0MiB/s-57.0MiB/s (60.8MB/s-60.8MB/s), io=40.0MiB (41.9MB), run=690-690msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout