== sanity test 398c: run fio to test AIO ================= 21:43:20 (1781228600) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.216468 s, 194 MB/s osc.lustre-OST0000-osc-ffff9f25083a8000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=189943: Thu Jun 11 21:43:37 2026 write: IOPS=210, BW=844KiB/s (864kB/s)(10.0MiB/12133msec); 0 zone resets slat (usec): min=35, max=5598, avg=206.95, stdev=296.58 clat (msec): min=12, max=148, avg=75.43, stdev=15.47 lat (msec): min=12, max=148, avg=75.64, stdev=15.46 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 54], 10.00th=[ 57], 20.00th=[ 62], | 30.00th=[ 67], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 79], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 97], 95.00th=[ 104], | 99.00th=[ 110], 99.50th=[ 116], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 148] bw ( KiB/s): min= 654, max= 1049, per=25.13%, avg=847.52, stdev=124.71, samples=23 iops : min= 163, max= 262, avg=211.65, stdev=31.22, samples=23 lat (msec) : 20=0.12%, 50=2.66%, 100=89.84%, 250=7.38% cpu : usr=0.31%, sys=2.56%, ctx=2703, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=189944: Thu Jun 11 21:43:37 2026 write: IOPS=210, BW=843KiB/s (864kB/s)(10.0MiB/12141msec); 0 zone resets slat (usec): min=37, max=6746, avg=206.30, stdev=267.87 clat (msec): min=22, max=122, avg=75.34, stdev=15.18 lat (msec): min=23, max=122, avg=75.55, stdev=15.17 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 53], 10.00th=[ 57], 20.00th=[ 62], | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 79], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 96], 95.00th=[ 103], | 99.00th=[ 111], 99.50th=[ 116], 99.90th=[ 123], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min= 646, max= 1109, per=24.85%, avg=838.33, stdev=125.05, samples=24 iops : min= 161, max= 277, avg=209.38, stdev=31.28, samples=24 lat (msec) : 50=2.54%, 100=90.66%, 250=6.80% cpu : usr=0.27%, sys=2.64%, ctx=2836, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=189945: Thu Jun 11 21:43:37 2026 write: IOPS=211, BW=845KiB/s (865kB/s)(10.0MiB/12124msec); 0 zone resets slat (usec): min=40, max=5979, avg=207.42, stdev=264.56 clat (msec): min=10, max=147, avg=75.25, stdev=15.82 lat (msec): min=10, max=147, avg=75.46, stdev=15.82 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 53], 10.00th=[ 57], 20.00th=[ 62], | 30.00th=[ 66], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 79], | 70.00th=[ 84], 80.00th=[ 88], 90.00th=[ 97], 95.00th=[ 104], | 99.00th=[ 111], 99.50th=[ 120], 99.90th=[ 124], 99.95th=[ 142], | 99.99th=[ 148] bw ( KiB/s): min= 678, max= 1050, per=25.17%, avg=848.96, stdev=115.19, samples=23 iops : min= 169, max= 262, avg=212.00, stdev=28.82, samples=23 lat (msec) : 20=0.08%, 50=3.79%, 100=88.98%, 250=7.15% cpu : usr=0.30%, sys=2.68%, ctx=2577, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=189946: Thu Jun 11 21:43:37 2026 write: IOPS=211, BW=845KiB/s (866kB/s)(10.0MiB/12112msec); 0 zone resets slat (usec): min=45, max=4133, avg=218.63, stdev=275.22 clat (msec): min=16, max=135, avg=75.39, stdev=15.49 lat (msec): min=16, max=135, avg=75.61, stdev=15.49 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 53], 10.00th=[ 57], 20.00th=[ 62], | 30.00th=[ 67], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 79], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 97], 95.00th=[ 104], | 99.00th=[ 110], 99.50th=[ 114], 99.90th=[ 124], 99.95th=[ 125], | 99.99th=[ 136] bw ( KiB/s): min= 648, max= 1095, per=25.17%, avg=848.83, stdev=125.19, samples=23 iops : min= 162, max= 273, avg=211.96, stdev=31.27, samples=23 lat (msec) : 20=0.04%, 50=2.93%, 100=89.45%, 250=7.58% cpu : usr=0.31%, sys=2.85%, ctx=2759, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3374KiB/s (3455kB/s), 843KiB/s-845KiB/s (864kB/s-866kB/s), io=40.0MiB (41.9MB), run=12112-12141msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=189957: Thu Jun 11 21:43:51 2026 read: IOPS=120, BW=483KiB/s (495kB/s)(5048KiB/10444msec) slat (usec): min=59, max=15945, avg=4267.47, stdev=1699.72 clat (msec): min=6, max=117, avg=62.32, stdev=13.15 lat (msec): min=9, max=123, avg=66.59, stdev=13.65 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 45], 10.00th=[ 49], 20.00th=[ 52], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 64], | 70.00th=[ 67], 80.00th=[ 72], 90.00th=[ 81], 95.00th=[ 88], | 99.00th=[ 102], 99.50th=[ 106], 99.90th=[ 110], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 367, max= 632, per=25.16%, avg=483.15, stdev=66.27, samples=20 iops : min= 91, max= 158, avg=120.60, stdev=16.71, samples=20 write: IOPS=124, BW=497KiB/s (509kB/s)(5192KiB/10444msec); 0 zone resets slat (usec): min=40, max=1906, avg=201.99, stdev=168.50 clat (msec): min=10, max=111, avg=63.31, stdev=12.10 lat (msec): min=10, max=111, avg=63.52, stdev=12.10 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 48], 10.00th=[ 51], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 73], 90.00th=[ 81], 95.00th=[ 86], | 99.00th=[ 96], 99.50th=[ 104], 99.90th=[ 112], 99.95th=[ 112], | 99.99th=[ 112] bw ( KiB/s): min= 359, max= 608, per=24.87%, avg=497.10, stdev=63.28, samples=20 iops : min= 89, max= 152, avg=124.10, stdev=15.94, samples=20 lat (msec) : 10=0.08%, 20=0.04%, 50=11.95%, 100=87.03%, 250=0.90% cpu : usr=0.42%, sys=4.10%, ctx=4659, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=189958: Thu Jun 11 21:43:51 2026 read: IOPS=119, BW=477KiB/s (488kB/s)(4976KiB/10431msec) slat (usec): min=67, max=15445, avg=4162.62, stdev=1661.38 clat (msec): min=12, max=131, avg=61.71, stdev=12.73 lat (msec): min=16, max=135, avg=65.87, stdev=13.08 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 45], 10.00th=[ 48], 20.00th=[ 52], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 71], 90.00th=[ 79], 95.00th=[ 85], | 99.00th=[ 101], 99.50th=[ 109], 99.90th=[ 122], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 360, max= 616, per=24.84%, avg=476.95, stdev=63.09, samples=20 iops : min= 90, max= 154, avg=119.05, stdev=15.89, samples=20 write: IOPS=126, BW=505KiB/s (517kB/s)(5264KiB/10431msec); 0 zone resets slat (usec): min=49, max=3822, avg=205.98, stdev=190.77 clat (msec): min=21, max=120, avg=63.81, stdev=11.97 lat (msec): min=21, max=120, avg=64.01, stdev=11.97 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 59], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 69], 80.00th=[ 74], 90.00th=[ 80], 95.00th=[ 85], | 99.00th=[ 97], 99.50th=[ 104], 99.90th=[ 109], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 399, max= 688, per=25.28%, avg=505.35, stdev=68.76, samples=20 iops : min= 99, max= 172, avg=126.20, stdev=17.32, samples=20 lat (msec) : 20=0.08%, 50=12.50%, 100=86.48%, 250=0.94% cpu : usr=0.29%, sys=4.23%, ctx=4609, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=189959: Thu Jun 11 21:43:51 2026 read: IOPS=119, BW=480KiB/s (491kB/s)(4984KiB/10390msec) slat (usec): min=57, max=15159, avg=4203.00, stdev=1674.54 clat (msec): min=22, max=123, avg=61.80, stdev=13.13 lat (msec): min=27, max=128, avg=66.01, stdev=13.55 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 44], 10.00th=[ 48], 20.00th=[ 52], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 72], 90.00th=[ 80], 95.00th=[ 87], | 99.00th=[ 102], 99.50th=[ 107], 99.90th=[ 113], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 392, max= 688, per=25.11%, avg=482.15, stdev=74.69, samples=20 iops : min= 98, max= 172, avg=120.35, stdev=18.75, samples=20 write: IOPS=126, BW=506KiB/s (518kB/s)(5256KiB/10390msec); 0 zone resets slat (usec): min=46, max=4601, avg=209.98, stdev=224.46 clat (msec): min=21, max=122, avg=63.47, stdev=12.58 lat (msec): min=21, max=123, avg=63.68, stdev=12.58 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 73], 90.00th=[ 81], 95.00th=[ 86], | 99.00th=[ 101], 99.50th=[ 103], 99.90th=[ 124], 99.95th=[ 124], | 99.99th=[ 124] bw ( KiB/s): min= 360, max= 632, per=25.29%, avg=505.55, stdev=70.68, samples=20 iops : min= 90, max= 158, avg=126.20, stdev=17.81, samples=20 lat (msec) : 50=12.89%, 100=86.09%, 250=1.02% cpu : usr=0.46%, sys=4.12%, ctx=4592, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=189960: Thu Jun 11 21:43:51 2026 read: IOPS=121, BW=484KiB/s (496kB/s)(5060KiB/10451msec) slat (usec): min=1439, max=14171, avg=4206.43, stdev=1568.11 clat (msec): min=30, max=114, avg=62.02, stdev=12.51 lat (msec): min=32, max=119, avg=66.23, stdev=12.91 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 45], 10.00th=[ 48], 20.00th=[ 52], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 64], | 70.00th=[ 67], 80.00th=[ 72], 90.00th=[ 80], 95.00th=[ 85], | 99.00th=[ 99], 99.50th=[ 102], 99.90th=[ 111], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min= 375, max= 616, per=25.21%, avg=484.10, stdev=78.27, samples=20 iops : min= 93, max= 154, avg=120.85, stdev=19.72, samples=20 write: IOPS=123, BW=496KiB/s (508kB/s)(5180KiB/10451msec); 0 zone resets slat (usec): min=61, max=3999, avg=204.59, stdev=217.69 clat (msec): min=32, max=117, avg=63.45, stdev=11.80 lat (msec): min=32, max=117, avg=63.65, stdev=11.81 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 48], 10.00th=[ 51], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 74], 90.00th=[ 80], 95.00th=[ 85], | 99.00th=[ 96], 99.50th=[ 101], 99.90th=[ 112], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 407, max= 608, per=24.70%, avg=493.80, stdev=61.91, samples=20 iops : min= 101, max= 152, avg=123.30, stdev=15.59, samples=20 lat (msec) : 50=12.73%, 100=86.64%, 250=0.62% cpu : usr=0.45%, sys=4.07%, ctx=4672, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1920KiB/s (1966kB/s), 477KiB/s-484KiB/s (488kB/s-496kB/s), io=19.6MiB (20.5MB), run=10390-10451msec WRITE: bw=1999KiB/s (2047kB/s), 496KiB/s-506KiB/s (508kB/s-518kB/s), io=20.4MiB (21.4MB), run=10390-10451msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=189965: Thu Jun 11 21:43:55 2026 read: IOPS=7, BW=282MiB/s (295MB/s)(40.0MiB/142msec) slat (nsec): min=15871k, max=15871k, avg=15871241.00, stdev= 0.00 clat (nsec): min=120348k, max=120348k, avg=120347842.00, stdev= 0.00 lat (nsec): min=136224k, max=136224k, avg=136223561.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 121], 5.00th=[ 121], 10.00th=[ 121], 20.00th=[ 121], | 30.00th=[ 121], 40.00th=[ 121], 50.00th=[ 121], 60.00th=[ 121], | 70.00th=[ 121], 80.00th=[ 121], 90.00th=[ 121], 95.00th=[ 121], | 99.00th=[ 121], 99.50th=[ 121], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=8.51%, ctx=12, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=282MiB/s (295MB/s), 282MiB/s-282MiB/s (295MB/s-295MB/s), io=40.0MiB (41.9MB), run=142-142msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout