== sanity test 398c: run fio to test AIO ================= 11:43:38 (1754408618) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.142032 s, 295 MB/s osc.lustre-OST0000-osc-ffff8f9d0b051000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=188034: Tue Aug 5 11:43:52 2025 write: IOPS=261, BW=1045KiB/s (1070kB/s)(10.0MiB/9799msec); 0 zone resets slat (usec): min=22, max=26230, avg=130.86, stdev=525.40 clat (msec): min=6, max=118, avg=60.70, stdev=10.48 lat (msec): min=7, max=118, avg=60.84, stdev=10.46 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 47], 10.00th=[ 50], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 60], 60.00th=[ 62], | 70.00th=[ 65], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 80], | 99.00th=[ 92], 99.50th=[ 103], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 118] bw ( KiB/s): min= 776, max= 1232, per=25.01%, avg=1040.26, stdev=122.60, samples=19 iops : min= 194, max= 308, avg=260.05, stdev=30.65, samples=19 lat (msec) : 10=0.08%, 20=0.12%, 50=11.91%, 100=87.15%, 250=0.74% cpu : usr=0.36%, sys=3.23%, ctx=1563, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=188035: Tue Aug 5 11:43:52 2025 write: IOPS=261, BW=1047KiB/s (1072kB/s)(10.0MiB/9778msec); 0 zone resets slat (usec): min=28, max=1351, avg=120.59, stdev=95.25 clat (msec): min=7, max=117, avg=60.91, stdev=10.35 lat (msec): min=8, max=117, avg=61.03, stdev=10.35 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 47], 10.00th=[ 50], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 63], | 70.00th=[ 65], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 81], | 99.00th=[ 92], 99.50th=[ 103], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 117] bw ( KiB/s): min= 776, max= 1256, per=25.10%, avg=1044.16, stdev=125.12, samples=19 iops : min= 194, max= 314, avg=261.00, stdev=31.26, samples=19 lat (msec) : 10=0.04%, 20=0.08%, 50=11.80%, 100=87.30%, 250=0.78% cpu : usr=0.56%, sys=2.75%, ctx=1603, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=188036: Tue Aug 5 11:43:52 2025 write: IOPS=260, BW=1040KiB/s (1065kB/s)(10.0MiB/9845msec); 0 zone resets slat (usec): min=21, max=2669, avg=117.32, stdev=123.37 clat (msec): min=10, max=117, avg=60.91, stdev=10.58 lat (msec): min=10, max=118, avg=61.03, stdev=10.58 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 47], 10.00th=[ 50], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 63], | 70.00th=[ 65], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 81], | 99.00th=[ 93], 99.50th=[ 106], 99.90th=[ 114], 99.95th=[ 115], | 99.99th=[ 118] bw ( KiB/s): min= 768, max= 1272, per=24.92%, avg=1036.58, stdev=127.72, samples=19 iops : min= 192, max= 318, avg=259.11, stdev=31.96, samples=19 lat (msec) : 20=0.27%, 50=11.95%, 100=87.11%, 250=0.66% cpu : usr=0.44%, sys=2.68%, ctx=1346, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=188037: Tue Aug 5 11:43:52 2025 write: IOPS=260, BW=1040KiB/s (1065kB/s)(10.0MiB/9845msec); 0 zone resets slat (usec): min=24, max=3413, avg=113.01, stdev=110.98 clat (msec): min=13, max=117, avg=60.95, stdev=10.32 lat (msec): min=13, max=118, avg=61.07, stdev=10.32 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 48], 10.00th=[ 50], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 63], | 70.00th=[ 65], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 80], | 99.00th=[ 99], 99.50th=[ 106], 99.90th=[ 114], 99.95th=[ 118], | 99.99th=[ 118] bw ( KiB/s): min= 768, max= 1272, per=24.92%, avg=1036.58, stdev=131.59, samples=19 iops : min= 192, max= 318, avg=259.11, stdev=32.93, samples=19 lat (msec) : 20=0.08%, 50=11.91%, 100=87.27%, 250=0.74% cpu : usr=0.41%, sys=2.98%, ctx=1325, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=4160KiB/s (4260kB/s), 1040KiB/s-1047KiB/s (1065kB/s-1072kB/s), io=40.0MiB (41.9MB), run=9778-9845msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=188048: Tue Aug 5 11:44:05 2025 read: IOPS=133, BW=533KiB/s (546kB/s)(5048KiB/9468msec) slat (usec): min=1595, max=14099, avg=3915.61, stdev=1368.42 clat (msec): min=13, max=141, avg=56.20, stdev=12.81 lat (msec): min=15, max=145, avg=60.12, stdev=13.30 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 46], | 30.00th=[ 49], 40.00th=[ 52], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 74], 95.00th=[ 81], | 99.00th=[ 91], 99.50th=[ 94], 99.90th=[ 134], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 432, max= 680, per=25.37%, avg=537.06, stdev=77.77, samples=18 iops : min= 108, max= 170, avg=134.06, stdev=19.56, samples=18 write: IOPS=137, BW=548KiB/s (562kB/s)(5192KiB/9468msec); 0 zone resets slat (usec): min=27, max=1183, avg=110.94, stdev=82.79 clat (msec): min=18, max=133, avg=57.28, stdev=11.08 lat (msec): min=18, max=133, avg=57.39, stdev=11.08 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 67], 90.00th=[ 72], 95.00th=[ 77], | 99.00th=[ 87], 99.50th=[ 93], 99.90th=[ 131], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 376, max= 704, per=25.06%, avg=552.33, stdev=95.59, samples=18 iops : min= 94, max= 176, avg=137.94, stdev=23.88, samples=18 lat (msec) : 20=0.12%, 50=31.13%, 100=68.44%, 250=0.31% cpu : usr=0.55%, sys=4.11%, ctx=2422, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=188049: Tue Aug 5 11:44:05 2025 read: IOPS=131, BW=525KiB/s (538kB/s)(4976KiB/9477msec) slat (usec): min=1633, max=13542, avg=3889.05, stdev=1338.38 clat (msec): min=12, max=140, avg=56.68, stdev=12.82 lat (msec): min=14, max=145, avg=60.57, stdev=13.26 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 41], 10.00th=[ 43], 20.00th=[ 47], | 30.00th=[ 50], 40.00th=[ 53], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 75], 95.00th=[ 81], | 99.00th=[ 92], 99.50th=[ 94], 99.90th=[ 138], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 384, max= 688, per=24.96%, avg=528.44, stdev=86.78, samples=18 iops : min= 96, max= 172, avg=131.89, stdev=21.79, samples=18 write: IOPS=138, BW=555KiB/s (569kB/s)(5264KiB/9477msec); 0 zone resets slat (usec): min=32, max=1866, avg=110.60, stdev=108.40 clat (msec): min=22, max=136, avg=57.37, stdev=11.39 lat (msec): min=22, max=136, avg=57.48, stdev=11.39 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[ 48], | 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 73], 95.00th=[ 79], | 99.00th=[ 92], 99.50th=[ 94], 99.90th=[ 111], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 344, max= 704, per=25.40%, avg=559.89, stdev=98.03, samples=18 iops : min= 86, max= 176, avg=139.78, stdev=24.50, samples=18 lat (msec) : 20=0.04%, 50=29.65%, 100=69.92%, 250=0.39% cpu : usr=0.58%, sys=4.03%, ctx=2416, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=188050: Tue Aug 5 11:44:05 2025 read: IOPS=131, BW=527KiB/s (540kB/s)(4984KiB/9452msec) slat (usec): min=1505, max=12317, avg=3878.40, stdev=1305.65 clat (msec): min=16, max=167, avg=55.68, stdev=12.58 lat (msec): min=24, max=171, avg=59.57, stdev=13.02 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 46], | 30.00th=[ 49], 40.00th=[ 52], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 72], 95.00th=[ 79], | 99.00th=[ 92], 99.50th=[ 99], 99.90th=[ 136], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 384, max= 648, per=25.25%, avg=534.44, stdev=83.20, samples=18 iops : min= 96, max= 162, avg=133.39, stdev=20.86, samples=18 write: IOPS=139, BW=556KiB/s (569kB/s)(5256KiB/9452msec); 0 zone resets slat (usec): min=23, max=1145, avg=112.49, stdev=90.86 clat (msec): min=33, max=166, avg=57.79, stdev=12.42 lat (msec): min=33, max=166, avg=57.90, stdev=12.43 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 73], 95.00th=[ 80], | 99.00th=[ 89], 99.50th=[ 116], 99.90th=[ 167], 99.95th=[ 167], | 99.99th=[ 167] bw ( KiB/s): min= 408, max= 728, per=25.32%, avg=558.00, stdev=99.71, samples=18 iops : min= 102, max= 182, avg=139.33, stdev=24.99, samples=18 lat (msec) : 20=0.04%, 50=30.82%, 100=68.67%, 250=0.47% cpu : usr=0.50%, sys=4.24%, ctx=2489, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=188051: Tue Aug 5 11:44:05 2025 read: IOPS=134, BW=537KiB/s (550kB/s)(5060KiB/9417msec) slat (usec): min=51, max=12416, avg=3864.28, stdev=1346.77 clat (msec): min=10, max=129, avg=55.59, stdev=12.74 lat (msec): min=10, max=136, avg=59.46, stdev=13.18 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 46], | 30.00th=[ 49], 40.00th=[ 52], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 72], 95.00th=[ 79], | 99.00th=[ 91], 99.50th=[ 100], 99.90th=[ 127], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 384, max= 680, per=25.56%, avg=541.11, stdev=78.59, samples=18 iops : min= 96, max= 170, avg=135.11, stdev=19.70, samples=18 write: IOPS=137, BW=550KiB/s (563kB/s)(5180KiB/9417msec); 0 zone resets slat (usec): min=24, max=3582, avg=122.62, stdev=170.21 clat (msec): min=17, max=158, avg=57.94, stdev=12.17 lat (msec): min=17, max=159, avg=58.06, stdev=12.18 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[ 49], | 30.00th=[ 52], 40.00th=[ 54], 50.00th=[ 57], 60.00th=[ 59], | 70.00th=[ 63], 80.00th=[ 67], 90.00th=[ 75], 95.00th=[ 81], | 99.00th=[ 92], 99.50th=[ 95], 99.90th=[ 138], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 392, max= 680, per=24.99%, avg=550.89, stdev=93.00, samples=18 iops : min= 98, max= 170, avg=137.56, stdev=23.30, samples=18 lat (msec) : 20=0.20%, 50=30.16%, 100=69.26%, 250=0.39% cpu : usr=0.69%, sys=4.12%, ctx=2468, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=2118KiB/s (2168kB/s), 525KiB/s-537KiB/s (538kB/s-550kB/s), io=19.6MiB (20.5MB), run=9417-9477msec WRITE: bw=2204KiB/s (2257kB/s), 548KiB/s-556KiB/s (562kB/s-569kB/s), io=20.4MiB (21.4MB), run=9417-9477msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=188056: Tue Aug 5 11:44:09 2025 read: IOPS=5, BW=211MiB/s (221MB/s)(40.0MiB/190msec) slat (nsec): min=12562k, max=12562k, avg=12561596.00, stdev= 0.00 clat (nsec): min=168159k, max=168159k, avg=168159144.00, stdev= 0.00 lat (nsec): min=180725k, max=180725k, avg=180725110.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 169], 10.00th=[ 169], 20.00th=[ 169], | 30.00th=[ 169], 40.00th=[ 169], 50.00th=[ 169], 60.00th=[ 169], | 70.00th=[ 169], 80.00th=[ 169], 90.00th=[ 169], 95.00th=[ 169], | 99.00th=[ 169], 99.50th=[ 169], 99.90th=[ 169], 99.95th=[ 169], | 99.99th=[ 169] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=6.88%, ctx=2, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=211MiB/s (221MB/s), 211MiB/s-211MiB/s (221MB/s-221MB/s), io=40.0MiB (41.9MB), run=190-190msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout