== sanity test 398c: run fio to test AIO ================= 01:58:37 (1746079117) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.195123 s, 215 MB/s osc.lustre-OST0000-osc-ffff8801373a5000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=26753: Thu May 1 01:58:52 2025 write: IOPS=199, BW=798KiB/s (817kB/s)(10.0MiB/12828msec) slat (usec): min=12, max=10037, avg=103.91, stdev=304.46 clat (msec): min=12, max=172, avg=79.77, stdev=17.70 lat (msec): min=12, max=173, avg=79.88, stdev=17.70 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 56], 10.00th=[ 59], 20.00th=[ 66], | 30.00th=[ 70], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 83], | 70.00th=[ 87], 80.00th=[ 93], 90.00th=[ 104], 95.00th=[ 111], | 99.00th=[ 140], 99.50th=[ 142], 99.90th=[ 174], 99.95th=[ 174], | 99.99th=[ 174] bw ( KiB/s): min= 638, max= 982, per=24.80%, avg=790.72, stdev=92.38, samples=25 iops : min= 159, max= 245, avg=197.56, stdev=23.15, samples=25 lat (msec) : 20=0.12%, 50=2.30%, 100=85.39%, 250=12.19% cpu : usr=0.62%, sys=2.59%, ctx=962, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=26754: Thu May 1 01:58:52 2025 write: IOPS=200, BW=800KiB/s (820kB/s)(10.0MiB/12795msec) slat (usec): min=11, max=3701, avg=119.75, stdev=250.12 clat (msec): min=11, max=180, avg=79.59, stdev=18.58 lat (msec): min=11, max=180, avg=79.71, stdev=18.57 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 53], 10.00th=[ 59], 20.00th=[ 66], | 30.00th=[ 71], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 87], 80.00th=[ 92], 90.00th=[ 101], 95.00th=[ 111], | 99.00th=[ 132], 99.50th=[ 153], 99.90th=[ 182], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 632, max= 968, per=24.90%, avg=794.08, stdev=100.06, samples=25 iops : min= 158, max= 242, avg=198.44, stdev=25.06, samples=25 lat (msec) : 20=0.55%, 50=3.16%, 100=85.82%, 250=10.47% cpu : usr=0.94%, sys=2.71%, ctx=1314, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=26755: Thu May 1 01:58:52 2025 write: IOPS=199, BW=798KiB/s (817kB/s)(10.0MiB/12837msec) slat (usec): min=10, max=9269, avg=117.84, stdev=310.08 clat (msec): min=26, max=181, avg=79.64, stdev=17.92 lat (msec): min=26, max=181, avg=79.76, stdev=17.93 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 54], 10.00th=[ 59], 20.00th=[ 66], | 30.00th=[ 70], 40.00th=[ 75], 50.00th=[ 80], 60.00th=[ 82], | 70.00th=[ 87], 80.00th=[ 93], 90.00th=[ 103], 95.00th=[ 110], | 99.00th=[ 130], 99.50th=[ 144], 99.90th=[ 180], 99.95th=[ 182], | 99.99th=[ 182] bw ( KiB/s): min= 638, max= 974, per=24.77%, avg=789.84, stdev=90.92, samples=25 iops : min= 159, max= 243, avg=197.36, stdev=22.77, samples=25 lat (msec) : 50=2.34%, 100=86.09%, 250=11.56% cpu : usr=0.76%, sys=2.63%, ctx=1162, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=26756: Thu May 1 01:58:52 2025 write: IOPS=199, BW=797KiB/s (816kB/s)(10.0MiB/12843msec) slat (usec): min=15, max=10191, avg=111.73, stdev=308.48 clat (msec): min=4, max=182, avg=79.79, stdev=17.77 lat (msec): min=4, max=182, avg=79.90, stdev=17.78 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 55], 10.00th=[ 60], 20.00th=[ 65], | 30.00th=[ 71], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 83], | 70.00th=[ 88], 80.00th=[ 93], 90.00th=[ 102], 95.00th=[ 110], | 99.00th=[ 132], 99.50th=[ 144], 99.90th=[ 182], 99.95th=[ 184], | 99.99th=[ 184] bw ( KiB/s): min= 584, max= 960, per=24.75%, avg=789.40, stdev=100.78, samples=25 iops : min= 146, max= 240, avg=197.20, stdev=25.19, samples=25 lat (msec) : 10=0.04%, 20=0.04%, 50=2.03%, 100=86.45%, 250=11.45% cpu : usr=0.59%, sys=2.75%, ctx=1082, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3189KiB/s (3266kB/s), 797KiB/s-800KiB/s (816kB/s-820kB/s), io=40.0MiB (41.9MB), run=12795-12843msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26776: Thu May 1 01:59:06 2025 read: IOPS=103, BW=412KiB/s (422kB/s)(5048KiB/12251msec) slat (usec): min=16, max=27127, avg=5223.31, stdev=2687.79 clat (msec): min=15, max=139, avg=72.13, stdev=21.23 lat (msec): min=15, max=146, avg=77.36, stdev=22.40 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 41], 10.00th=[ 46], 20.00th=[ 55], | 30.00th=[ 61], 40.00th=[ 66], 50.00th=[ 71], 60.00th=[ 77], | 70.00th=[ 82], 80.00th=[ 89], 90.00th=[ 102], 95.00th=[ 111], | 99.00th=[ 130], 99.50th=[ 132], 99.90th=[ 138], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 271, max= 670, per=25.18%, avg=409.00, stdev=106.32, samples=24 iops : min= 67, max= 167, avg=102.04, stdev=26.64, samples=24 write: IOPS=105, BW=424KiB/s (434kB/s)(5192KiB/12251msec) slat (usec): min=14, max=2960, avg=112.32, stdev=208.40 clat (msec): min=31, max=140, avg=75.46, stdev=20.26 lat (msec): min=31, max=140, avg=75.57, stdev=20.27 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 45], 10.00th=[ 50], 20.00th=[ 58], | 30.00th=[ 64], 40.00th=[ 69], 50.00th=[ 74], 60.00th=[ 81], | 70.00th=[ 87], 80.00th=[ 95], 90.00th=[ 103], 95.00th=[ 109], | 99.00th=[ 124], 99.50th=[ 132], 99.90th=[ 136], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 320, max= 630, per=25.05%, avg=423.33, stdev=87.92, samples=24 iops : min= 80, max= 157, avg=105.63, stdev=22.03, samples=24 lat (msec) : 20=0.16%, 50=12.42%, 100=75.66%, 250=11.76% cpu : usr=1.60%, sys=4.84%, ctx=2340, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26777: Thu May 1 01:59:06 2025 read: IOPS=100, BW=403KiB/s (413kB/s)(4976KiB/12351msec) slat (usec): min=1281, max=16551, avg=5185.59, stdev=2445.22 clat (msec): min=15, max=155, avg=72.93, stdev=22.00 lat (msec): min=19, max=162, avg=78.12, stdev=23.06 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 42], 10.00th=[ 46], 20.00th=[ 55], | 30.00th=[ 60], 40.00th=[ 66], 50.00th=[ 71], 60.00th=[ 78], | 70.00th=[ 83], 80.00th=[ 92], 90.00th=[ 102], 95.00th=[ 113], | 99.00th=[ 131], 99.50th=[ 138], 99.90th=[ 146], 99.95th=[ 157], | 99.99th=[ 157] bw ( KiB/s): min= 216, max= 614, per=24.55%, avg=398.67, stdev=100.82, samples=24 iops : min= 54, max= 153, avg=99.46, stdev=25.22, samples=24 write: IOPS=106, BW=426KiB/s (436kB/s)(5264KiB/12351msec) slat (usec): min=13, max=4944, avg=114.72, stdev=267.74 clat (msec): min=19, max=142, avg=75.10, stdev=20.28 lat (msec): min=19, max=142, avg=75.22, stdev=20.27 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 44], 10.00th=[ 50], 20.00th=[ 57], | 30.00th=[ 64], 40.00th=[ 69], 50.00th=[ 74], 60.00th=[ 81], | 70.00th=[ 86], 80.00th=[ 93], 90.00th=[ 102], 95.00th=[ 109], | 99.00th=[ 128], 99.50th=[ 136], 99.90th=[ 142], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 279, max= 678, per=25.13%, avg=424.67, stdev=110.16, samples=24 iops : min= 69, max= 169, avg=105.96, stdev=27.61, samples=24 lat (msec) : 20=0.08%, 50=12.73%, 100=75.90%, 250=11.29% cpu : usr=1.12%, sys=5.64%, ctx=2322, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26778: Thu May 1 01:59:06 2025 read: IOPS=100, BW=404KiB/s (413kB/s)(4984KiB/12345msec) slat (usec): min=1042, max=27949, avg=5215.11, stdev=2501.23 clat (msec): min=7, max=142, avg=73.56, stdev=21.33 lat (msec): min=9, max=150, avg=78.78, stdev=22.40 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 42], 10.00th=[ 47], 20.00th=[ 55], | 30.00th=[ 62], 40.00th=[ 67], 50.00th=[ 72], 60.00th=[ 78], | 70.00th=[ 84], 80.00th=[ 92], 90.00th=[ 104], 95.00th=[ 114], | 99.00th=[ 125], 99.50th=[ 130], 99.90th=[ 138], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 232, max= 718, per=24.83%, avg=403.29, stdev=106.26, samples=24 iops : min= 58, max= 179, avg=100.62, stdev=26.51, samples=24 write: IOPS=106, BW=426KiB/s (436kB/s)(5256KiB/12345msec) slat (usec): min=13, max=4673, avg=111.88, stdev=232.20 clat (msec): min=20, max=136, avg=74.58, stdev=19.73 lat (msec): min=21, max=136, avg=74.70, stdev=19.74 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 45], 10.00th=[ 50], 20.00th=[ 58], | 30.00th=[ 64], 40.00th=[ 68], 50.00th=[ 73], 60.00th=[ 80], | 70.00th=[ 85], 80.00th=[ 91], 90.00th=[ 101], 95.00th=[ 110], | 99.00th=[ 128], 99.50th=[ 134], 99.90th=[ 136], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 280, max= 702, per=24.95%, avg=421.62, stdev=103.18, samples=24 iops : min= 70, max= 175, avg=105.21, stdev=25.80, samples=24 lat (msec) : 10=0.04%, 20=0.04%, 50=12.03%, 100=76.76%, 250=11.13% cpu : usr=0.70%, sys=5.87%, ctx=2394, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26779: Thu May 1 01:59:06 2025 read: IOPS=102, BW=409KiB/s (419kB/s)(5060KiB/12357msec) slat (usec): min=1184, max=18589, avg=5156.13, stdev=2395.88 clat (msec): min=4, max=144, avg=72.34, stdev=21.17 lat (msec): min=10, max=151, avg=77.51, stdev=22.21 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 42], 10.00th=[ 47], 20.00th=[ 55], | 30.00th=[ 60], 40.00th=[ 65], 50.00th=[ 70], 60.00th=[ 77], | 70.00th=[ 84], 80.00th=[ 91], 90.00th=[ 102], 95.00th=[ 112], | 99.00th=[ 124], 99.50th=[ 130], 99.90th=[ 144], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 224, max= 654, per=24.96%, avg=405.33, stdev=112.25, samples=24 iops : min= 56, max= 163, avg=101.12, stdev=28.16, samples=24 write: IOPS=104, BW=419KiB/s (429kB/s)(5180KiB/12357msec) slat (usec): min=15, max=9395, avg=130.39, stdev=366.98 clat (msec): min=10, max=138, avg=75.69, stdev=20.98 lat (msec): min=12, max=139, avg=75.82, stdev=21.00 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 44], 10.00th=[ 50], 20.00th=[ 57], | 30.00th=[ 64], 40.00th=[ 69], 50.00th=[ 74], 60.00th=[ 81], | 70.00th=[ 87], 80.00th=[ 94], 90.00th=[ 105], 95.00th=[ 112], | 99.00th=[ 128], 99.50th=[ 134], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 296, max= 648, per=24.52%, avg=414.33, stdev=87.90, samples=24 iops : min= 74, max= 162, avg=103.38, stdev=21.97, samples=24 lat (msec) : 10=0.04%, 20=0.08%, 50=12.70%, 100=74.77%, 250=12.42% cpu : usr=0.00%, sys=6.70%, ctx=2354, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1624KiB/s (1663kB/s), 403KiB/s-412KiB/s (413kB/s-422kB/s), io=19.6MiB (20.5MB), run=12251-12357msec WRITE: bw=1691KiB/s (1731kB/s), 419KiB/s-426KiB/s (429kB/s-436kB/s), io=20.4MiB (21.4MB), run=12251-12357msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=26793: Thu May 1 01:59:08 2025 read: IOPS=3, BW=121MiB/s (127MB/s)(40.0MiB/330msec) slat (nsec): min=16445k, max=16445k, avg=16444782.00, stdev= 0.00 clat (nsec): min=301935k, max=301935k, avg=301934913.00, stdev= 0.00 lat (nsec): min=318407k, max=318407k, avg=318407108.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 300], 10.00th=[ 300], 20.00th=[ 300], | 30.00th=[ 300], 40.00th=[ 300], 50.00th=[ 300], 60.00th=[ 300], | 70.00th=[ 300], 80.00th=[ 300], 90.00th=[ 300], 95.00th=[ 300], | 99.00th=[ 300], 99.50th=[ 300], 99.90th=[ 300], 99.95th=[ 300], | 99.99th=[ 300] lat (msec) : 500=100.00% cpu : usr=0.61%, sys=2.74%, ctx=7, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=121MiB/s (127MB/s), 121MiB/s-121MiB/s (127MB/s-127MB/s), io=40.0MiB (41.9MB), run=330-330msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout