== sanity test 398c: run fio to test AIO ================= 19:57:53 (1746143873) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.220075 s, 191 MB/s osc.lustre-OST0000-osc-ffff8800b676e800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=23389: Thu May 1 19:58:10 2025 write: IOPS=174, BW=698KiB/s (714kB/s)(10.0MiB/14681msec) slat (usec): min=9, max=3590, avg=96.84, stdev=238.82 clat (msec): min=12, max=234, avg=91.36, stdev=18.00 lat (msec): min=12, max=234, avg=91.46, stdev=18.01 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 66], 10.00th=[ 71], 20.00th=[ 80], | 30.00th=[ 84], 40.00th=[ 87], 50.00th=[ 90], 60.00th=[ 95], | 70.00th=[ 100], 80.00th=[ 104], 90.00th=[ 112], 95.00th=[ 121], | 99.00th=[ 138], 99.50th=[ 144], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 558, max= 880, per=25.10%, avg=699.14, stdev=73.99, samples=29 iops : min= 139, max= 220, avg=174.62, stdev=18.55, samples=29 lat (msec) : 20=0.08%, 50=1.25%, 100=72.15%, 250=26.52% cpu : usr=0.72%, sys=1.84%, ctx=953, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23390: Thu May 1 19:58:10 2025 write: IOPS=174, BW=696KiB/s (713kB/s)(10.0MiB/14706msec) slat (usec): min=11, max=3339, avg=102.65, stdev=234.99 clat (msec): min=6, max=214, avg=91.36, stdev=17.72 lat (msec): min=6, max=214, avg=91.47, stdev=17.72 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 68], 10.00th=[ 70], 20.00th=[ 79], | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 90], 60.00th=[ 94], | 70.00th=[ 99], 80.00th=[ 105], 90.00th=[ 112], 95.00th=[ 122], | 99.00th=[ 142], 99.50th=[ 150], 99.90th=[ 215], 99.95th=[ 215], | 99.99th=[ 215] bw ( KiB/s): min= 528, max= 856, per=25.07%, avg=698.14, stdev=72.87, samples=29 iops : min= 132, max= 214, avg=174.34, stdev=18.23, samples=29 lat (msec) : 10=0.04%, 50=0.78%, 100=72.66%, 250=26.52% cpu : usr=0.50%, sys=2.37%, ctx=1004, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23391: Thu May 1 19:58:10 2025 write: IOPS=174, BW=696KiB/s (713kB/s)(10.0MiB/14705msec) slat (usec): min=10, max=3339, avg=114.50, stdev=270.98 clat (msec): min=8, max=234, avg=91.49, stdev=18.01 lat (msec): min=8, max=234, avg=91.61, stdev=18.01 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 66], 10.00th=[ 71], 20.00th=[ 79], | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 91], 60.00th=[ 95], | 70.00th=[ 99], 80.00th=[ 105], 90.00th=[ 112], 95.00th=[ 121], | 99.00th=[ 140], 99.50th=[ 144], 99.90th=[ 234], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 568, max= 864, per=25.06%, avg=697.79, stdev=65.99, samples=29 iops : min= 142, max= 216, avg=174.28, stdev=16.54, samples=29 lat (msec) : 10=0.04%, 20=0.08%, 50=0.59%, 100=72.46%, 250=26.84% cpu : usr=0.50%, sys=2.69%, ctx=1278, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23392: Thu May 1 19:58:10 2025 write: IOPS=174, BW=696KiB/s (713kB/s)(10.0MiB/14705msec) slat (usec): min=10, max=5955, avg=117.00, stdev=279.62 clat (msec): min=10, max=242, avg=91.40, stdev=17.31 lat (msec): min=10, max=242, avg=91.51, stdev=17.30 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 66], 10.00th=[ 70], 20.00th=[ 79], | 30.00th=[ 84], 40.00th=[ 87], 50.00th=[ 90], 60.00th=[ 95], | 70.00th=[ 99], 80.00th=[ 105], 90.00th=[ 113], 95.00th=[ 121], | 99.00th=[ 140], 99.50th=[ 146], 99.90th=[ 180], 99.95th=[ 234], | 99.99th=[ 243] bw ( KiB/s): min= 544, max= 816, per=25.04%, avg=697.45, stdev=61.60, samples=29 iops : min= 136, max= 204, avg=174.28, stdev=15.39, samples=29 lat (msec) : 20=0.12%, 50=0.04%, 100=73.52%, 250=26.33% cpu : usr=0.19%, sys=2.99%, ctx=1399, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2785KiB/s (2852kB/s), 696KiB/s-698KiB/s (713kB/s-714kB/s), io=40.0MiB (41.9MB), run=14681-14706msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23414: Thu May 1 19:58:22 2025 read: IOPS=123, BW=493KiB/s (505kB/s)(5048KiB/10241msec) slat (usec): min=1300, max=13659, avg=4103.25, stdev=1649.52 clat (msec): min=8, max=117, avg=59.31, stdev=14.72 lat (msec): min=12, max=123, avg=63.42, stdev=15.38 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 39], 10.00th=[ 42], 20.00th=[ 47], | 30.00th=[ 51], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 71], 90.00th=[ 80], 95.00th=[ 86], | 99.00th=[ 99], 99.50th=[ 101], 99.90th=[ 109], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 271, max= 712, per=24.87%, avg=487.30, stdev=113.94, samples=20 iops : min= 67, max= 178, avg=121.70, stdev=28.50, samples=20 write: IOPS=126, BW=507KiB/s (519kB/s)(5192KiB/10241msec) slat (usec): min=12, max=2608, avg=91.48, stdev=186.93 clat (msec): min=8, max=118, avg=63.02, stdev=14.72 lat (msec): min=8, max=118, avg=63.11, stdev=14.72 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[ 51], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 62], 60.00th=[ 66], | 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 84], 95.00th=[ 89], | 99.00th=[ 103], 99.50th=[ 108], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 327, max= 632, per=24.46%, avg=498.90, stdev=87.58, samples=20 iops : min= 81, max= 158, avg=124.60, stdev=21.96, samples=20 lat (msec) : 10=0.08%, 20=0.16%, 50=24.26%, 100=74.61%, 250=0.90% cpu : usr=0.73%, sys=5.95%, ctx=2321, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23415: Thu May 1 19:58:22 2025 read: IOPS=121, BW=487KiB/s (499kB/s)(4976KiB/10210msec) slat (usec): min=1504, max=13574, avg=4098.17, stdev=1734.01 clat (msec): min=9, max=118, avg=59.81, stdev=14.69 lat (msec): min=12, max=124, avg=63.91, stdev=15.36 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 48], | 30.00th=[ 52], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 62], | 70.00th=[ 66], 80.00th=[ 72], 90.00th=[ 81], 95.00th=[ 87], | 99.00th=[ 101], 99.50th=[ 105], 99.90th=[ 120], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min= 304, max= 680, per=24.69%, avg=483.70, stdev=100.17, samples=20 iops : min= 76, max= 170, avg=120.80, stdev=25.06, samples=20 write: IOPS=128, BW=516KiB/s (528kB/s)(5264KiB/10210msec) slat (usec): min=11, max=2727, avg=98.25, stdev=196.54 clat (msec): min=30, max=120, avg=62.92, stdev=14.30 lat (msec): min=30, max=120, avg=63.02, stdev=14.31 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 43], 10.00th=[ 46], 20.00th=[ 51], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 70], 80.00th=[ 75], 90.00th=[ 83], 95.00th=[ 89], | 99.00th=[ 100], 99.50th=[ 111], 99.90th=[ 115], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 359, max= 694, per=25.06%, avg=511.30, stdev=93.06, samples=20 iops : min= 89, max= 173, avg=127.70, stdev=23.27, samples=20 lat (msec) : 10=0.04%, 20=0.04%, 50=22.81%, 100=76.13%, 250=0.98% cpu : usr=0.00%, sys=6.72%, ctx=2282, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23416: Thu May 1 19:58:22 2025 read: IOPS=121, BW=487KiB/s (499kB/s)(4984KiB/10236msec) slat (usec): min=1235, max=14196, avg=4206.57, stdev=1691.07 clat (msec): min=3, max=117, avg=60.65, stdev=15.04 lat (msec): min=5, max=125, avg=64.86, stdev=15.73 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 48], | 30.00th=[ 52], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 63], | 70.00th=[ 68], 80.00th=[ 73], 90.00th=[ 82], 95.00th=[ 88], | 99.00th=[ 103], 99.50th=[ 110], 99.90th=[ 115], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 295, max= 646, per=24.71%, avg=484.05, stdev=87.30, samples=20 iops : min= 73, max= 161, avg=120.90, stdev=21.82, samples=20 write: IOPS=128, BW=513KiB/s (526kB/s)(5256KiB/10236msec) slat (usec): min=11, max=3041, avg=89.29, stdev=153.88 clat (msec): min=31, max=120, avg=61.88, stdev=14.08 lat (msec): min=31, max=120, avg=61.97, stdev=14.09 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 57], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 73], 90.00th=[ 81], 95.00th=[ 87], | 99.00th=[ 101], 99.50th=[ 111], 99.90th=[ 121], 99.95th=[ 121], | 99.99th=[ 121] bw ( KiB/s): min= 304, max= 742, per=24.85%, avg=506.90, stdev=119.80, samples=20 iops : min= 76, max= 185, avg=126.60, stdev=29.99, samples=20 lat (msec) : 4=0.04%, 50=24.38%, 100=74.49%, 250=1.09% cpu : usr=1.91%, sys=4.79%, ctx=2354, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23417: Thu May 1 19:58:22 2025 read: IOPS=124, BW=498KiB/s (510kB/s)(5060KiB/10157msec) slat (usec): min=40, max=12946, avg=4102.30, stdev=1689.03 clat (msec): min=6, max=115, avg=60.10, stdev=14.75 lat (msec): min=6, max=121, avg=64.21, stdev=15.44 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 39], 10.00th=[ 44], 20.00th=[ 48], | 30.00th=[ 52], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 72], 90.00th=[ 81], 95.00th=[ 88], | 99.00th=[ 100], 99.50th=[ 105], 99.90th=[ 116], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 336, max= 656, per=25.33%, avg=496.25, stdev=92.46, samples=20 iops : min= 84, max= 164, avg=124.00, stdev=23.11, samples=20 write: IOPS=127, BW=510KiB/s (522kB/s)(5180KiB/10157msec) slat (usec): min=12, max=2569, avg=92.91, stdev=161.57 clat (msec): min=14, max=124, avg=62.43, stdev=14.45 lat (msec): min=14, max=124, avg=62.52, stdev=14.45 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 50], | 30.00th=[ 54], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 65], | 70.00th=[ 70], 80.00th=[ 75], 90.00th=[ 83], 95.00th=[ 88], | 99.00th=[ 102], 99.50th=[ 111], 99.90th=[ 118], 99.95th=[ 125], | 99.99th=[ 125] bw ( KiB/s): min= 335, max= 656, per=24.82%, avg=506.25, stdev=83.81, samples=20 iops : min= 83, max= 164, avg=126.50, stdev=21.01, samples=20 lat (msec) : 10=0.12%, 20=0.12%, 50=23.01%, 100=75.78%, 250=0.98% cpu : usr=0.00%, sys=6.81%, ctx=2363, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1960KiB/s (2007kB/s), 487KiB/s-498KiB/s (499kB/s-510kB/s), io=19.6MiB (20.5MB), run=10157-10241msec WRITE: bw=2040KiB/s (2089kB/s), 507KiB/s-516KiB/s (519kB/s-528kB/s), io=20.4MiB (21.4MB), run=10157-10241msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=23429: Thu May 1 19:58:23 2025 read: IOPS=6, BW=276MiB/s (289MB/s)(40.0MiB/145msec) slat (nsec): min=3134.5k, max=3134.5k, avg=3134481.00, stdev= 0.00 clat (nsec): min=131814k, max=131814k, avg=131813509.00, stdev= 0.00 lat (nsec): min=134970k, max=134970k, avg=134970304.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 132], 10.00th=[ 132], 20.00th=[ 132], | 30.00th=[ 132], 40.00th=[ 132], 50.00th=[ 132], 60.00th=[ 132], | 70.00th=[ 132], 80.00th=[ 132], 90.00th=[ 132], 95.00th=[ 132], | 99.00th=[ 132], 99.50th=[ 132], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 132] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=1.39%, ctx=5, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=276MiB/s (289MB/s), 276MiB/s-276MiB/s (289MB/s-289MB/s), io=40.0MiB (41.9MB), run=145-145msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout