== sanity test 398c: run fio to test AIO ================= 19:04:35 (1746140675) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.137467 s, 305 MB/s osc.lustre-OST0000-osc-ffff88008bb8c000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=25081: Thu May 1 19:04:49 2025 write: IOPS=220, BW=883KiB/s (904kB/s)(10.0MiB/11597msec) slat (usec): min=11, max=5799, avg=102.04, stdev=233.15 clat (msec): min=34, max=135, avg=72.18, stdev=16.80 lat (msec): min=34, max=135, avg=72.28, stdev=16.80 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 46], 10.00th=[ 53], 20.00th=[ 59], | 30.00th=[ 63], 40.00th=[ 67], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 80], 80.00th=[ 85], 90.00th=[ 94], 95.00th=[ 104], | 99.00th=[ 121], 99.50th=[ 124], 99.90th=[ 134], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 680, max= 1168, per=24.94%, avg=879.83, stdev=125.92, samples=23 iops : min= 170, max= 292, avg=219.83, stdev=31.44, samples=23 lat (msec) : 50=8.28%, 100=85.62%, 250=6.09% cpu : usr=0.00%, sys=3.49%, ctx=1389, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=25082: Thu May 1 19:04:49 2025 write: IOPS=220, BW=882KiB/s (904kB/s)(10.0MiB/11605msec) slat (usec): min=11, max=3716, avg=115.57, stdev=241.27 clat (msec): min=28, max=134, avg=72.17, stdev=16.64 lat (msec): min=28, max=134, avg=72.29, stdev=16.65 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 48], 10.00th=[ 52], 20.00th=[ 59], | 30.00th=[ 63], 40.00th=[ 67], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 81], 80.00th=[ 86], 90.00th=[ 94], 95.00th=[ 102], | 99.00th=[ 121], 99.50th=[ 127], 99.90th=[ 132], 99.95th=[ 132], | 99.99th=[ 136] bw ( KiB/s): min= 662, max= 1184, per=24.91%, avg=879.00, stdev=129.46, samples=23 iops : min= 165, max= 296, avg=219.70, stdev=32.41, samples=23 lat (msec) : 50=7.46%, 100=86.21%, 250=6.33% cpu : usr=0.48%, sys=3.45%, ctx=1577, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=25083: Thu May 1 19:04:49 2025 write: IOPS=222, BW=888KiB/s (910kB/s)(10.0MiB/11528msec) slat (usec): min=11, max=5440, avg=99.84, stdev=246.17 clat (msec): min=16, max=154, avg=71.77, stdev=17.34 lat (msec): min=16, max=154, avg=71.87, stdev=17.34 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 47], 10.00th=[ 52], 20.00th=[ 59], | 30.00th=[ 63], 40.00th=[ 67], 50.00th=[ 70], 60.00th=[ 74], | 70.00th=[ 80], 80.00th=[ 85], 90.00th=[ 93], 95.00th=[ 104], | 99.00th=[ 124], 99.50th=[ 134], 99.90th=[ 134], 99.95th=[ 136], | 99.99th=[ 155] bw ( KiB/s): min= 640, max= 1152, per=25.08%, avg=884.65, stdev=132.06, samples=23 iops : min= 160, max= 288, avg=221.09, stdev=33.01, samples=23 lat (msec) : 20=0.31%, 50=7.77%, 100=86.17%, 250=5.74% cpu : usr=0.56%, sys=2.88%, ctx=1232, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=25084: Thu May 1 19:04:49 2025 write: IOPS=220, BW=882KiB/s (903kB/s)(10.0MiB/11609msec) slat (usec): min=11, max=4956, avg=108.98, stdev=255.22 clat (msec): min=21, max=134, avg=72.15, stdev=16.50 lat (msec): min=21, max=135, avg=72.26, stdev=16.50 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 48], 10.00th=[ 53], 20.00th=[ 59], | 30.00th=[ 63], 40.00th=[ 67], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 80], 80.00th=[ 85], 90.00th=[ 93], 95.00th=[ 103], | 99.00th=[ 121], 99.50th=[ 125], 99.90th=[ 136], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 686, max= 1213, per=24.89%, avg=878.13, stdev=125.09, samples=23 iops : min= 171, max= 303, avg=219.39, stdev=31.30, samples=23 lat (msec) : 50=7.85%, 100=85.74%, 250=6.41% cpu : usr=0.52%, sys=3.15%, ctx=1385, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3528KiB/s (3613kB/s), 882KiB/s-888KiB/s (903kB/s-910kB/s), io=40.0MiB (41.9MB), run=11528-11609msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25103: Thu May 1 19:05:01 2025 read: IOPS=109, BW=438KiB/s (448kB/s)(5048KiB/11532msec) slat (usec): min=1298, max=23854, avg=4649.99, stdev=2127.68 clat (msec): min=11, max=124, avg=68.14, stdev=16.84 lat (msec): min=15, max=130, avg=72.80, stdev=17.43 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 45], 10.00th=[ 48], 20.00th=[ 54], | 30.00th=[ 58], 40.00th=[ 63], 50.00th=[ 67], 60.00th=[ 71], | 70.00th=[ 77], 80.00th=[ 83], 90.00th=[ 91], 95.00th=[ 100], | 99.00th=[ 113], 99.50th=[ 117], 99.90th=[ 125], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 288, max= 576, per=25.05%, avg=435.30, stdev=79.03, samples=23 iops : min= 72, max= 144, avg=108.61, stdev=19.81, samples=23 write: IOPS=112, BW=450KiB/s (461kB/s)(5192KiB/11532msec) slat (usec): min=15, max=6937, avg=106.28, stdev=286.58 clat (msec): min=13, max=125, avg=70.28, stdev=16.50 lat (msec): min=13, max=126, avg=70.39, stdev=16.51 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 47], 10.00th=[ 51], 20.00th=[ 56], | 30.00th=[ 61], 40.00th=[ 64], 50.00th=[ 69], 60.00th=[ 73], | 70.00th=[ 79], 80.00th=[ 86], 90.00th=[ 93], 95.00th=[ 101], | 99.00th=[ 116], 99.50th=[ 122], 99.90th=[ 126], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min= 336, max= 560, per=24.81%, avg=448.83, stdev=74.60, samples=23 iops : min= 84, max= 140, avg=112.00, stdev=18.70, samples=23 lat (msec) : 20=0.08%, 50=10.62%, 100=84.57%, 250=4.73% cpu : usr=1.46%, sys=4.91%, ctx=2343, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25104: Thu May 1 19:05:01 2025 read: IOPS=107, BW=431KiB/s (442kB/s)(4976KiB/11532msec) slat (usec): min=1255, max=23956, avg=4660.09, stdev=2044.61 clat (msec): min=30, max=135, avg=68.50, stdev=17.01 lat (msec): min=33, max=145, avg=73.16, stdev=17.68 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 45], 10.00th=[ 50], 20.00th=[ 55], | 30.00th=[ 58], 40.00th=[ 62], 50.00th=[ 66], 60.00th=[ 71], | 70.00th=[ 77], 80.00th=[ 83], 90.00th=[ 92], 95.00th=[ 101], | 99.00th=[ 116], 99.50th=[ 121], 99.90th=[ 134], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 304, max= 576, per=24.67%, avg=428.78, stdev=68.20, samples=23 iops : min= 76, max= 144, avg=107.00, stdev=16.97, samples=23 write: IOPS=114, BW=456KiB/s (467kB/s)(5264KiB/11532msec) slat (usec): min=15, max=2134, avg=96.29, stdev=180.77 clat (msec): min=16, max=132, avg=69.89, stdev=15.57 lat (msec): min=16, max=133, avg=69.99, stdev=15.57 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 48], 10.00th=[ 53], 20.00th=[ 57], | 30.00th=[ 61], 40.00th=[ 65], 50.00th=[ 69], 60.00th=[ 73], | 70.00th=[ 78], 80.00th=[ 84], 90.00th=[ 90], 95.00th=[ 97], | 99.00th=[ 110], 99.50th=[ 120], 99.90th=[ 126], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 328, max= 606, per=25.18%, avg=455.48, stdev=82.79, samples=23 iops : min= 82, max= 151, avg=113.70, stdev=20.71, samples=23 lat (msec) : 20=0.08%, 50=9.61%, 100=86.29%, 250=4.02% cpu : usr=0.48%, sys=5.84%, ctx=2438, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25105: Thu May 1 19:05:01 2025 read: IOPS=107, BW=432KiB/s (442kB/s)(4984KiB/11544msec) slat (usec): min=1147, max=22443, avg=4740.84, stdev=2113.84 clat (msec): min=9, max=134, avg=68.43, stdev=17.07 lat (msec): min=13, max=146, avg=73.17, stdev=17.70 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 45], 10.00th=[ 49], 20.00th=[ 54], | 30.00th=[ 58], 40.00th=[ 63], 50.00th=[ 67], 60.00th=[ 71], | 70.00th=[ 77], 80.00th=[ 84], 90.00th=[ 92], 95.00th=[ 100], | 99.00th=[ 112], 99.50th=[ 124], 99.90th=[ 129], 99.95th=[ 136], | 99.99th=[ 136] bw ( KiB/s): min= 295, max= 536, per=24.82%, avg=431.39, stdev=67.09, samples=23 iops : min= 73, max= 134, avg=107.61, stdev=16.84, samples=23 write: IOPS=113, BW=455KiB/s (466kB/s)(5256KiB/11544msec) slat (usec): min=16, max=1971, avg=97.45, stdev=171.16 clat (msec): min=9, max=132, avg=70.05, stdev=16.19 lat (msec): min=9, max=132, avg=70.15, stdev=16.19 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 49], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 60], 40.00th=[ 64], 50.00th=[ 68], 60.00th=[ 72], | 70.00th=[ 79], 80.00th=[ 85], 90.00th=[ 92], 95.00th=[ 99], | 99.00th=[ 117], 99.50th=[ 121], 99.90th=[ 129], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 312, max= 616, per=25.02%, avg=452.52, stdev=86.07, samples=23 iops : min= 78, max= 154, avg=112.91, stdev=21.52, samples=23 lat (msec) : 10=0.08%, 20=0.04%, 50=10.00%, 100=85.31%, 250=4.57% cpu : usr=0.00%, sys=6.31%, ctx=2357, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25106: Thu May 1 19:05:01 2025 read: IOPS=110, BW=443KiB/s (454kB/s)(5060KiB/11411msec) slat (usec): min=27, max=24070, avg=4564.51, stdev=2033.07 clat (msec): min=11, max=134, avg=67.32, stdev=17.20 lat (msec): min=11, max=142, avg=71.89, stdev=17.91 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 44], 10.00th=[ 48], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 61], 50.00th=[ 66], 60.00th=[ 70], | 70.00th=[ 75], 80.00th=[ 83], 90.00th=[ 91], 95.00th=[ 100], | 99.00th=[ 113], 99.50th=[ 116], 99.90th=[ 134], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 304, max= 592, per=25.59%, avg=444.73, stdev=81.38, samples=22 iops : min= 76, max= 148, avg=111.05, stdev=20.32, samples=22 write: IOPS=113, BW=454KiB/s (465kB/s)(5180KiB/11411msec) slat (usec): min=12, max=3577, avg=104.19, stdev=229.92 clat (msec): min=18, max=151, avg=70.41, stdev=16.64 lat (msec): min=18, max=151, avg=70.51, stdev=16.64 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 47], 10.00th=[ 52], 20.00th=[ 56], | 30.00th=[ 61], 40.00th=[ 65], 50.00th=[ 69], 60.00th=[ 73], | 70.00th=[ 80], 80.00th=[ 85], 90.00th=[ 93], 95.00th=[ 99], | 99.00th=[ 112], 99.50th=[ 121], 99.90th=[ 153], 99.95th=[ 153], | 99.99th=[ 153] bw ( KiB/s): min= 328, max= 568, per=25.17%, avg=455.27, stdev=61.67, samples=22 iops : min= 82, max= 142, avg=113.68, stdev=15.39, samples=22 lat (msec) : 20=0.23%, 50=10.94%, 100=84.45%, 250=4.38% cpu : usr=0.96%, sys=5.30%, ctx=2448, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1738KiB/s (1780kB/s), 431KiB/s-443KiB/s (442kB/s-454kB/s), io=19.6MiB (20.5MB), run=11411-11544msec WRITE: bw=1810KiB/s (1853kB/s), 450KiB/s-456KiB/s (461kB/s-467kB/s), io=20.4MiB (21.4MB), run=11411-11544msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=25119: Thu May 1 19:05:03 2025 read: IOPS=3, BW=133MiB/s (140MB/s)(40.0MiB/300msec) slat (nsec): min=9556.7k, max=9556.7k, avg=9556684.00, stdev= 0.00 clat (nsec): min=278908k, max=278908k, avg=278907847.00, stdev= 0.00 lat (nsec): min=288487k, max=288487k, avg=288487335.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 279], 10.00th=[ 279], 20.00th=[ 279], | 30.00th=[ 279], 40.00th=[ 279], 50.00th=[ 279], 60.00th=[ 279], | 70.00th=[ 279], 80.00th=[ 279], 90.00th=[ 279], 95.00th=[ 279], | 99.00th=[ 279], 99.50th=[ 279], 99.90th=[ 279], 99.95th=[ 279], | 99.99th=[ 279] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=3.01%, ctx=5, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=133MiB/s (140MB/s), 133MiB/s-133MiB/s (140MB/s-140MB/s), io=40.0MiB (41.9MB), run=300-300msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout