== sanity test 398c: run fio to test AIO ================= 20:33:10 (1746145990) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.17437 s, 241 MB/s osc.lustre-OST0000-osc-ffff8800b6d64000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=23533: Thu May 1 20:33:24 2025 write: IOPS=216, BW=866KiB/s (887kB/s)(10.0MiB/11823msec) slat (usec): min=10, max=3377, avg=88.45, stdev=192.05 clat (msec): min=14, max=141, avg=73.41, stdev=17.82 lat (msec): min=14, max=141, avg=73.50, stdev=17.82 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 59], | 30.00th=[ 63], 40.00th=[ 66], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 79], 80.00th=[ 87], 90.00th=[ 100], 95.00th=[ 110], | 99.00th=[ 126], 99.50th=[ 140], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 142] bw ( KiB/s): min= 512, max= 1040, per=24.98%, avg=865.43, stdev=153.43, samples=23 iops : min= 128, max= 260, avg=216.26, stdev=38.42, samples=23 lat (msec) : 20=0.08%, 50=3.52%, 100=87.34%, 250=9.06% cpu : usr=0.00%, sys=3.01%, ctx=982, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23534: Thu May 1 20:33:24 2025 write: IOPS=217, BW=869KiB/s (889kB/s)(10.0MiB/11790msec) slat (usec): min=9, max=4810, avg=97.99, stdev=235.31 clat (msec): min=13, max=150, avg=73.25, stdev=17.24 lat (msec): min=13, max=150, avg=73.35, stdev=17.25 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 53], 10.00th=[ 56], 20.00th=[ 59], | 30.00th=[ 64], 40.00th=[ 67], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 80], 80.00th=[ 86], 90.00th=[ 100], 95.00th=[ 108], | 99.00th=[ 123], 99.50th=[ 134], 99.90th=[ 140], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 550, max= 1064, per=25.07%, avg=868.39, stdev=149.81, samples=23 iops : min= 137, max= 266, avg=216.96, stdev=37.58, samples=23 lat (msec) : 20=0.04%, 50=3.05%, 100=88.28%, 250=8.63% cpu : usr=0.29%, sys=2.94%, ctx=1085, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23535: Thu May 1 20:33:24 2025 write: IOPS=218, BW=872KiB/s (893kB/s)(10.0MiB/11739msec) slat (usec): min=9, max=4088, avg=101.53, stdev=210.30 clat (msec): min=7, max=139, avg=73.00, stdev=17.58 lat (msec): min=7, max=139, avg=73.11, stdev=17.59 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 52], 10.00th=[ 55], 20.00th=[ 59], | 30.00th=[ 63], 40.00th=[ 67], 50.00th=[ 69], 60.00th=[ 73], | 70.00th=[ 79], 80.00th=[ 88], 90.00th=[ 99], 95.00th=[ 108], | 99.00th=[ 125], 99.50th=[ 130], 99.90th=[ 133], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 556, max= 1088, per=25.11%, avg=869.87, stdev=145.84, samples=23 iops : min= 139, max= 272, avg=217.35, stdev=36.47, samples=23 lat (msec) : 10=0.04%, 20=0.16%, 50=3.83%, 100=87.54%, 250=8.44% cpu : usr=0.84%, sys=2.55%, ctx=1345, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=23536: Thu May 1 20:33:24 2025 write: IOPS=216, BW=867KiB/s (888kB/s)(10.0MiB/11809msec) slat (usec): min=10, max=2197, avg=92.86, stdev=185.02 clat (msec): min=16, max=144, avg=73.24, stdev=17.78 lat (msec): min=16, max=144, avg=73.33, stdev=17.78 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 52], 10.00th=[ 55], 20.00th=[ 59], | 30.00th=[ 63], 40.00th=[ 66], 50.00th=[ 69], 60.00th=[ 73], | 70.00th=[ 79], 80.00th=[ 88], 90.00th=[ 99], 95.00th=[ 111], | 99.00th=[ 125], 99.50th=[ 136], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 144] bw ( KiB/s): min= 512, max= 1085, per=25.04%, avg=867.22, stdev=153.64, samples=23 iops : min= 128, max= 271, avg=216.74, stdev=38.39, samples=23 lat (msec) : 20=0.04%, 50=3.32%, 100=87.93%, 250=8.71% cpu : usr=1.19%, sys=1.98%, ctx=1110, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=3464KiB/s (3548kB/s), 866KiB/s-872KiB/s (887kB/s-893kB/s), io=40.0MiB (41.9MB), run=11739-11823msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=23555: Thu May 1 20:33:38 2025 read: IOPS=102, BW=410KiB/s (420kB/s)(5048KiB/12298msec) slat (usec): min=13, max=22401, avg=4953.90, stdev=2221.27 clat (msec): min=33, max=131, avg=72.10, stdev=17.80 lat (msec): min=36, max=138, avg=77.05, stdev=18.54 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 47], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 62], 40.00th=[ 66], 50.00th=[ 70], 60.00th=[ 74], | 70.00th=[ 81], 80.00th=[ 87], 90.00th=[ 97], 95.00th=[ 106], | 99.00th=[ 122], 99.50th=[ 124], 99.90th=[ 131], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 280, max= 552, per=25.07%, avg=408.46, stdev=67.69, samples=24 iops : min= 70, max= 138, avg=101.96, stdev=16.91, samples=24 write: IOPS=105, BW=422KiB/s (432kB/s)(5192KiB/12298msec) slat (usec): min=13, max=4365, avg=108.01, stdev=296.99 clat (msec): min=31, max=133, avg=76.01, stdev=16.51 lat (msec): min=31, max=133, avg=76.12, stdev=16.51 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 62], | 30.00th=[ 67], 40.00th=[ 71], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 85], 80.00th=[ 90], 90.00th=[ 99], 95.00th=[ 105], | 99.00th=[ 123], 99.50th=[ 126], 99.90th=[ 132], 99.95th=[ 134], | 99.99th=[ 134] bw ( KiB/s): min= 296, max= 536, per=24.83%, avg=421.12, stdev=59.22, samples=24 iops : min= 74, max= 134, avg=105.12, stdev=14.82, samples=24 lat (msec) : 50=5.90%, 100=85.94%, 250=8.16% cpu : usr=1.20%, sys=4.85%, ctx=2442, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23556: Thu May 1 20:33:38 2025 read: IOPS=101, BW=407KiB/s (417kB/s)(4976KiB/12226msec) slat (usec): min=13, max=14570, avg=5000.85, stdev=2234.05 clat (msec): min=9, max=150, avg=71.86, stdev=18.20 lat (msec): min=9, max=157, avg=76.87, stdev=19.04 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 46], 10.00th=[ 52], 20.00th=[ 57], | 30.00th=[ 62], 40.00th=[ 66], 50.00th=[ 70], 60.00th=[ 75], | 70.00th=[ 81], 80.00th=[ 87], 90.00th=[ 99], 95.00th=[ 104], | 99.00th=[ 121], 99.50th=[ 124], 99.90th=[ 140], 99.95th=[ 150], | 99.99th=[ 150] bw ( KiB/s): min= 280, max= 536, per=24.92%, avg=405.88, stdev=68.28, samples=24 iops : min= 70, max= 134, avg=101.21, stdev=17.11, samples=24 write: IOPS=107, BW=431KiB/s (441kB/s)(5264KiB/12226msec) slat (usec): min=10, max=3324, avg=113.76, stdev=255.06 clat (msec): min=22, max=136, avg=75.56, stdev=18.49 lat (msec): min=22, max=136, avg=75.67, stdev=18.50 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 51], 10.00th=[ 54], 20.00th=[ 60], | 30.00th=[ 65], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 80], | 70.00th=[ 85], 80.00th=[ 91], 90.00th=[ 101], 95.00th=[ 108], | 99.00th=[ 124], 99.50th=[ 127], 99.90th=[ 133], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 328, max= 590, per=25.44%, avg=431.54, stdev=63.52, samples=24 iops : min= 82, max= 147, avg=107.62, stdev=15.90, samples=24 lat (msec) : 10=0.04%, 20=0.08%, 50=6.88%, 100=83.83%, 250=9.18% cpu : usr=1.64%, sys=4.52%, ctx=2419, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23557: Thu May 1 20:33:38 2025 read: IOPS=101, BW=405KiB/s (414kB/s)(4984KiB/12313msec) slat (usec): min=1059, max=19725, avg=5161.76, stdev=2285.23 clat (msec): min=6, max=140, avg=72.70, stdev=17.78 lat (msec): min=10, max=148, avg=77.87, stdev=18.50 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 47], 10.00th=[ 52], 20.00th=[ 58], | 30.00th=[ 63], 40.00th=[ 67], 50.00th=[ 71], 60.00th=[ 77], | 70.00th=[ 82], 80.00th=[ 88], 90.00th=[ 97], 95.00th=[ 105], | 99.00th=[ 122], 99.50th=[ 127], 99.90th=[ 138], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 287, max= 495, per=24.79%, avg=403.83, stdev=60.62, samples=24 iops : min= 71, max= 123, avg=100.71, stdev=15.16, samples=24 write: IOPS=106, BW=427KiB/s (437kB/s)(5256KiB/12313msec) slat (usec): min=11, max=3577, avg=108.63, stdev=249.76 clat (msec): min=21, max=137, avg=74.95, stdev=17.08 lat (msec): min=21, max=137, avg=75.06, stdev=17.09 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 52], 10.00th=[ 55], 20.00th=[ 61], | 30.00th=[ 65], 40.00th=[ 69], 50.00th=[ 73], 60.00th=[ 78], | 70.00th=[ 83], 80.00th=[ 89], 90.00th=[ 99], 95.00th=[ 105], | 99.00th=[ 125], 99.50th=[ 131], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 286, max= 680, per=24.93%, avg=422.83, stdev=96.07, samples=24 iops : min= 71, max= 170, avg=105.46, stdev=24.16, samples=24 lat (msec) : 10=0.04%, 50=6.13%, 100=86.09%, 250=7.73% cpu : usr=0.40%, sys=5.60%, ctx=2402, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=23558: Thu May 1 20:33:38 2025 read: IOPS=102, BW=411KiB/s (421kB/s)(5060KiB/12311msec) slat (usec): min=1232, max=22932, avg=5031.67, stdev=2290.22 clat (msec): min=13, max=131, avg=72.90, stdev=17.84 lat (msec): min=16, max=142, avg=77.94, stdev=18.64 clat percentiles (msec): | 1.00th=[ 41], 5.00th=[ 47], 10.00th=[ 52], 20.00th=[ 58], | 30.00th=[ 63], 40.00th=[ 67], 50.00th=[ 71], 60.00th=[ 77], | 70.00th=[ 82], 80.00th=[ 88], 90.00th=[ 96], 95.00th=[ 105], | 99.00th=[ 122], 99.50th=[ 125], 99.90th=[ 131], 99.95th=[ 132], | 99.99th=[ 132] bw ( KiB/s): min= 288, max= 550, per=25.06%, avg=408.29, stdev=63.60, samples=24 iops : min= 72, max= 137, avg=101.83, stdev=15.77, samples=24 write: IOPS=105, BW=421KiB/s (431kB/s)(5180KiB/12311msec) slat (usec): min=14, max=7431, avg=116.41, stdev=351.65 clat (msec): min=34, max=132, avg=75.01, stdev=16.37 lat (msec): min=34, max=132, avg=75.13, stdev=16.40 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 62], | 30.00th=[ 65], 40.00th=[ 69], 50.00th=[ 73], 60.00th=[ 78], | 70.00th=[ 83], 80.00th=[ 89], 90.00th=[ 99], 95.00th=[ 105], | 99.00th=[ 118], 99.50th=[ 124], 99.90th=[ 131], 99.95th=[ 133], | 99.99th=[ 133] bw ( KiB/s): min= 304, max= 552, per=24.64%, avg=417.88, stdev=71.66, samples=24 iops : min= 76, max= 138, avg=104.25, stdev=17.88, samples=24 lat (msec) : 20=0.08%, 50=5.86%, 100=86.05%, 250=8.01% cpu : usr=0.54%, sys=5.40%, ctx=2431, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1630KiB/s (1669kB/s), 405KiB/s-411KiB/s (414kB/s-421kB/s), io=19.6MiB (20.5MB), run=12226-12313msec WRITE: bw=1697KiB/s (1737kB/s), 421KiB/s-431KiB/s (431kB/s-441kB/s), io=20.4MiB (21.4MB), run=12226-12313msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=23572: Thu May 1 20:33:40 2025 read: IOPS=2, BW=101MiB/s (106MB/s)(40.0MiB/397msec) slat (nsec): min=6326.0k, max=6326.0k, avg=6326037.00, stdev= 0.00 clat (nsec): min=378325k, max=378325k, avg=378324538.00, stdev= 0.00 lat (nsec): min=384676k, max=384676k, avg=384676471.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 380], 10.00th=[ 380], 20.00th=[ 380], | 30.00th=[ 380], 40.00th=[ 380], 50.00th=[ 380], 60.00th=[ 380], | 70.00th=[ 380], 80.00th=[ 380], 90.00th=[ 380], 95.00th=[ 380], | 99.00th=[ 380], 99.50th=[ 380], 99.90th=[ 380], 99.95th=[ 380], | 99.99th=[ 380] lat (msec) : 500=100.00% cpu : usr=0.25%, sys=0.51%, ctx=4, majf=0, minf=30 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=101MiB/s (106MB/s), 101MiB/s-101MiB/s (106MB/s-106MB/s), io=40.0MiB (41.9MB), run=397-397msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout