== sanity test 398c: run fio to test AIO ================= 07:59:21 (1744891161) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.125326 s, 335 MB/s osc.lustre-OST0000-osc-ffff8800b67cd000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=10354: Thu Apr 17 07:59:51 2025 write: IOPS=92, BW=369KiB/s (378kB/s)(10.0MiB/27718msec) slat (usec): min=11, max=604, avg=76.96, stdev=46.30 clat (msec): min=78, max=292, avg=172.87, stdev=35.41 lat (msec): min=79, max=292, avg=172.95, stdev=35.42 clat percentiles (msec): | 1.00th=[ 103], 5.00th=[ 117], 10.00th=[ 129], 20.00th=[ 144], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 188], 80.00th=[ 203], 90.00th=[ 222], 95.00th=[ 236], | 99.00th=[ 259], 99.50th=[ 271], 99.90th=[ 284], 99.95th=[ 292], | 99.99th=[ 292] bw ( KiB/s): min= 256, max= 512, per=24.96%, avg=368.67, stdev=59.00, samples=55 iops : min= 64, max= 128, avg=92.13, stdev=14.74, samples=55 lat (msec) : 100=0.86%, 250=97.07%, 500=2.07% cpu : usr=0.14%, sys=0.82%, ctx=2157, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=10355: Thu Apr 17 07:59:51 2025 write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27674msec) slat (usec): min=11, max=518, avg=78.64, stdev=48.07 clat (msec): min=32, max=305, avg=172.68, stdev=36.81 lat (msec): min=32, max=306, avg=172.76, stdev=36.81 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 116], 10.00th=[ 129], 20.00th=[ 142], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 190], 80.00th=[ 203], 90.00th=[ 222], 95.00th=[ 236], | 99.00th=[ 266], 99.50th=[ 284], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 256, max= 512, per=25.05%, avg=369.98, stdev=59.22, samples=55 iops : min= 64, max= 128, avg=92.45, stdev=14.79, samples=55 lat (msec) : 50=0.20%, 100=1.29%, 250=96.48%, 500=2.03% cpu : usr=0.14%, sys=0.81%, ctx=2183, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=10356: Thu Apr 17 07:59:51 2025 write: IOPS=92, BW=370KiB/s (379kB/s)(10.0MiB/27701msec) slat (usec): min=12, max=570, avg=81.01, stdev=46.86 clat (msec): min=52, max=305, avg=172.57, stdev=35.52 lat (msec): min=52, max=305, avg=172.65, stdev=35.52 clat percentiles (msec): | 1.00th=[ 104], 5.00th=[ 117], 10.00th=[ 129], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 180], | 70.00th=[ 190], 80.00th=[ 203], 90.00th=[ 222], 95.00th=[ 236], | 99.00th=[ 259], 99.50th=[ 271], 99.90th=[ 305], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 263, max= 512, per=25.02%, avg=369.55, stdev=57.21, samples=55 iops : min= 65, max= 128, avg=92.35, stdev=14.29, samples=55 lat (msec) : 100=0.90%, 250=97.54%, 500=1.56% cpu : usr=0.21%, sys=0.80%, ctx=2259, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=10357: Thu Apr 17 07:59:51 2025 write: IOPS=92, BW=370KiB/s (378kB/s)(10.0MiB/27710msec) slat (usec): min=14, max=615, avg=82.08, stdev=47.34 clat (msec): min=32, max=302, avg=172.85, stdev=35.45 lat (msec): min=32, max=302, avg=172.94, stdev=35.45 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 117], 10.00th=[ 131], 20.00th=[ 144], | 30.00th=[ 153], 40.00th=[ 163], 50.00th=[ 171], 60.00th=[ 182], | 70.00th=[ 190], 80.00th=[ 203], 90.00th=[ 222], 95.00th=[ 236], | 99.00th=[ 259], 99.50th=[ 279], 99.90th=[ 288], 99.95th=[ 305], | 99.99th=[ 305] bw ( KiB/s): min= 264, max= 504, per=24.98%, avg=369.00, stdev=55.86, samples=55 iops : min= 66, max= 126, avg=92.24, stdev=13.99, samples=55 lat (msec) : 50=0.04%, 100=0.94%, 250=97.19%, 500=1.84% cpu : usr=0.19%, sys=0.83%, ctx=2373, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1478KiB/s (1513kB/s), 369KiB/s-370KiB/s (378kB/s-379kB/s), io=40.0MiB (41.9MB), run=27674-27718msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=10389: Thu Apr 17 08:00:08 2025 read: IOPS=78, BW=313KiB/s (321kB/s)(5048KiB/16109msec) slat (usec): min=677, max=3387, avg=1293.04, stdev=303.18 clat (usec): min=1850, max=196501, avg=67167.60, stdev=15802.54 lat (msec): min=3, max=197, avg=68.46, stdev=15.84 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 47], 10.00th=[ 52], 20.00th=[ 56], | 30.00th=[ 61], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 70], | 70.00th=[ 73], 80.00th=[ 78], 90.00th=[ 84], 95.00th=[ 89], | 99.00th=[ 124], 99.50th=[ 130], 99.90th=[ 188], 99.95th=[ 197], | 99.99th=[ 197] bw ( KiB/s): min= 200, max= 416, per=25.34%, avg=314.16, stdev=50.68, samples=32 iops : min= 50, max= 104, avg=78.47, stdev=12.64, samples=32 write: IOPS=80, BW=322KiB/s (330kB/s)(5192KiB/16109msec) slat (usec): min=14, max=1327, avg=63.52, stdev=47.73 clat (msec): min=64, max=249, avg=131.48, stdev=29.47 lat (msec): min=64, max=249, avg=131.54, stdev=29.47 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 86], 10.00th=[ 94], 20.00th=[ 106], | 30.00th=[ 115], 40.00th=[ 123], 50.00th=[ 129], 60.00th=[ 140], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 171], 95.00th=[ 182], | 99.00th=[ 201], 99.50th=[ 213], 99.90th=[ 249], 99.95th=[ 249], | 99.99th=[ 249] bw ( KiB/s): min= 216, max= 392, per=24.93%, avg=321.91, stdev=38.13, samples=32 iops : min= 54, max= 98, avg=80.41, stdev= 9.58, samples=32 lat (msec) : 2=0.04%, 4=0.04%, 20=0.43%, 50=3.67%, 100=52.58% lat (msec) : 250=43.24% cpu : usr=0.24%, sys=1.86%, ctx=2841, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10390: Thu Apr 17 08:00:08 2025 read: IOPS=76, BW=308KiB/s (315kB/s)(4976KiB/16160msec) slat (usec): min=11, max=4069, avg=1286.83, stdev=315.01 clat (msec): min=2, max=138, avg=66.93, stdev=14.29 lat (msec): min=3, max=138, avg=68.22, stdev=14.31 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 48], 10.00th=[ 53], 20.00th=[ 57], | 30.00th=[ 62], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 70], | 70.00th=[ 73], 80.00th=[ 78], 90.00th=[ 83], 95.00th=[ 88], | 99.00th=[ 113], 99.50th=[ 127], 99.90th=[ 133], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 192, max= 431, per=24.83%, avg=307.91, stdev=67.53, samples=32 iops : min= 48, max= 107, avg=76.91, stdev=16.83, samples=32 write: IOPS=81, BW=326KiB/s (334kB/s)(5264KiB/16160msec) slat (usec): min=12, max=428, avg=64.07, stdev=32.23 clat (msec): min=27, max=235, avg=131.74, stdev=30.06 lat (msec): min=27, max=235, avg=131.80, stdev=30.06 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 87], 10.00th=[ 93], 20.00th=[ 104], | 30.00th=[ 115], 40.00th=[ 125], 50.00th=[ 131], 60.00th=[ 140], | 70.00th=[ 148], 80.00th=[ 159], 90.00th=[ 171], 95.00th=[ 184], | 99.00th=[ 201], 99.50th=[ 213], 99.90th=[ 232], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 272, max= 408, per=25.21%, avg=325.41, stdev=38.55, samples=32 iops : min= 68, max= 102, avg=81.28, stdev= 9.70, samples=32 lat (msec) : 4=0.16%, 10=0.16%, 20=0.35%, 50=3.20%, 100=53.05% lat (msec) : 250=43.09% cpu : usr=0.20%, sys=1.91%, ctx=2935, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10391: Thu Apr 17 08:00:08 2025 read: IOPS=77, BW=308KiB/s (315kB/s)(4984KiB/16178msec) slat (usec): min=754, max=3536, avg=1289.39, stdev=298.00 clat (usec): min=1463, max=143320, avg=67238.24, stdev=13790.51 lat (msec): min=2, max=144, avg=68.53, stdev=13.83 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 48], 10.00th=[ 53], 20.00th=[ 57], | 30.00th=[ 62], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 70], | 70.00th=[ 74], 80.00th=[ 78], 90.00th=[ 84], 95.00th=[ 89], | 99.00th=[ 104], 99.50th=[ 122], 99.90th=[ 124], 99.95th=[ 144], | 99.99th=[ 144] bw ( KiB/s): min= 176, max= 432, per=24.77%, avg=307.16, stdev=64.97, samples=32 iops : min= 44, max= 108, avg=76.72, stdev=16.26, samples=32 write: IOPS=81, BW=325KiB/s (333kB/s)(5256KiB/16178msec) slat (usec): min=13, max=283, avg=64.43, stdev=33.27 clat (msec): min=16, max=236, avg=131.56, stdev=30.74 lat (msec): min=16, max=236, avg=131.62, stdev=30.74 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 86], 10.00th=[ 94], 20.00th=[ 106], | 30.00th=[ 115], 40.00th=[ 124], 50.00th=[ 131], 60.00th=[ 138], | 70.00th=[ 148], 80.00th=[ 159], 90.00th=[ 174], 95.00th=[ 184], | 99.00th=[ 201], 99.50th=[ 209], 99.90th=[ 232], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 216, max= 408, per=24.86%, avg=320.91, stdev=41.84, samples=32 iops : min= 54, max= 102, avg=80.16, stdev=10.43, samples=32 lat (msec) : 2=0.04%, 4=0.08%, 20=0.55%, 50=3.09%, 100=52.58% lat (msec) : 250=43.67% cpu : usr=0.24%, sys=1.87%, ctx=2873, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=10392: Thu Apr 17 08:00:08 2025 read: IOPS=78, BW=313KiB/s (321kB/s)(5060KiB/16160msec) slat (usec): min=719, max=5368, avg=1292.43, stdev=328.44 clat (msec): min=11, max=146, avg=67.71, stdev=13.20 lat (msec): min=12, max=147, avg=69.00, stdev=13.25 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 48], 10.00th=[ 53], 20.00th=[ 58], | 30.00th=[ 62], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 74], 80.00th=[ 78], 90.00th=[ 84], 95.00th=[ 88], | 99.00th=[ 106], 99.50th=[ 117], 99.90th=[ 144], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 232, max= 456, per=25.34%, avg=314.16, stdev=49.44, samples=32 iops : min= 58, max= 114, avg=78.47, stdev=12.32, samples=32 write: IOPS=80, BW=321KiB/s (328kB/s)(5180KiB/16160msec) slat (usec): min=13, max=591, avg=67.53, stdev=42.51 clat (msec): min=62, max=220, avg=131.71, stdev=29.50 lat (msec): min=62, max=220, avg=131.78, stdev=29.50 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 86], 10.00th=[ 94], 20.00th=[ 106], | 30.00th=[ 115], 40.00th=[ 124], 50.00th=[ 131], 60.00th=[ 140], | 70.00th=[ 146], 80.00th=[ 157], 90.00th=[ 171], 95.00th=[ 184], | 99.00th=[ 203], 99.50th=[ 209], 99.90th=[ 220], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 200, max= 416, per=24.72%, avg=319.16, stdev=43.32, samples=32 iops : min= 50, max= 104, avg=79.72, stdev=10.84, samples=32 lat (msec) : 20=0.27%, 50=3.09%, 100=53.32%, 250=43.32% cpu : usr=0.22%, sys=1.84%, ctx=3116, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1240KiB/s (1270kB/s), 308KiB/s-313KiB/s (315kB/s-321kB/s), io=19.6MiB (20.5MB), run=16109-16178msec WRITE: bw=1291KiB/s (1322kB/s), 321KiB/s-326KiB/s (328kB/s-334kB/s), io=20.4MiB (21.4MB), run=16109-16178msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=10409: Thu Apr 17 08:00:09 2025 read: IOPS=9, BW=392MiB/s (411MB/s)(40.0MiB/102msec) slat (nsec): min=5867.4k, max=5867.4k, avg=5867448.00, stdev= 0.00 clat (nsec): min=91662k, max=91662k, avg=91661505.00, stdev= 0.00 lat (nsec): min=97539k, max=97539k, avg=97538878.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[91751], 5.00th=[91751], 10.00th=[91751], 20.00th=[91751], | 30.00th=[91751], 40.00th=[91751], 50.00th=[91751], 60.00th=[91751], | 70.00th=[91751], 80.00th=[91751], 90.00th=[91751], 95.00th=[91751], | 99.00th=[91751], 99.50th=[91751], 99.90th=[91751], 99.95th=[91751], | 99.99th=[91751] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=5.94%, ctx=5, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=392MiB/s (411MB/s), 392MiB/s-392MiB/s (411MB/s-411MB/s), io=40.0MiB (41.9MB), run=102-102msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout