== sanity test 398c: run fio to test AIO ================= 19:34:01 (1736987641) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.103206 s, 406 MB/s osc.lustre-OST0000-osc-ffff8800aad98000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=26789: Wed Jan 15 19:34:31 2025 write: IOPS=88, BW=356KiB/s (364kB/s)(10.0MiB/28800msec) slat (usec): min=12, max=336, avg=75.56, stdev=43.58 clat (msec): min=64, max=312, avg=179.82, stdev=36.77 lat (msec): min=64, max=312, avg=179.90, stdev=36.78 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 117], 10.00th=[ 138], 20.00th=[ 153], | 30.00th=[ 163], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 190], | 70.00th=[ 201], 80.00th=[ 209], 90.00th=[ 224], 95.00th=[ 236], | 99.00th=[ 268], 99.50th=[ 271], 99.90th=[ 305], 99.95th=[ 313], | 99.99th=[ 313] bw ( KiB/s): min= 264, max= 640, per=24.97%, avg=355.02, stdev=59.51, samples=57 iops : min= 66, max= 160, avg=88.70, stdev=14.89, samples=57 lat (msec) : 100=3.55%, 250=94.02%, 500=2.42% cpu : usr=0.18%, sys=0.76%, ctx=1950, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=26790: Wed Jan 15 19:34:31 2025 write: IOPS=89, BW=356KiB/s (365kB/s)(10.0MiB/28751msec) slat (usec): min=11, max=400, avg=76.82, stdev=43.71 clat (msec): min=20, max=363, avg=179.52, stdev=38.90 lat (msec): min=20, max=363, avg=179.60, stdev=38.90 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 112], 10.00th=[ 136], 20.00th=[ 153], | 30.00th=[ 163], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 190], | 70.00th=[ 199], 80.00th=[ 209], 90.00th=[ 226], 95.00th=[ 236], | 99.00th=[ 275], 99.50th=[ 305], 99.90th=[ 355], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 272, max= 720, per=25.01%, avg=355.70, stdev=67.77, samples=57 iops : min= 68, max= 180, avg=88.86, stdev=16.97, samples=57 lat (msec) : 50=0.04%, 100=4.61%, 250=92.62%, 500=2.73% cpu : usr=0.18%, sys=0.76%, ctx=2087, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=26791: Wed Jan 15 19:34:31 2025 write: IOPS=88, BW=356KiB/s (364kB/s)(10.0MiB/28790msec) slat (usec): min=12, max=340, avg=78.57, stdev=44.69 clat (msec): min=64, max=325, avg=179.77, stdev=36.83 lat (msec): min=64, max=325, avg=179.85, stdev=36.84 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 117], 10.00th=[ 138], 20.00th=[ 153], | 30.00th=[ 163], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 190], | 70.00th=[ 199], 80.00th=[ 209], 90.00th=[ 224], 95.00th=[ 236], | 99.00th=[ 266], 99.50th=[ 284], 99.90th=[ 309], 99.95th=[ 313], | 99.99th=[ 326] bw ( KiB/s): min= 272, max= 640, per=24.96%, avg=354.88, stdev=59.11, samples=57 iops : min= 68, max= 160, avg=88.67, stdev=14.79, samples=57 lat (msec) : 100=3.55%, 250=93.83%, 500=2.62% cpu : usr=0.16%, sys=0.78%, ctx=2189, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=26792: Wed Jan 15 19:34:31 2025 write: IOPS=88, BW=356KiB/s (364kB/s)(10.0MiB/28800msec) slat (usec): min=13, max=523, avg=75.75, stdev=44.18 clat (msec): min=64, max=318, avg=179.83, stdev=36.95 lat (msec): min=64, max=318, avg=179.91, stdev=36.95 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 117], 10.00th=[ 136], 20.00th=[ 153], | 30.00th=[ 163], 40.00th=[ 171], 50.00th=[ 182], 60.00th=[ 188], | 70.00th=[ 199], 80.00th=[ 211], 90.00th=[ 226], 95.00th=[ 236], | 99.00th=[ 268], 99.50th=[ 271], 99.90th=[ 305], 99.95th=[ 313], | 99.99th=[ 317] bw ( KiB/s): min= 279, max= 640, per=24.95%, avg=354.86, stdev=61.44, samples=57 iops : min= 69, max= 160, avg=88.65, stdev=15.38, samples=57 lat (msec) : 100=3.55%, 250=94.22%, 500=2.23% cpu : usr=0.16%, sys=0.76%, ctx=2016, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1422KiB/s (1456kB/s), 356KiB/s-356KiB/s (364kB/s-365kB/s), io=40.0MiB (41.9MB), run=28751-28800msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26825: Wed Jan 15 19:34:50 2025 read: IOPS=72, BW=291KiB/s (298kB/s)(5048KiB/17368msec) slat (usec): min=534, max=3034, avg=958.35, stdev=216.44 clat (msec): min=2, max=120, avg=71.80, stdev=14.00 lat (msec): min=3, max=121, avg=72.76, stdev=14.02 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 52], 10.00th=[ 57], 20.00th=[ 62], | 30.00th=[ 65], 40.00th=[ 69], 50.00th=[ 72], 60.00th=[ 75], | 70.00th=[ 79], 80.00th=[ 83], 90.00th=[ 88], 95.00th=[ 94], | 99.00th=[ 107], 99.50th=[ 111], 99.90th=[ 116], 99.95th=[ 122], | 99.99th=[ 122] bw ( KiB/s): min= 184, max= 392, per=25.12%, avg=289.85, stdev=55.88, samples=34 iops : min= 46, max= 98, avg=72.44, stdev=13.97, samples=34 write: IOPS=74, BW=299KiB/s (306kB/s)(5192KiB/17368msec) slat (usec): min=14, max=1164, avg=71.96, stdev=64.57 clat (msec): min=54, max=232, avg=143.08, stdev=30.41 lat (msec): min=54, max=232, avg=143.16, stdev=30.41 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 97], 10.00th=[ 106], 20.00th=[ 117], | 30.00th=[ 126], 40.00th=[ 134], 50.00th=[ 144], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 186], 95.00th=[ 197], | 99.00th=[ 213], 99.50th=[ 218], 99.90th=[ 228], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 208, max= 368, per=24.78%, avg=297.85, stdev=38.39, samples=34 iops : min= 52, max= 92, avg=74.44, stdev= 9.63, samples=34 lat (msec) : 4=0.04%, 10=0.08%, 20=0.16%, 50=1.88%, 100=49.41% lat (msec) : 250=48.44% cpu : usr=0.35%, sys=1.63%, ctx=3108, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26826: Wed Jan 15 19:34:50 2025 read: IOPS=71, BW=287KiB/s (294kB/s)(4976KiB/17327msec) slat (usec): min=12, max=3026, avg=959.91, stdev=241.31 clat (usec): min=1357, max=116490, avg=71648.02, stdev=14204.63 lat (usec): min=1374, max=117261, avg=72609.94, stdev=14234.89 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 53], 10.00th=[ 57], 20.00th=[ 62], | 30.00th=[ 65], 40.00th=[ 69], 50.00th=[ 72], 60.00th=[ 74], | 70.00th=[ 79], 80.00th=[ 84], 90.00th=[ 89], 95.00th=[ 94], | 99.00th=[ 107], 99.50th=[ 112], 99.90th=[ 115], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 192, max= 416, per=24.83%, avg=286.56, stdev=59.26, samples=34 iops : min= 48, max= 104, avg=71.44, stdev=14.81, samples=34 write: IOPS=75, BW=304KiB/s (311kB/s)(5264KiB/17327msec) slat (usec): min=10, max=568, avg=70.71, stdev=44.95 clat (msec): min=13, max=246, avg=141.82, stdev=32.40 lat (msec): min=13, max=246, avg=141.89, stdev=32.40 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 94], 10.00th=[ 103], 20.00th=[ 115], | 30.00th=[ 125], 40.00th=[ 133], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 184], 95.00th=[ 194], | 99.00th=[ 213], 99.50th=[ 218], 99.90th=[ 247], 99.95th=[ 247], | 99.99th=[ 247] bw ( KiB/s): min= 240, max= 384, per=25.21%, avg=303.03, stdev=38.00, samples=34 iops : min= 60, max= 96, avg=75.56, stdev= 9.42, samples=34 lat (msec) : 2=0.12%, 4=0.04%, 20=0.43%, 50=1.48%, 100=49.57% lat (msec) : 250=48.36% cpu : usr=0.26%, sys=1.68%, ctx=3269, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26827: Wed Jan 15 19:34:50 2025 read: IOPS=71, BW=287KiB/s (294kB/s)(4984KiB/17378msec) slat (usec): min=596, max=2533, avg=956.81, stdev=207.57 clat (usec): min=1798, max=115407, avg=71776.57, stdev=14422.47 lat (msec): min=3, max=116, avg=72.74, stdev=14.45 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 52], 10.00th=[ 57], 20.00th=[ 61], | 30.00th=[ 64], 40.00th=[ 69], 50.00th=[ 72], 60.00th=[ 75], | 70.00th=[ 80], 80.00th=[ 84], 90.00th=[ 89], 95.00th=[ 96], | 99.00th=[ 106], 99.50th=[ 111], 99.90th=[ 115], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min= 152, max= 383, per=24.89%, avg=287.18, stdev=53.19, samples=34 iops : min= 38, max= 95, avg=71.71, stdev=13.30, samples=34 write: IOPS=75, BW=302KiB/s (310kB/s)(5256KiB/17378msec) slat (usec): min=13, max=324, avg=69.92, stdev=41.27 clat (msec): min=34, max=233, avg=142.42, stdev=30.84 lat (msec): min=34, max=233, avg=142.49, stdev=30.84 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 96], 10.00th=[ 106], 20.00th=[ 114], | 30.00th=[ 126], 40.00th=[ 133], 50.00th=[ 142], 60.00th=[ 150], | 70.00th=[ 161], 80.00th=[ 169], 90.00th=[ 184], 95.00th=[ 194], | 99.00th=[ 213], 99.50th=[ 218], 99.90th=[ 232], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 224, max= 408, per=24.95%, avg=299.88, stdev=38.44, samples=34 iops : min= 56, max= 102, avg=74.88, stdev= 9.60, samples=34 lat (msec) : 2=0.04%, 20=0.35%, 50=1.72%, 100=49.06%, 250=48.83% cpu : usr=0.29%, sys=1.69%, ctx=3046, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26828: Wed Jan 15 19:34:50 2025 read: IOPS=73, BW=293KiB/s (300kB/s)(5060KiB/17297msec) slat (usec): min=575, max=3847, avg=968.66, stdev=241.26 clat (msec): min=9, max=158, avg=71.47, stdev=14.11 lat (msec): min=10, max=159, avg=72.44, stdev=14.12 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 53], 10.00th=[ 57], 20.00th=[ 61], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 72], 60.00th=[ 74], | 70.00th=[ 79], 80.00th=[ 83], 90.00th=[ 88], 95.00th=[ 93], | 99.00th=[ 108], 99.50th=[ 111], 99.90th=[ 159], 99.95th=[ 159], | 99.99th=[ 159] bw ( KiB/s): min= 184, max= 424, per=25.48%, avg=294.06, stdev=56.39, samples=34 iops : min= 46, max= 106, avg=73.47, stdev=14.09, samples=34 write: IOPS=74, BW=299KiB/s (307kB/s)(5180KiB/17297msec) slat (usec): min=12, max=394, avg=70.96, stdev=41.56 clat (msec): min=56, max=286, avg=142.66, stdev=32.84 lat (msec): min=56, max=286, avg=142.73, stdev=32.84 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 96], 10.00th=[ 105], 20.00th=[ 114], | 30.00th=[ 123], 40.00th=[ 133], 50.00th=[ 140], 60.00th=[ 150], | 70.00th=[ 159], 80.00th=[ 171], 90.00th=[ 186], 95.00th=[ 201], | 99.00th=[ 226], 99.50th=[ 259], 99.90th=[ 279], 99.95th=[ 288], | 99.99th=[ 288] bw ( KiB/s): min= 232, max= 376, per=24.89%, avg=299.24, stdev=37.50, samples=34 iops : min= 58, max= 94, avg=74.76, stdev= 9.39, samples=34 lat (msec) : 10=0.04%, 20=0.31%, 50=1.64%, 100=50.23%, 250=47.50% lat (msec) : 500=0.27% cpu : usr=0.30%, sys=1.68%, ctx=3180, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1155KiB/s (1183kB/s), 287KiB/s-293KiB/s (294kB/s-300kB/s), io=19.6MiB (20.5MB), run=17297-17378msec WRITE: bw=1202KiB/s (1231kB/s), 299KiB/s-304KiB/s (306kB/s-311kB/s), io=20.4MiB (21.4MB), run=17297-17378msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=26846: Wed Jan 15 19:34:51 2025 read: IOPS=13, BW=533MiB/s (559MB/s)(40.0MiB/75msec) slat (nsec): min=6605.8k, max=6605.8k, avg=6605753.00, stdev= 0.00 clat (nsec): min=64356k, max=64356k, avg=64356296.00, stdev= 0.00 lat (nsec): min=70974k, max=70974k, avg=70973515.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[64226], 5.00th=[64226], 10.00th=[64226], 20.00th=[64226], | 30.00th=[64226], 40.00th=[64226], 50.00th=[64226], 60.00th=[64226], | 70.00th=[64226], 80.00th=[64226], 90.00th=[64226], 95.00th=[64226], | 99.00th=[64226], 99.50th=[64226], 99.90th=[64226], 99.95th=[64226], | 99.99th=[64226] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=9.46%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=533MiB/s (559MB/s), 533MiB/s-533MiB/s (559MB/s-559MB/s), io=40.0MiB (41.9MB), run=75-75msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck