== sanity test 398c: run fio to test AIO ================= 05:57:20 (1739530640) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.144266 s, 291 MB/s osc.lustre-OST0000-osc-ffff8800b6cef000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=31397: Fri Feb 14 05:59:00 2025 write: IOPS=26, BW=104KiB/s (107kB/s)(10.0MiB/98254msec) slat (usec): min=11, max=4728, avg=130.80, stdev=290.61 clat (msec): min=107, max=990, avg=613.07, stdev=133.41 lat (msec): min=107, max=990, avg=613.20, stdev=133.43 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 418], 10.00th=[ 447], 20.00th=[ 498], | 30.00th=[ 535], 40.00th=[ 575], 50.00th=[ 609], 60.00th=[ 651], | 70.00th=[ 693], 80.00th=[ 735], 90.00th=[ 785], 95.00th=[ 818], | 99.00th=[ 911], 99.50th=[ 944], 99.90th=[ 995], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 24, max= 191, per=24.90%, avg=103.60, stdev=28.41, samples=196 iops : min= 6, max= 47, avg=25.69, stdev= 7.12, samples=196 lat (msec) : 250=0.94%, 500=19.41%, 750=63.28%, 1000=16.37% cpu : usr=0.11%, sys=0.49%, ctx=1424, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31398: Fri Feb 14 05:59:00 2025 write: IOPS=26, BW=104KiB/s (107kB/s)(10.0MiB/98250msec) slat (usec): min=11, max=3536, avg=136.32, stdev=276.83 clat (msec): min=106, max=1014, avg=613.03, stdev=135.44 lat (msec): min=106, max=1014, avg=613.17, stdev=135.44 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 426], 10.00th=[ 451], 20.00th=[ 493], | 30.00th=[ 527], 40.00th=[ 567], 50.00th=[ 609], 60.00th=[ 651], | 70.00th=[ 701], 80.00th=[ 735], 90.00th=[ 785], 95.00th=[ 827], | 99.00th=[ 902], 99.50th=[ 936], 99.90th=[ 1011], 99.95th=[ 1011], | 99.99th=[ 1011] bw ( KiB/s): min= 39, max= 207, per=24.89%, avg=103.53, stdev=28.33, samples=196 iops : min= 9, max= 51, avg=25.63, stdev= 7.12, samples=196 lat (msec) : 250=1.02%, 500=20.66%, 750=61.60%, 1000=16.60% cpu : usr=0.08%, sys=0.54%, ctx=1470, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31399: Fri Feb 14 05:59:00 2025 write: IOPS=26, BW=104KiB/s (107kB/s)(10.0MiB/98250msec) slat (usec): min=12, max=4069, avg=132.67, stdev=272.34 clat (msec): min=208, max=1042, avg=612.87, stdev=131.55 lat (msec): min=208, max=1042, avg=613.00, stdev=131.54 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 422], 10.00th=[ 443], 20.00th=[ 493], | 30.00th=[ 531], 40.00th=[ 567], 50.00th=[ 609], 60.00th=[ 642], | 70.00th=[ 693], 80.00th=[ 735], 90.00th=[ 785], 95.00th=[ 818], | 99.00th=[ 919], 99.50th=[ 936], 99.90th=[ 1028], 99.95th=[ 1028], | 99.99th=[ 1045] bw ( KiB/s): min= 40, max= 168, per=24.89%, avg=103.54, stdev=26.24, samples=196 iops : min= 10, max= 42, avg=25.63, stdev= 6.61, samples=196 lat (msec) : 250=0.62%, 500=21.02%, 750=61.60%, 1000=16.60% cpu : usr=0.08%, sys=0.56%, ctx=1474, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=31400: Fri Feb 14 05:59:00 2025 write: IOPS=26, BW=104KiB/s (107kB/s)(10.0MiB/98159msec) slat (usec): min=11, max=3906, avg=134.85, stdev=277.82 clat (msec): min=81, max=991, avg=612.72, stdev=136.58 lat (msec): min=81, max=991, avg=612.86, stdev=136.59 clat percentiles (msec): | 1.00th=[ 249], 5.00th=[ 426], 10.00th=[ 447], 20.00th=[ 493], | 30.00th=[ 531], 40.00th=[ 575], 50.00th=[ 609], 60.00th=[ 651], | 70.00th=[ 693], 80.00th=[ 735], 90.00th=[ 793], 95.00th=[ 827], | 99.00th=[ 911], 99.50th=[ 936], 99.90th=[ 986], 99.95th=[ 995], | 99.99th=[ 995] bw ( KiB/s): min= 23, max= 255, per=24.90%, avg=103.58, stdev=32.20, samples=196 iops : min= 5, max= 63, avg=25.67, stdev= 8.11, samples=196 lat (msec) : 100=0.62%, 250=0.62%, 500=20.70%, 750=61.33%, 1000=16.72% cpu : usr=0.06%, sys=0.54%, ctx=1359, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=417KiB/s (427kB/s), 104KiB/s-104KiB/s (107kB/s-107kB/s), io=40.0MiB (41.9MB), run=98159-98254msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=31489: Fri Feb 14 05:59:53 2025 read: IOPS=24, BW=99.1KiB/s (101kB/s)(5048KiB/50940msec) slat (usec): min=1196, max=22353, avg=4115.97, stdev=2000.22 clat (usec): min=1663, max=388797, avg=204686.26, stdev=55108.91 lat (msec): min=3, max=391, avg=208.81, stdev=55.86 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 131], 10.00th=[ 144], 20.00th=[ 163], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 232], 80.00th=[ 247], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 347], 99.50th=[ 363], 99.90th=[ 376], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 16, max= 199, per=25.37%, avg=99.69, stdev=34.32, samples=101 iops : min= 4, max= 49, avg=24.72, stdev= 8.62, samples=101 write: IOPS=25, BW=102KiB/s (104kB/s)(5192KiB/50940msec) slat (usec): min=15, max=3239, avg=128.70, stdev=225.19 clat (msec): min=138, max=892, avg=422.46, stdev=127.47 lat (msec): min=138, max=892, avg=422.59, stdev=127.48 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 239], 10.00th=[ 271], 20.00th=[ 313], | 30.00th=[ 342], 40.00th=[ 376], 50.00th=[ 405], 60.00th=[ 439], | 70.00th=[ 489], 80.00th=[ 527], 90.00th=[ 609], 95.00th=[ 651], | 99.00th=[ 743], 99.50th=[ 776], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 39, max= 192, per=24.78%, avg=101.36, stdev=31.49, samples=101 iops : min= 9, max= 48, avg=25.14, stdev= 7.90, samples=101 lat (msec) : 2=0.04%, 4=0.04%, 10=0.04%, 50=0.62%, 100=0.47% lat (msec) : 250=42.27%, 500=42.97%, 750=13.16%, 1000=0.39% cpu : usr=0.08%, sys=1.69%, ctx=2830, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31490: Fri Feb 14 05:59:53 2025 read: IOPS=24, BW=97.7KiB/s (100kB/s)(4976KiB/50940msec) slat (usec): min=1303, max=21888, avg=4130.51, stdev=2105.57 clat (msec): min=3, max=696, avg=203.27, stdev=61.87 lat (msec): min=6, max=701, avg=207.41, stdev=62.58 clat percentiles (msec): | 1.00th=[ 38], 5.00th=[ 126], 10.00th=[ 140], 20.00th=[ 161], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 199], 60.00th=[ 213], | 70.00th=[ 228], 80.00th=[ 243], 90.00th=[ 271], 95.00th=[ 305], | 99.00th=[ 363], 99.50th=[ 380], 99.90th=[ 693], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 24, max= 200, per=24.91%, avg=97.89, stdev=36.85, samples=101 iops : min= 6, max= 50, avg=24.23, stdev= 9.25, samples=101 write: IOPS=25, BW=103KiB/s (106kB/s)(5264KiB/50940msec) slat (usec): min=12, max=3426, avg=130.86, stdev=224.39 clat (msec): min=134, max=929, avg=421.50, stdev=127.78 lat (msec): min=135, max=929, avg=421.63, stdev=127.79 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 234], 10.00th=[ 268], 20.00th=[ 309], | 30.00th=[ 342], 40.00th=[ 376], 50.00th=[ 405], 60.00th=[ 447], | 70.00th=[ 489], 80.00th=[ 531], 90.00th=[ 600], 95.00th=[ 642], | 99.00th=[ 709], 99.50th=[ 785], 99.90th=[ 927], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 39, max= 176, per=25.19%, avg=103.04, stdev=29.15, samples=101 iops : min= 9, max= 44, avg=25.51, stdev= 7.35, samples=101 lat (msec) : 4=0.04%, 10=0.12%, 20=0.08%, 50=0.55%, 100=0.43% lat (msec) : 250=43.20%, 500=41.56%, 750=13.59%, 1000=0.43% cpu : usr=0.26%, sys=1.48%, ctx=2798, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31491: Fri Feb 14 05:59:53 2025 read: IOPS=24, BW=97.8KiB/s (100kB/s)(4984KiB/50939msec) slat (usec): min=15, max=15642, avg=4216.25, stdev=2071.99 clat (msec): min=21, max=769, avg=207.62, stdev=57.13 lat (msec): min=25, max=774, avg=211.84, stdev=57.87 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 133], 10.00th=[ 148], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 201], 60.00th=[ 215], | 70.00th=[ 232], 80.00th=[ 249], 90.00th=[ 275], 95.00th=[ 300], | 99.00th=[ 359], 99.50th=[ 376], 99.90th=[ 693], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 24, max= 216, per=25.02%, avg=98.35, stdev=36.31, samples=101 iops : min= 6, max= 54, avg=24.45, stdev= 9.13, samples=101 write: IOPS=25, BW=103KiB/s (106kB/s)(5256KiB/50939msec) slat (usec): min=13, max=4009, avg=130.16, stdev=248.23 clat (msec): min=79, max=931, avg=418.50, stdev=127.28 lat (msec): min=79, max=931, avg=418.63, stdev=127.30 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 224], 10.00th=[ 271], 20.00th=[ 309], | 30.00th=[ 342], 40.00th=[ 372], 50.00th=[ 409], 60.00th=[ 443], | 70.00th=[ 481], 80.00th=[ 523], 90.00th=[ 592], 95.00th=[ 642], | 99.00th=[ 751], 99.50th=[ 776], 99.90th=[ 844], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 47, max= 216, per=25.15%, avg=102.85, stdev=32.74, samples=101 iops : min= 11, max= 54, avg=25.57, stdev= 8.23, samples=101 lat (msec) : 50=0.35%, 100=0.66%, 250=41.99%, 500=44.14%, 750=12.30% lat (msec) : 1000=0.55% cpu : usr=0.13%, sys=1.67%, ctx=2801, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=31492: Fri Feb 14 05:59:53 2025 read: IOPS=24, BW=99.3KiB/s (102kB/s)(5060KiB/50970msec) slat (usec): min=1241, max=18161, avg=4133.06, stdev=2055.64 clat (msec): min=4, max=525, avg=208.54, stdev=55.42 lat (msec): min=7, max=530, avg=212.68, stdev=56.12 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 133], 10.00th=[ 146], 20.00th=[ 165], | 30.00th=[ 182], 40.00th=[ 192], 50.00th=[ 207], 60.00th=[ 218], | 70.00th=[ 234], 80.00th=[ 251], 90.00th=[ 279], 95.00th=[ 309], | 99.00th=[ 359], 99.50th=[ 368], 99.90th=[ 451], 99.95th=[ 527], | 99.99th=[ 527] bw ( KiB/s): min= 24, max= 192, per=25.36%, avg=99.66, stdev=32.71, samples=101 iops : min= 6, max= 48, avg=24.69, stdev= 8.17, samples=101 write: IOPS=25, BW=102KiB/s (104kB/s)(5180KiB/50970msec) slat (usec): min=13, max=3329, avg=127.02, stdev=223.32 clat (msec): min=142, max=763, avg=420.17, stdev=123.97 lat (msec): min=142, max=763, avg=420.30, stdev=123.97 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 228], 10.00th=[ 266], 20.00th=[ 313], | 30.00th=[ 347], 40.00th=[ 380], 50.00th=[ 409], 60.00th=[ 443], | 70.00th=[ 485], 80.00th=[ 527], 90.00th=[ 592], 95.00th=[ 642], | 99.00th=[ 718], 99.50th=[ 735], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 32, max= 207, per=24.66%, avg=100.85, stdev=32.29, samples=101 iops : min= 8, max= 51, avg=24.99, stdev= 8.08, samples=101 lat (msec) : 10=0.04%, 20=0.04%, 50=0.43%, 100=0.27%, 250=42.73% lat (msec) : 500=43.36%, 750=13.05%, 1000=0.08% cpu : usr=0.40%, sys=1.36%, ctx=2817, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=394KiB/s (403kB/s), 97.7KiB/s-99.3KiB/s (100kB/s-102kB/s), io=19.6MiB (20.5MB), run=50939-50970msec WRITE: bw=410KiB/s (420kB/s), 102KiB/s-103KiB/s (104kB/s-106kB/s), io=20.4MiB (21.4MB), run=50939-50970msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=31536: Fri Feb 14 05:59:54 2025 read: IOPS=5, BW=231MiB/s (242MB/s)(40.0MiB/173msec) slat (nsec): min=7960.1k, max=7960.1k, avg=7960131.00, stdev= 0.00 clat (nsec): min=155780k, max=155780k, avg=155779703.00, stdev= 0.00 lat (nsec): min=163768k, max=163768k, avg=163768298.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 157], 10.00th=[ 157], 20.00th=[ 157], | 30.00th=[ 157], 40.00th=[ 157], 50.00th=[ 157], 60.00th=[ 157], | 70.00th=[ 157], 80.00th=[ 157], 90.00th=[ 157], 95.00th=[ 157], | 99.00th=[ 157], 99.50th=[ 157], 99.90th=[ 157], 99.95th=[ 157], | 99.99th=[ 157] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=4.65%, ctx=9, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=231MiB/s (242MB/s), 231MiB/s-231MiB/s (242MB/s-242MB/s), io=40.0MiB (41.9MB), run=173-173msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout