== sanity test 398c: run fio to test AIO ================= 12:38:01 (1737135481) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.135012 s, 311 MB/s osc.lustre-OST0000-osc-ffff880089135800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=26421: Fri Jan 17 12:39:17 2025 write: IOPS=34, BW=137KiB/s (140kB/s)(10.0MiB/74856msec) slat (usec): min=14, max=2945, avg=110.31, stdev=199.74 clat (msec): min=42, max=886, avg=467.57, stdev=116.36 lat (msec): min=42, max=886, avg=467.68, stdev=116.36 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 292], 10.00th=[ 321], 20.00th=[ 363], | 30.00th=[ 409], 40.00th=[ 443], 50.00th=[ 472], 60.00th=[ 502], | 70.00th=[ 531], 80.00th=[ 567], 90.00th=[ 617], 95.00th=[ 651], | 99.00th=[ 726], 99.50th=[ 760], 99.90th=[ 885], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 40, max= 368, per=24.96%, avg=136.26, stdev=38.41, samples=149 iops : min= 10, max= 92, avg=33.86, stdev= 9.61, samples=149 lat (msec) : 50=0.04%, 100=0.62%, 250=1.76%, 500=57.46%, 750=39.57% lat (msec) : 1000=0.55% cpu : usr=0.26%, sys=0.44%, ctx=1395, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=26422: Fri Jan 17 12:39:17 2025 write: IOPS=34, BW=137KiB/s (140kB/s)(10.0MiB/74982msec) slat (usec): min=13, max=6192, avg=108.62, stdev=233.47 clat (msec): min=76, max=851, avg=468.35, stdev=111.98 lat (msec): min=76, max=851, avg=468.46, stdev=111.98 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 292], 10.00th=[ 321], 20.00th=[ 368], | 30.00th=[ 414], 40.00th=[ 443], 50.00th=[ 472], 60.00th=[ 498], | 70.00th=[ 527], 80.00th=[ 558], 90.00th=[ 609], 95.00th=[ 651], | 99.00th=[ 726], 99.50th=[ 768], 99.90th=[ 810], 99.95th=[ 818], | 99.99th=[ 852] bw ( KiB/s): min= 63, max= 256, per=24.86%, avg=135.72, stdev=35.83, samples=149 iops : min= 15, max= 64, avg=33.72, stdev= 8.97, samples=149 lat (msec) : 100=0.04%, 250=2.03%, 500=58.71%, 750=38.59%, 1000=0.62% cpu : usr=0.15%, sys=0.56%, ctx=1473, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=26423: Fri Jan 17 12:39:17 2025 write: IOPS=34, BW=137KiB/s (140kB/s)(10.0MiB/74983msec) slat (usec): min=12, max=3951, avg=103.19, stdev=184.15 clat (msec): min=107, max=882, avg=468.33, stdev=112.27 lat (msec): min=107, max=882, avg=468.44, stdev=112.28 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 296], 10.00th=[ 321], 20.00th=[ 372], | 30.00th=[ 414], 40.00th=[ 443], 50.00th=[ 472], 60.00th=[ 498], | 70.00th=[ 527], 80.00th=[ 567], 90.00th=[ 609], 95.00th=[ 642], | 99.00th=[ 735], 99.50th=[ 776], 99.90th=[ 852], 99.95th=[ 852], | 99.99th=[ 885] bw ( KiB/s): min= 64, max= 288, per=24.94%, avg=136.18, stdev=35.79, samples=149 iops : min= 16, max= 72, avg=33.90, stdev= 8.96, samples=149 lat (msec) : 250=2.27%, 500=58.55%, 750=38.44%, 1000=0.74% cpu : usr=0.09%, sys=0.62%, ctx=1390, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=26424: Fri Jan 17 12:39:17 2025 write: IOPS=34, BW=137KiB/s (140kB/s)(10.0MiB/75009msec) slat (usec): min=11, max=3283, avg=102.73, stdev=188.53 clat (msec): min=119, max=846, avg=468.49, stdev=112.02 lat (msec): min=119, max=846, avg=468.60, stdev=112.03 clat percentiles (msec): | 1.00th=[ 211], 5.00th=[ 292], 10.00th=[ 326], 20.00th=[ 372], | 30.00th=[ 409], 40.00th=[ 439], 50.00th=[ 468], 60.00th=[ 498], | 70.00th=[ 527], 80.00th=[ 567], 90.00th=[ 609], 95.00th=[ 651], | 99.00th=[ 735], 99.50th=[ 760], 99.90th=[ 810], 99.95th=[ 818], | 99.99th=[ 844] bw ( KiB/s): min= 63, max= 256, per=24.83%, avg=135.55, stdev=36.42, samples=150 iops : min= 15, max= 64, avg=33.75, stdev= 9.11, samples=150 lat (msec) : 250=2.15%, 500=58.75%, 750=38.40%, 1000=0.70% cpu : usr=0.17%, sys=0.53%, ctx=1405, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=546KiB/s (559kB/s), 137KiB/s-137KiB/s (140kB/s-140kB/s), io=40.0MiB (41.9MB), run=74856-75009msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=26494: Fri Jan 17 12:40:00 2025 read: IOPS=30, BW=122KiB/s (125kB/s)(5048KiB/41230msec) slat (usec): min=23, max=14697, avg=2883.30, stdev=1433.19 clat (msec): min=4, max=335, avg=174.51, stdev=48.36 lat (msec): min=4, max=337, avg=177.40, stdev=48.89 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 102], 10.00th=[ 116], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 184], | 70.00th=[ 199], 80.00th=[ 213], 90.00th=[ 239], 95.00th=[ 262], | 99.00th=[ 305], 99.50th=[ 317], 99.90th=[ 334], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 47, max= 248, per=25.30%, avg=122.20, stdev=42.76, samples=82 iops : min= 11, max= 62, avg=30.37, stdev=10.72, samples=82 write: IOPS=31, BW=126KiB/s (129kB/s)(5192KiB/41230msec) slat (usec): min=13, max=3033, avg=107.64, stdev=175.58 clat (msec): min=55, max=696, avg=335.40, stdev=103.80 lat (msec): min=55, max=696, avg=335.51, stdev=103.81 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 188], 10.00th=[ 213], 20.00th=[ 247], | 30.00th=[ 275], 40.00th=[ 300], 50.00th=[ 321], 60.00th=[ 347], | 70.00th=[ 384], 80.00th=[ 426], 90.00th=[ 485], 95.00th=[ 531], | 99.00th=[ 609], 99.50th=[ 642], 99.90th=[ 667], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 56, max= 223, per=24.89%, avg=125.22, stdev=34.35, samples=82 iops : min= 14, max= 55, avg=31.12, stdev= 8.61, samples=82 lat (msec) : 10=0.12%, 50=0.20%, 100=2.27%, 250=54.26%, 500=39.18% lat (msec) : 750=3.98% cpu : usr=0.52%, sys=1.43%, ctx=2776, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26495: Fri Jan 17 12:40:00 2025 read: IOPS=30, BW=120KiB/s (123kB/s)(4976KiB/41459msec) slat (usec): min=774, max=14326, avg=2957.97, stdev=1507.90 clat (usec): min=1295, max=333288, avg=175082.69, stdev=47575.03 lat (msec): min=3, max=334, avg=178.05, stdev=48.04 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 106], 10.00th=[ 117], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 174], 60.00th=[ 186], | 70.00th=[ 201], 80.00th=[ 215], 90.00th=[ 236], 95.00th=[ 257], | 99.00th=[ 288], 99.50th=[ 313], 99.90th=[ 330], 99.95th=[ 334], | 99.99th=[ 334] bw ( KiB/s): min= 16, max= 240, per=24.61%, avg=118.87, stdev=44.30, samples=82 iops : min= 4, max= 60, avg=29.60, stdev=11.11, samples=82 write: IOPS=31, BW=127KiB/s (130kB/s)(5264KiB/41459msec) slat (usec): min=14, max=2630, avg=109.47, stdev=172.74 clat (msec): min=123, max=665, avg=335.10, stdev=103.39 lat (msec): min=123, max=666, avg=335.22, stdev=103.41 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 184], 10.00th=[ 211], 20.00th=[ 251], | 30.00th=[ 271], 40.00th=[ 292], 50.00th=[ 317], 60.00th=[ 347], | 70.00th=[ 380], 80.00th=[ 426], 90.00th=[ 481], 95.00th=[ 527], | 99.00th=[ 609], 99.50th=[ 634], 99.90th=[ 659], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 48, max= 256, per=25.03%, avg=125.89, stdev=35.21, samples=82 iops : min= 12, max= 64, avg=31.35, stdev= 8.79, samples=82 lat (msec) : 2=0.04%, 10=0.12%, 20=0.04%, 50=0.23%, 100=1.41% lat (msec) : 250=54.14%, 500=40.12%, 750=3.91% cpu : usr=0.31%, sys=1.58%, ctx=2649, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26496: Fri Jan 17 12:40:00 2025 read: IOPS=30, BW=120KiB/s (123kB/s)(4984KiB/41485msec) slat (usec): min=859, max=13762, avg=2920.28, stdev=1403.56 clat (msec): min=4, max=320, avg=177.08, stdev=46.61 lat (msec): min=6, max=325, avg=180.01, stdev=47.13 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 107], 10.00th=[ 121], 20.00th=[ 138], | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 201], 80.00th=[ 213], 90.00th=[ 239], 95.00th=[ 259], | 99.00th=[ 300], 99.50th=[ 309], 99.90th=[ 317], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 24, max= 232, per=24.77%, avg=119.66, stdev=40.04, samples=82 iops : min= 6, max= 58, avg=29.74, stdev=10.05, samples=82 write: IOPS=31, BW=127KiB/s (130kB/s)(5256KiB/41485msec) slat (usec): min=12, max=3829, avg=117.89, stdev=233.59 clat (msec): min=125, max=660, avg=333.78, stdev=102.29 lat (msec): min=125, max=660, avg=333.90, stdev=102.30 clat percentiles (msec): | 1.00th=[ 159], 5.00th=[ 190], 10.00th=[ 215], 20.00th=[ 247], | 30.00th=[ 271], 40.00th=[ 292], 50.00th=[ 321], 60.00th=[ 347], | 70.00th=[ 376], 80.00th=[ 422], 90.00th=[ 481], 95.00th=[ 523], | 99.00th=[ 609], 99.50th=[ 634], 99.90th=[ 659], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 48, max= 232, per=24.90%, avg=125.23, stdev=36.96, samples=82 iops : min= 12, max= 58, avg=31.13, stdev= 9.30, samples=82 lat (msec) : 10=0.04%, 50=0.16%, 100=1.64%, 250=54.57%, 500=39.77% lat (msec) : 750=3.83% cpu : usr=0.56%, sys=1.41%, ctx=2816, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=26497: Fri Jan 17 12:40:00 2025 read: IOPS=30, BW=122KiB/s (125kB/s)(5060KiB/41308msec) slat (usec): min=775, max=8858, avg=2860.39, stdev=1317.53 clat (usec): min=1437, max=330298, avg=176678.91, stdev=48449.63 lat (msec): min=2, max=333, avg=179.54, stdev=48.94 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 103], 10.00th=[ 115], 20.00th=[ 136], | 30.00th=[ 150], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 205], 80.00th=[ 218], 90.00th=[ 239], 95.00th=[ 257], | 99.00th=[ 288], 99.50th=[ 309], 99.90th=[ 330], 99.95th=[ 330], | 99.99th=[ 330] bw ( KiB/s): min= 23, max= 247, per=25.34%, avg=122.41, stdev=40.44, samples=82 iops : min= 5, max= 61, avg=30.44, stdev=10.14, samples=82 write: IOPS=31, BW=125KiB/s (128kB/s)(5180KiB/41308msec) slat (usec): min=11, max=2583, avg=113.70, stdev=165.52 clat (msec): min=70, max=665, avg=334.26, stdev=100.67 lat (msec): min=70, max=665, avg=334.37, stdev=100.68 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 190], 10.00th=[ 215], 20.00th=[ 249], | 30.00th=[ 271], 40.00th=[ 300], 50.00th=[ 321], 60.00th=[ 347], | 70.00th=[ 380], 80.00th=[ 418], 90.00th=[ 477], 95.00th=[ 518], | 99.00th=[ 600], 99.50th=[ 651], 99.90th=[ 659], 99.95th=[ 667], | 99.99th=[ 667] bw ( KiB/s): min= 64, max= 224, per=24.76%, avg=124.56, stdev=34.40, samples=82 iops : min= 16, max= 56, avg=30.98, stdev= 8.59, samples=82 lat (msec) : 2=0.04%, 20=0.12%, 50=0.27%, 100=1.84%, 250=54.57% lat (msec) : 500=39.92%, 750=3.24% cpu : usr=0.38%, sys=1.54%, ctx=2764, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=484KiB/s (495kB/s), 120KiB/s-122KiB/s (123kB/s-125kB/s), io=19.6MiB (20.5MB), run=41230-41485msec WRITE: bw=504KiB/s (516kB/s), 125KiB/s-127KiB/s (128kB/s-130kB/s), io=20.4MiB (21.4MB), run=41230-41485msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=26534: Fri Jan 17 12:40:01 2025 read: IOPS=7, BW=296MiB/s (311MB/s)(40.0MiB/135msec) slat (nsec): min=13596k, max=13596k, avg=13595829.00, stdev= 0.00 clat (nsec): min=113687k, max=113687k, avg=113686735.00, stdev= 0.00 lat (nsec): min=127322k, max=127322k, avg=127321973.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 114], 5.00th=[ 114], 10.00th=[ 114], 20.00th=[ 114], | 30.00th=[ 114], 40.00th=[ 114], 50.00th=[ 114], 60.00th=[ 114], | 70.00th=[ 114], 80.00th=[ 114], 90.00th=[ 114], 95.00th=[ 114], | 99.00th=[ 114], 99.50th=[ 114], 99.90th=[ 114], 99.95th=[ 114], | 99.99th=[ 114] lat (msec) : 250=100.00% cpu : usr=0.75%, sys=8.96%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=296MiB/s (311MB/s), 296MiB/s-296MiB/s (311MB/s-311MB/s), io=40.0MiB (41.9MB), run=135-135msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck