== sanity test 398c: run fio to test AIO ================= 19:23:21 (1746141801) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.130287 s, 322 MB/s osc.lustre-OST0000-osc-ffff88007d74b000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=1735: Thu May 1 19:25:04 2025 write: IOPS=25, BW=101KiB/s (104kB/s)(10.0MiB/100973msec) slat (usec): min=10, max=7633, avg=137.09, stdev=348.21 clat (msec): min=193, max=1309, avg=630.38, stdev=150.14 lat (msec): min=193, max=1309, avg=630.52, stdev=150.17 clat percentiles (msec): | 1.00th=[ 224], 5.00th=[ 363], 10.00th=[ 456], 20.00th=[ 523], | 30.00th=[ 558], 40.00th=[ 592], 50.00th=[ 634], 60.00th=[ 667], | 70.00th=[ 709], 80.00th=[ 751], 90.00th=[ 818], 95.00th=[ 877], | 99.00th=[ 969], 99.50th=[ 1062], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1318] bw ( KiB/s): min= 31, max= 256, per=24.92%, avg=100.94, stdev=30.57, samples=201 iops : min= 7, max= 64, avg=24.98, stdev= 7.62, samples=201 lat (msec) : 250=1.25%, 500=15.04%, 750=64.10%, 1000=18.95% cpu : usr=0.10%, sys=0.50%, ctx=1447, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=1736: Thu May 1 19:25:04 2025 write: IOPS=25, BW=101KiB/s (104kB/s)(10.0MiB/100973msec) slat (usec): min=13, max=5187, avg=141.40, stdev=338.01 clat (msec): min=151, max=1308, avg=630.63, stdev=149.49 lat (msec): min=152, max=1309, avg=630.77, stdev=149.51 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 355], 10.00th=[ 460], 20.00th=[ 523], | 30.00th=[ 558], 40.00th=[ 592], 50.00th=[ 634], 60.00th=[ 667], | 70.00th=[ 709], 80.00th=[ 751], 90.00th=[ 818], 95.00th=[ 869], | 99.00th=[ 961], 99.50th=[ 1011], 99.90th=[ 1267], 99.95th=[ 1267], | 99.99th=[ 1318] bw ( KiB/s): min= 32, max= 256, per=24.90%, avg=100.85, stdev=32.23, samples=201 iops : min= 8, max= 64, avg=24.96, stdev= 8.03, samples=201 lat (msec) : 250=1.37%, 500=14.14%, 750=64.30%, 1000=19.65% cpu : usr=0.06%, sys=0.54%, ctx=1449, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=1737: Thu May 1 19:25:04 2025 write: IOPS=25, BW=102KiB/s (104kB/s)(10.0MiB/100816msec) slat (usec): min=12, max=5627, avg=138.16, stdev=334.10 clat (msec): min=37, max=1295, avg=629.50, stdev=157.79 lat (msec): min=37, max=1295, avg=629.64, stdev=157.80 clat percentiles (msec): | 1.00th=[ 201], 5.00th=[ 326], 10.00th=[ 443], 20.00th=[ 523], | 30.00th=[ 558], 40.00th=[ 600], 50.00th=[ 634], 60.00th=[ 667], | 70.00th=[ 701], 80.00th=[ 751], 90.00th=[ 818], 95.00th=[ 885], | 99.00th=[ 1011], 99.50th=[ 1150], 99.90th=[ 1234], 99.95th=[ 1267], | 99.99th=[ 1301] bw ( KiB/s): min= 7, max= 255, per=24.93%, avg=100.96, stdev=34.40, samples=201 iops : min= 1, max= 63, avg=25.00, stdev= 8.57, samples=201 lat (msec) : 50=0.08%, 100=0.16%, 250=1.72%, 500=14.57%, 750=63.91% lat (msec) : 1000=18.52% cpu : usr=0.09%, sys=0.53%, ctx=1488, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=1738: Thu May 1 19:25:04 2025 write: IOPS=25, BW=101KiB/s (104kB/s)(10.0MiB/100944msec) slat (usec): min=10, max=6254, avg=126.42, stdev=303.55 clat (msec): min=206, max=1209, avg=630.17, stdev=151.49 lat (msec): min=206, max=1209, avg=630.30, stdev=151.50 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 351], 10.00th=[ 456], 20.00th=[ 518], | 30.00th=[ 558], 40.00th=[ 592], 50.00th=[ 634], 60.00th=[ 667], | 70.00th=[ 709], 80.00th=[ 751], 90.00th=[ 818], 95.00th=[ 869], | 99.00th=[ 995], 99.50th=[ 1028], 99.90th=[ 1200], 99.95th=[ 1200], | 99.99th=[ 1217] bw ( KiB/s): min= 39, max= 256, per=24.92%, avg=100.94, stdev=34.44, samples=201 iops : min= 9, max= 64, avg=24.98, stdev= 8.61, samples=201 lat (msec) : 250=1.76%, 500=14.49%, 750=63.98%, 1000=18.79% cpu : usr=0.13%, sys=0.45%, ctx=1411, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=406KiB/s (415kB/s), 101KiB/s-102KiB/s (104kB/s-104kB/s), io=40.0MiB (41.9MB), run=100816-100973msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=1829: Thu May 1 19:26:15 2025 read: IOPS=18, BW=72.9KiB/s (74.6kB/s)(5048KiB/69280msec) slat (usec): min=1708, max=23980, avg=6467.67, stdev=2766.49 clat (msec): min=44, max=661, avg=281.10, stdev=59.89 lat (msec): min=50, max=667, avg=287.58, stdev=60.31 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 192], 10.00th=[ 213], 20.00th=[ 236], | 30.00th=[ 253], 40.00th=[ 268], 50.00th=[ 279], 60.00th=[ 292], | 70.00th=[ 305], 80.00th=[ 330], 90.00th=[ 355], 95.00th=[ 376], | 99.00th=[ 426], 99.50th=[ 456], 99.90th=[ 651], 99.95th=[ 659], | 99.99th=[ 659] bw ( KiB/s): min= 23, max= 168, per=25.02%, avg=72.30, stdev=25.69, samples=138 iops : min= 5, max= 42, avg=17.78, stdev= 6.47, samples=138 write: IOPS=18, BW=74.9KiB/s (76.7kB/s)(5192KiB/69280msec) slat (usec): min=14, max=7312, avg=156.69, stdev=377.02 clat (msec): min=250, max=1184, avg=572.27, stdev=154.33 lat (msec): min=250, max=1184, avg=572.42, stdev=154.33 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 351], 10.00th=[ 388], 20.00th=[ 439], | 30.00th=[ 481], 40.00th=[ 514], 50.00th=[ 558], 60.00th=[ 592], | 70.00th=[ 642], 80.00th=[ 701], 90.00th=[ 785], 95.00th=[ 835], | 99.00th=[ 1045], 99.50th=[ 1083], 99.90th=[ 1167], 99.95th=[ 1183], | 99.99th=[ 1183] bw ( KiB/s): min= 24, max= 143, per=24.73%, avg=74.43, stdev=23.12, samples=138 iops : min= 6, max= 35, avg=18.32, stdev= 5.79, samples=138 lat (msec) : 50=0.08%, 100=0.20%, 250=13.71%, 500=53.55%, 750=25.55% lat (msec) : 1000=6.37% cpu : usr=0.18%, sys=1.41%, ctx=2714, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1830: Thu May 1 19:26:15 2025 read: IOPS=17, BW=71.8KiB/s (73.5kB/s)(4976KiB/69280msec) slat (usec): min=1688, max=28514, avg=6437.06, stdev=2818.33 clat (msec): min=6, max=565, avg=278.06, stdev=60.20 lat (msec): min=11, max=569, avg=284.50, stdev=60.52 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 194], 10.00th=[ 209], 20.00th=[ 232], | 30.00th=[ 251], 40.00th=[ 264], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 326], 90.00th=[ 351], 95.00th=[ 376], | 99.00th=[ 414], 99.50th=[ 426], 99.90th=[ 464], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 15, max= 143, per=24.71%, avg=71.41, stdev=26.42, samples=138 iops : min= 3, max= 35, avg=17.60, stdev= 6.64, samples=138 write: IOPS=18, BW=75.0KiB/s (77.8kB/s)(5264KiB/69280msec) slat (usec): min=11, max=3800, avg=163.11, stdev=373.99 clat (msec): min=195, max=1223, avg=572.08, stdev=159.06 lat (msec): min=195, max=1223, avg=572.25, stdev=159.06 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 342], 10.00th=[ 380], 20.00th=[ 439], | 30.00th=[ 477], 40.00th=[ 514], 50.00th=[ 550], 60.00th=[ 600], | 70.00th=[ 651], 80.00th=[ 709], 90.00th=[ 785], 95.00th=[ 835], | 99.00th=[ 978], 99.50th=[ 1150], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 144, per=25.06%, avg=75.42, stdev=23.75, samples=138 iops : min= 1, max= 36, avg=18.60, stdev= 5.98, samples=138 lat (msec) : 10=0.12%, 20=0.04%, 50=0.23%, 100=0.20%, 250=14.57% lat (msec) : 500=51.84%, 750=25.82%, 1000=6.68% cpu : usr=0.44%, sys=1.16%, ctx=2717, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1831: Thu May 1 19:26:15 2025 read: IOPS=18, BW=72.0KiB/s (73.8kB/s)(4984KiB/69179msec) slat (usec): min=1803, max=34176, avg=6463.09, stdev=2985.44 clat (msec): min=10, max=603, avg=278.18, stdev=60.18 lat (msec): min=16, max=607, avg=284.65, stdev=60.64 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 190], 10.00th=[ 209], 20.00th=[ 234], | 30.00th=[ 251], 40.00th=[ 266], 50.00th=[ 275], 60.00th=[ 288], | 70.00th=[ 305], 80.00th=[ 326], 90.00th=[ 351], 95.00th=[ 372], | 99.00th=[ 414], 99.50th=[ 430], 99.90th=[ 558], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 8, max= 175, per=24.99%, avg=72.22, stdev=28.18, samples=137 iops : min= 2, max= 43, avg=17.83, stdev= 7.04, samples=137 write: IOPS=18, BW=75.0KiB/s (77.8kB/s)(5256KiB/69179msec) slat (usec): min=15, max=3690, avg=170.55, stdev=373.30 clat (msec): min=182, max=1155, avg=570.61, stdev=158.17 lat (msec): min=182, max=1155, avg=570.78, stdev=158.17 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 338], 10.00th=[ 376], 20.00th=[ 439], | 30.00th=[ 481], 40.00th=[ 510], 50.00th=[ 550], 60.00th=[ 584], | 70.00th=[ 651], 80.00th=[ 709], 90.00th=[ 785], 95.00th=[ 844], | 99.00th=[ 978], 99.50th=[ 1062], 99.90th=[ 1150], 99.95th=[ 1150], | 99.99th=[ 1150] bw ( KiB/s): min= 15, max= 136, per=24.99%, avg=75.22, stdev=23.57, samples=138 iops : min= 3, max= 34, avg=18.59, stdev= 5.93, samples=138 lat (msec) : 20=0.04%, 50=0.12%, 100=0.39%, 250=13.40%, 500=53.28% lat (msec) : 750=25.08%, 1000=7.23% cpu : usr=0.25%, sys=1.30%, ctx=2688, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=1832: Thu May 1 19:26:15 2025 read: IOPS=18, BW=73.1KiB/s (74.9kB/s)(5060KiB/69220msec) slat (usec): min=18, max=35109, avg=6414.32, stdev=2916.23 clat (msec): min=35, max=596, avg=280.08, stdev=61.63 lat (msec): min=41, max=606, avg=286.50, stdev=62.06 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 194], 10.00th=[ 211], 20.00th=[ 232], | 30.00th=[ 249], 40.00th=[ 264], 50.00th=[ 275], 60.00th=[ 292], | 70.00th=[ 309], 80.00th=[ 326], 90.00th=[ 351], 95.00th=[ 376], | 99.00th=[ 430], 99.50th=[ 510], 99.90th=[ 592], 99.95th=[ 600], | 99.99th=[ 600] bw ( KiB/s): min= 23, max= 160, per=25.48%, avg=73.65, stdev=26.94, samples=136 iops : min= 5, max= 40, avg=18.21, stdev= 6.77, samples=136 write: IOPS=18, BW=74.8KiB/s (76.6kB/s)(5180KiB/69220msec) slat (usec): min=14, max=3877, avg=174.05, stdev=357.99 clat (msec): min=86, max=1208, avg=574.41, stdev=159.77 lat (msec): min=87, max=1208, avg=574.58, stdev=159.77 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 338], 10.00th=[ 384], 20.00th=[ 443], | 30.00th=[ 481], 40.00th=[ 514], 50.00th=[ 550], 60.00th=[ 600], | 70.00th=[ 659], 80.00th=[ 709], 90.00th=[ 793], 95.00th=[ 844], | 99.00th=[ 986], 99.50th=[ 1062], 99.90th=[ 1200], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 23, max= 136, per=24.71%, avg=74.38, stdev=23.02, samples=138 iops : min= 5, max= 34, avg=18.39, stdev= 5.80, samples=138 lat (msec) : 50=0.16%, 100=0.23%, 250=15.12%, 500=51.45%, 750=25.31% lat (msec) : 1000=7.30% cpu : usr=0.00%, sys=1.61%, ctx=2701, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=290KiB/s (297kB/s), 71.8KiB/s-73.1KiB/s (73.5kB/s-74.9kB/s), io=19.6MiB (20.5MB), run=69179-69280msec WRITE: bw=302KiB/s (309kB/s), 74.8KiB/s-75.0KiB/s (76.6kB/s-77.8kB/s), io=20.4MiB (21.4MB), run=69179-69280msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=1892: Thu May 1 19:26:17 2025 read: IOPS=3, BW=138MiB/s (145MB/s)(40.0MiB/290msec) slat (nsec): min=2062.6k, max=2062.6k, avg=2062589.00, stdev= 0.00 clat (nsec): min=270129k, max=270129k, avg=270128840.00, stdev= 0.00 lat (nsec): min=272207k, max=272207k, avg=272207095.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 271], 5.00th=[ 271], 10.00th=[ 271], 20.00th=[ 271], | 30.00th=[ 271], 40.00th=[ 271], 50.00th=[ 271], 60.00th=[ 271], | 70.00th=[ 271], 80.00th=[ 271], 90.00th=[ 271], 95.00th=[ 271], | 99.00th=[ 271], 99.50th=[ 271], 99.90th=[ 271], 99.95th=[ 271], | 99.99th=[ 271] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=0.69%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=138MiB/s (145MB/s), 138MiB/s-138MiB/s (145MB/s-145MB/s), io=40.0MiB (41.9MB), run=290-290msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout