== sanity test 398c: run fio to test AIO ================= 23:18:24 (1708834704) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.102589 s, 409 MB/s osc.lustre-OST0000-osc-ffff8800aad3b000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=25351: Sat Feb 24 23:18:58 2024 write: IOPS=80, BW=321KiB/s (329kB/s)(10.0MiB/31879msec) slat (usec): min=12, max=476, avg=73.80, stdev=44.02 clat (msec): min=41, max=389, avg=198.85, stdev=38.65 lat (msec): min=41, max=389, avg=198.93, stdev=38.65 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 144], 10.00th=[ 155], 20.00th=[ 167], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 197], 60.00th=[ 203], | 70.00th=[ 215], 80.00th=[ 228], 90.00th=[ 247], 95.00th=[ 268], | 99.00th=[ 313], 99.50th=[ 351], 99.90th=[ 380], 99.95th=[ 388], | 99.99th=[ 388] bw ( KiB/s): min= 192, max= 440, per=24.99%, avg=320.68, stdev=45.94, samples=63 iops : min= 48, max= 110, avg=80.11, stdev=11.53, samples=63 lat (msec) : 50=0.04%, 100=0.12%, 250=91.45%, 500=8.40% cpu : usr=0.14%, sys=0.72%, ctx=2152, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=25352: Sat Feb 24 23:18:58 2024 write: IOPS=80, BW=321KiB/s (329kB/s)(10.0MiB/31879msec) slat (usec): min=12, max=591, avg=74.55, stdev=45.37 clat (msec): min=41, max=397, avg=198.86, stdev=38.51 lat (msec): min=41, max=397, avg=198.93, stdev=38.51 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 144], 10.00th=[ 155], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 188], 50.00th=[ 197], 60.00th=[ 205], | 70.00th=[ 215], 80.00th=[ 228], 90.00th=[ 247], 95.00th=[ 264], | 99.00th=[ 321], 99.50th=[ 347], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 397] bw ( KiB/s): min= 216, max= 432, per=24.99%, avg=320.68, stdev=45.14, samples=63 iops : min= 54, max= 108, avg=80.11, stdev=11.33, samples=63 lat (msec) : 50=0.04%, 100=0.12%, 250=91.05%, 500=8.79% cpu : usr=0.18%, sys=0.69%, ctx=2345, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=25353: Sat Feb 24 23:18:58 2024 write: IOPS=80, BW=321KiB/s (329kB/s)(10.0MiB/31903msec) slat (usec): min=13, max=452, avg=71.54, stdev=42.64 clat (msec): min=63, max=368, avg=199.03, stdev=37.40 lat (msec): min=63, max=368, avg=199.10, stdev=37.40 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 144], 10.00th=[ 155], 20.00th=[ 167], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 199], 60.00th=[ 205], | 70.00th=[ 215], 80.00th=[ 230], 90.00th=[ 245], 95.00th=[ 264], | 99.00th=[ 305], 99.50th=[ 347], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 168, max= 448, per=24.95%, avg=320.05, stdev=48.12, samples=63 iops : min= 42, max= 112, avg=79.95, stdev=12.05, samples=63 lat (msec) : 100=0.08%, 250=91.84%, 500=8.09% cpu : usr=0.15%, sys=0.71%, ctx=2005, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=25354: Sat Feb 24 23:18:58 2024 write: IOPS=80, BW=322KiB/s (329kB/s)(10.0MiB/31825msec) slat (usec): min=11, max=611, avg=73.10, stdev=44.10 clat (msec): min=21, max=380, avg=198.49, stdev=38.69 lat (msec): min=21, max=381, avg=198.57, stdev=38.69 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 142], 10.00th=[ 153], 20.00th=[ 167], | 30.00th=[ 178], 40.00th=[ 188], 50.00th=[ 197], 60.00th=[ 205], | 70.00th=[ 215], 80.00th=[ 230], 90.00th=[ 245], 95.00th=[ 264], | 99.00th=[ 313], 99.50th=[ 347], 99.90th=[ 368], 99.95th=[ 372], | 99.99th=[ 380] bw ( KiB/s): min= 216, max= 424, per=25.07%, avg=321.67, stdev=44.89, samples=63 iops : min= 54, max= 106, avg=80.33, stdev=11.27, samples=63 lat (msec) : 50=0.08%, 100=0.16%, 250=91.95%, 500=7.81% cpu : usr=0.14%, sys=0.73%, ctx=2072, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1284KiB/s (1315kB/s), 321KiB/s-322KiB/s (329kB/s-329kB/s), io=40.0MiB (41.9MB), run=31825-31903msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25390: Sat Feb 24 23:19:17 2024 read: IOPS=68, BW=273KiB/s (280kB/s)(5048KiB/18472msec) slat (usec): min=688, max=3120, avg=1293.83, stdev=351.20 clat (msec): min=3, max=146, avg=77.12, stdev=16.17 lat (msec): min=4, max=147, avg=78.42, stdev=16.27 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 55], 10.00th=[ 60], 20.00th=[ 67], | 30.00th=[ 70], 40.00th=[ 73], 50.00th=[ 77], 60.00th=[ 81], | 70.00th=[ 85], 80.00th=[ 89], 90.00th=[ 94], 95.00th=[ 103], | 99.00th=[ 126], 99.50th=[ 140], 99.90th=[ 144], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 200, max= 375, per=25.09%, avg=272.53, stdev=46.25, samples=36 iops : min= 50, max= 93, avg=68.03, stdev=11.52, samples=36 write: IOPS=70, BW=281KiB/s (288kB/s)(5192KiB/18472msec) slat (usec): min=12, max=742, avg=60.34, stdev=38.94 clat (msec): min=68, max=264, avg=151.07, stdev=35.86 lat (msec): min=68, max=264, avg=151.13, stdev=35.86 clat percentiles (msec): | 1.00th=[ 84], 5.00th=[ 97], 10.00th=[ 107], 20.00th=[ 118], | 30.00th=[ 128], 40.00th=[ 140], 50.00th=[ 150], 60.00th=[ 161], | 70.00th=[ 171], 80.00th=[ 182], 90.00th=[ 199], 95.00th=[ 215], | 99.00th=[ 239], 99.50th=[ 247], 99.90th=[ 257], 99.95th=[ 264], | 99.99th=[ 264] bw ( KiB/s): min= 200, max= 344, per=24.74%, avg=279.86, stdev=34.47, samples=36 iops : min= 50, max= 86, avg=69.86, stdev= 8.66, samples=36 lat (msec) : 4=0.08%, 10=0.08%, 20=0.39%, 50=0.70%, 100=48.71% lat (msec) : 250=49.88%, 500=0.16% cpu : usr=0.24%, sys=1.63%, ctx=2828, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25391: Sat Feb 24 23:19:17 2024 read: IOPS=67, BW=270KiB/s (276kB/s)(4976KiB/18461msec) slat (usec): min=632, max=3463, avg=1307.70, stdev=352.95 clat (msec): min=2, max=145, avg=76.70, stdev=15.88 lat (msec): min=3, max=147, avg=78.01, stdev=15.98 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 54], 10.00th=[ 61], 20.00th=[ 66], | 30.00th=[ 70], 40.00th=[ 73], 50.00th=[ 77], 60.00th=[ 81], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 95], 95.00th=[ 104], | 99.00th=[ 115], 99.50th=[ 121], 99.90th=[ 144], 99.95th=[ 146], | 99.99th=[ 146] bw ( KiB/s): min= 135, max= 392, per=24.66%, avg=267.86, stdev=55.23, samples=36 iops : min= 33, max= 98, avg=66.86, stdev=13.85, samples=36 write: IOPS=71, BW=285KiB/s (292kB/s)(5264KiB/18461msec) slat (usec): min=12, max=1021, avg=59.09, stdev=42.85 clat (msec): min=25, max=265, avg=150.37, stdev=35.05 lat (msec): min=25, max=265, avg=150.43, stdev=35.05 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 97], 10.00th=[ 107], 20.00th=[ 118], | 30.00th=[ 130], 40.00th=[ 140], 50.00th=[ 148], 60.00th=[ 159], | 70.00th=[ 169], 80.00th=[ 180], 90.00th=[ 197], 95.00th=[ 211], | 99.00th=[ 239], 99.50th=[ 247], 99.90th=[ 264], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 216, max= 392, per=25.06%, avg=283.42, stdev=35.32, samples=36 iops : min= 54, max= 98, avg=70.75, stdev= 8.82, samples=36 lat (msec) : 4=0.08%, 10=0.04%, 20=0.39%, 50=1.21%, 100=47.07% lat (msec) : 250=50.98%, 500=0.23% cpu : usr=0.26%, sys=1.61%, ctx=2804, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25392: Sat Feb 24 23:19:17 2024 read: IOPS=67, BW=271KiB/s (277kB/s)(4984KiB/18414msec) slat (usec): min=38, max=3860, avg=1280.61, stdev=346.02 clat (msec): min=3, max=191, avg=76.75, stdev=15.55 lat (msec): min=3, max=191, avg=78.03, stdev=15.66 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 55], 10.00th=[ 59], 20.00th=[ 66], | 30.00th=[ 69], 40.00th=[ 72], 50.00th=[ 77], 60.00th=[ 81], | 70.00th=[ 84], 80.00th=[ 88], 90.00th=[ 95], 95.00th=[ 103], | 99.00th=[ 114], 99.50th=[ 130], 99.90th=[ 142], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 136, max= 408, per=25.09%, avg=272.50, stdev=62.71, samples=36 iops : min= 34, max= 102, avg=68.00, stdev=15.70, samples=36 write: IOPS=71, BW=285KiB/s (292kB/s)(5256KiB/18414msec) slat (usec): min=12, max=1090, avg=61.67, stdev=56.93 clat (msec): min=28, max=321, avg=149.91, stdev=37.46 lat (msec): min=28, max=321, avg=149.97, stdev=37.47 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 96], 10.00th=[ 105], 20.00th=[ 117], | 30.00th=[ 127], 40.00th=[ 136], 50.00th=[ 148], 60.00th=[ 159], | 70.00th=[ 169], 80.00th=[ 180], 90.00th=[ 199], 95.00th=[ 215], | 99.00th=[ 253], 99.50th=[ 259], 99.90th=[ 309], 99.95th=[ 321], | 99.99th=[ 321] bw ( KiB/s): min= 208, max= 360, per=25.19%, avg=284.94, stdev=37.77, samples=36 iops : min= 52, max= 90, avg=71.11, stdev= 9.48, samples=36 lat (msec) : 4=0.04%, 10=0.08%, 20=0.16%, 50=1.13%, 100=48.09% lat (msec) : 250=49.96%, 500=0.55% cpu : usr=0.22%, sys=1.70%, ctx=2749, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25393: Sat Feb 24 23:19:17 2024 read: IOPS=68, BW=274KiB/s (281kB/s)(5060KiB/18461msec) slat (usec): min=680, max=3214, avg=1301.43, stdev=349.76 clat (usec): min=1237, max=139798, avg=76828.14, stdev=15493.00 lat (msec): min=2, max=142, avg=78.13, stdev=15.61 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 55], 10.00th=[ 60], 20.00th=[ 66], | 30.00th=[ 69], 40.00th=[ 72], 50.00th=[ 77], 60.00th=[ 80], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 96], 95.00th=[ 104], | 99.00th=[ 118], 99.50th=[ 132], 99.90th=[ 140], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 152, max= 368, per=25.30%, avg=274.72, stdev=52.57, samples=36 iops : min= 38, max= 92, avg=68.56, stdev=13.16, samples=36 write: IOPS=70, BW=281KiB/s (287kB/s)(5180KiB/18461msec) slat (usec): min=12, max=962, avg=58.58, stdev=41.36 clat (msec): min=77, max=266, avg=151.34, stdev=36.27 lat (msec): min=77, max=266, avg=151.40, stdev=36.27 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 97], 10.00th=[ 107], 20.00th=[ 117], | 30.00th=[ 129], 40.00th=[ 140], 50.00th=[ 150], 60.00th=[ 159], | 70.00th=[ 171], 80.00th=[ 182], 90.00th=[ 201], 95.00th=[ 213], | 99.00th=[ 247], 99.50th=[ 264], 99.90th=[ 266], 99.95th=[ 268], | 99.99th=[ 268] bw ( KiB/s): min= 200, max= 360, per=24.66%, avg=278.94, stdev=35.85, samples=36 iops : min= 50, max= 90, avg=69.61, stdev= 9.00, samples=36 lat (msec) : 2=0.04%, 10=0.08%, 20=0.23%, 50=0.90%, 100=48.05% lat (msec) : 250=50.27%, 500=0.43% cpu : usr=0.17%, sys=1.70%, ctx=2845, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1086KiB/s (1112kB/s), 270KiB/s-274KiB/s (276kB/s-281kB/s), io=19.6MiB (20.5MB), run=18414-18472msec WRITE: bw=1131KiB/s (1158kB/s), 281KiB/s-285KiB/s (287kB/s-292kB/s), io=20.4MiB (21.4MB), run=18414-18472msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=25411: Sat Feb 24 23:19:19 2024 read: IOPS=10, BW=404MiB/s (424MB/s)(40.0MiB/99msec) slat (nsec): min=5906.6k, max=5906.6k, avg=5906563.00, stdev= 0.00 clat (nsec): min=88901k, max=88901k, avg=88901172.00, stdev= 0.00 lat (nsec): min=94818k, max=94818k, avg=94817780.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[88605], 5.00th=[88605], 10.00th=[88605], 20.00th=[88605], | 30.00th=[88605], 40.00th=[88605], 50.00th=[88605], 60.00th=[88605], | 70.00th=[88605], 80.00th=[88605], 90.00th=[88605], 95.00th=[88605], | 99.00th=[88605], 99.50th=[88605], 99.90th=[88605], 99.95th=[88605], | 99.99th=[88605] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=6.12%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=404MiB/s (424MB/s), 404MiB/s-404MiB/s (424MB/s-424MB/s), io=40.0MiB (41.9MB), run=99-99msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout