== sanity test 398c: run fio to test AIO ================= 00:40:17 (1722487217) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.154556 s, 271 MB/s osc.lustre-OST0000-osc-ffff88012c87e800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=27961: Thu Aug 1 00:40:49 2024 write: IOPS=82, BW=330KiB/s (337kB/s)(10.0MiB/31074msec) slat (usec): min=11, max=223, avg=49.61, stdev=25.63 clat (msec): min=48, max=417, avg=194.04, stdev=36.84 lat (msec): min=48, max=417, avg=194.09, stdev=36.84 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 132], 10.00th=[ 153], 20.00th=[ 167], | 30.00th=[ 176], 40.00th=[ 184], 50.00th=[ 192], 60.00th=[ 201], | 70.00th=[ 211], 80.00th=[ 222], 90.00th=[ 243], 95.00th=[ 257], | 99.00th=[ 292], 99.50th=[ 305], 99.90th=[ 326], 99.95th=[ 338], | 99.99th=[ 418] bw ( KiB/s): min= 200, max= 534, per=24.91%, avg=328.29, stdev=58.19, samples=62 iops : min= 50, max= 133, avg=82.02, stdev=14.52, samples=62 lat (msec) : 50=0.04%, 100=0.12%, 250=92.30%, 500=7.54% cpu : usr=0.11%, sys=0.51%, ctx=1761, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27962: Thu Aug 1 00:40:49 2024 write: IOPS=82, BW=330KiB/s (337kB/s)(10.0MiB/31074msec) slat (usec): min=12, max=250, avg=52.47, stdev=28.97 clat (msec): min=77, max=340, avg=194.07, stdev=37.71 lat (msec): min=77, max=340, avg=194.12, stdev=37.71 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 130], 10.00th=[ 153], 20.00th=[ 167], | 30.00th=[ 176], 40.00th=[ 184], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 211], 80.00th=[ 222], 90.00th=[ 243], 95.00th=[ 259], | 99.00th=[ 296], 99.50th=[ 309], 99.90th=[ 326], 99.95th=[ 338], | 99.99th=[ 342] bw ( KiB/s): min= 224, max= 526, per=24.91%, avg=328.27, stdev=52.87, samples=62 iops : min= 56, max= 131, avg=82.00, stdev=13.21, samples=62 lat (msec) : 100=0.43%, 250=91.76%, 500=7.81% cpu : usr=0.12%, sys=0.49%, ctx=2167, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27963: Thu Aug 1 00:40:49 2024 write: IOPS=82, BW=330KiB/s (338kB/s)(10.0MiB/31010msec) slat (usec): min=11, max=223, avg=49.98, stdev=26.85 clat (msec): min=57, max=366, avg=193.67, stdev=38.48 lat (msec): min=57, max=366, avg=193.72, stdev=38.48 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 126], 10.00th=[ 153], 20.00th=[ 167], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 211], 80.00th=[ 222], 90.00th=[ 241], 95.00th=[ 259], | 99.00th=[ 305], 99.50th=[ 321], 99.90th=[ 368], 99.95th=[ 368], | 99.99th=[ 368] bw ( KiB/s): min= 192, max= 512, per=24.91%, avg=328.26, stdev=57.10, samples=62 iops : min= 48, max= 128, avg=81.98, stdev=14.26, samples=62 lat (msec) : 100=0.86%, 250=92.15%, 500=6.99% cpu : usr=0.12%, sys=0.49%, ctx=1873, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=27964: Thu Aug 1 00:40:49 2024 write: IOPS=82, BW=330KiB/s (338kB/s)(10.0MiB/31050msec) slat (usec): min=12, max=278, avg=52.33, stdev=27.71 clat (msec): min=48, max=406, avg=193.89, stdev=38.37 lat (msec): min=48, max=406, avg=193.94, stdev=38.37 clat percentiles (msec): | 1.00th=[ 108], 5.00th=[ 133], 10.00th=[ 150], 20.00th=[ 165], | 30.00th=[ 176], 40.00th=[ 186], 50.00th=[ 192], 60.00th=[ 203], | 70.00th=[ 209], 80.00th=[ 222], 90.00th=[ 241], 95.00th=[ 255], | 99.00th=[ 305], 99.50th=[ 321], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] bw ( KiB/s): min= 200, max= 526, per=24.91%, avg=328.27, stdev=56.76, samples=62 iops : min= 50, max= 131, avg=82.00, stdev=14.18, samples=62 lat (msec) : 50=0.04%, 100=0.51%, 250=92.77%, 500=6.68% cpu : usr=0.11%, sys=0.51%, ctx=2174, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1318KiB/s (1350kB/s), 330KiB/s-330KiB/s (337kB/s-338kB/s), io=40.0MiB (41.9MB), run=31010-31074msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=27999: Thu Aug 1 00:41:10 2024 read: IOPS=64, BW=258KiB/s (264kB/s)(5048KiB/19591msec) slat (usec): min=465, max=2784, avg=1095.50, stdev=303.88 clat (msec): min=10, max=138, avg=79.87, stdev=15.30 lat (msec): min=11, max=140, avg=80.97, stdev=15.34 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 57], 10.00th=[ 63], 20.00th=[ 69], | 30.00th=[ 73], 40.00th=[ 77], 50.00th=[ 81], 60.00th=[ 84], | 70.00th=[ 87], 80.00th=[ 91], 90.00th=[ 100], 95.00th=[ 104], | 99.00th=[ 117], 99.50th=[ 132], 99.90th=[ 138], 99.95th=[ 138], | 99.99th=[ 138] bw ( KiB/s): min= 168, max= 400, per=25.17%, avg=257.72, stdev=50.14, samples=39 iops : min= 42, max= 100, avg=64.33, stdev=12.56, samples=39 write: IOPS=66, BW=265KiB/s (271kB/s)(5192KiB/19591msec) slat (usec): min=13, max=383, avg=69.69, stdev=38.79 clat (msec): min=49, max=270, avg=162.39, stdev=37.94 lat (msec): min=49, max=270, avg=162.46, stdev=37.94 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 101], 10.00th=[ 111], 20.00th=[ 128], | 30.00th=[ 140], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 211], 95.00th=[ 226], | 99.00th=[ 253], 99.50th=[ 255], 99.90th=[ 259], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 216, max= 360, per=24.75%, avg=263.87, stdev=32.58, samples=39 iops : min= 54, max= 90, avg=65.87, stdev= 8.11, samples=39 lat (msec) : 20=0.39%, 50=0.62%, 100=46.60%, 250=51.76%, 500=0.62% cpu : usr=0.26%, sys=1.73%, ctx=3057, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28000: Thu Aug 1 00:41:10 2024 read: IOPS=63, BW=254KiB/s (260kB/s)(4976KiB/19570msec) slat (usec): min=506, max=2686, avg=1100.21, stdev=294.16 clat (usec): min=741, max=186772, avg=79631.61, stdev=15664.85 lat (usec): min=1487, max=187688, avg=80734.62, stdev=15684.00 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 81], 60.00th=[ 84], | 70.00th=[ 87], 80.00th=[ 91], 90.00th=[ 99], 95.00th=[ 103], | 99.00th=[ 125], 99.50th=[ 133], 99.90th=[ 186], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 176, max= 392, per=24.77%, avg=253.62, stdev=48.60, samples=39 iops : min= 44, max= 98, avg=63.31, stdev=12.11, samples=39 write: IOPS=67, BW=269KiB/s (275kB/s)(5264KiB/19570msec) slat (usec): min=12, max=441, avg=71.40, stdev=42.89 clat (msec): min=42, max=264, avg=161.28, stdev=37.03 lat (msec): min=42, max=264, avg=161.35, stdev=37.03 clat percentiles (msec): | 1.00th=[ 88], 5.00th=[ 103], 10.00th=[ 112], 20.00th=[ 127], | 30.00th=[ 140], 40.00th=[ 150], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 184], 80.00th=[ 194], 90.00th=[ 211], 95.00th=[ 222], | 99.00th=[ 241], 99.50th=[ 253], 99.90th=[ 259], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 192, max= 352, per=25.16%, avg=268.18, stdev=34.85, samples=39 iops : min= 48, max= 88, avg=66.95, stdev= 8.70, samples=39 lat (usec) : 750=0.04% lat (msec) : 20=0.27%, 50=0.62%, 100=46.05%, 250=52.66%, 500=0.35% cpu : usr=0.26%, sys=1.75%, ctx=2895, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28001: Thu Aug 1 00:41:10 2024 read: IOPS=63, BW=255KiB/s (261kB/s)(4984KiB/19570msec) slat (usec): min=25, max=3129, avg=1093.45, stdev=315.27 clat (usec): min=1745, max=222638, avg=79156.32, stdev=16289.47 lat (usec): min=1790, max=223481, avg=80252.72, stdev=16334.66 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 57], 10.00th=[ 62], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 80], 60.00th=[ 84], | 70.00th=[ 87], 80.00th=[ 91], 90.00th=[ 97], 95.00th=[ 103], | 99.00th=[ 116], 99.50th=[ 118], 99.90th=[ 188], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 176, max= 352, per=24.91%, avg=255.05, stdev=47.25, samples=39 iops : min= 44, max= 88, avg=63.67, stdev=11.76, samples=39 write: IOPS=67, BW=269KiB/s (275kB/s)(5256KiB/19570msec) slat (usec): min=12, max=871, avg=72.32, stdev=48.57 clat (msec): min=34, max=298, avg=161.93, stdev=38.87 lat (msec): min=34, max=298, avg=162.01, stdev=38.87 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 99], 10.00th=[ 113], 20.00th=[ 128], | 30.00th=[ 140], 40.00th=[ 150], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 213], 95.00th=[ 226], | 99.00th=[ 243], 99.50th=[ 264], 99.90th=[ 275], 99.95th=[ 300], | 99.99th=[ 300] bw ( KiB/s): min= 216, max= 360, per=25.02%, avg=266.74, stdev=39.40, samples=39 iops : min= 54, max= 90, avg=66.59, stdev= 9.80, samples=39 lat (msec) : 2=0.08%, 4=0.04%, 10=0.04%, 20=0.43%, 50=0.78% lat (msec) : 100=46.80%, 250=51.41%, 500=0.43% cpu : usr=0.22%, sys=1.72%, ctx=3155, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=28002: Thu Aug 1 00:41:10 2024 read: IOPS=64, BW=258KiB/s (264kB/s)(5060KiB/19590msec) slat (usec): min=471, max=2702, avg=1092.38, stdev=300.17 clat (usec): min=903, max=138820, avg=79746.96, stdev=15238.60 lat (msec): min=2, max=141, avg=80.84, stdev=15.27 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 57], 10.00th=[ 62], 20.00th=[ 68], | 30.00th=[ 72], 40.00th=[ 77], 50.00th=[ 81], 60.00th=[ 84], | 70.00th=[ 88], 80.00th=[ 91], 90.00th=[ 99], 95.00th=[ 104], | 99.00th=[ 117], 99.50th=[ 133], 99.90th=[ 138], 99.95th=[ 140], | 99.99th=[ 140] bw ( KiB/s): min= 168, max= 376, per=25.27%, avg=258.74, stdev=45.65, samples=39 iops : min= 42, max= 94, avg=64.59, stdev=11.42, samples=39 write: IOPS=66, BW=264KiB/s (271kB/s)(5180KiB/19590msec) slat (usec): min=14, max=549, avg=71.16, stdev=42.26 clat (msec): min=72, max=264, avg=162.73, stdev=36.54 lat (msec): min=72, max=264, avg=162.80, stdev=36.54 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 104], 10.00th=[ 114], 20.00th=[ 130], | 30.00th=[ 142], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 174], | 70.00th=[ 184], 80.00th=[ 197], 90.00th=[ 209], 95.00th=[ 224], | 99.00th=[ 249], 99.50th=[ 253], 99.90th=[ 255], 99.95th=[ 266], | 99.99th=[ 266] bw ( KiB/s): min= 184, max= 336, per=24.66%, avg=262.85, stdev=34.96, samples=39 iops : min= 46, max= 84, avg=65.62, stdev= 8.73, samples=39 lat (usec) : 1000=0.04% lat (msec) : 2=0.04%, 20=0.16%, 50=0.74%, 100=46.68%, 250=51.91% lat (msec) : 500=0.43% cpu : usr=0.28%, sys=1.68%, ctx=3158, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1024KiB/s (1049kB/s), 254KiB/s-258KiB/s (260kB/s-264kB/s), io=19.6MiB (20.5MB), run=19570-19591msec WRITE: bw=1066KiB/s (1092kB/s), 264KiB/s-269KiB/s (271kB/s-275kB/s), io=20.4MiB (21.4MB), run=19570-19591msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=28021: Thu Aug 1 00:41:11 2024 read: IOPS=8, BW=336MiB/s (352MB/s)(40.0MiB/119msec) slat (nsec): min=6946.9k, max=6946.9k, avg=6946888.00, stdev= 0.00 clat (nsec): min=106312k, max=106312k, avg=106312256.00, stdev= 0.00 lat (nsec): min=113269k, max=113269k, avg=113269494.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 107], 5.00th=[ 107], 10.00th=[ 107], 20.00th=[ 107], | 30.00th=[ 107], 40.00th=[ 107], 50.00th=[ 107], 60.00th=[ 107], | 70.00th=[ 107], 80.00th=[ 107], 90.00th=[ 107], 95.00th=[ 107], | 99.00th=[ 107], 99.50th=[ 107], 99.90th=[ 107], 99.95th=[ 107], | 99.99th=[ 107] lat (msec) : 250=100.00% cpu : usr=0.00%, sys=5.93%, ctx=7, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=336MiB/s (352MB/s), 336MiB/s-336MiB/s (352MB/s-352MB/s), io=40.0MiB (41.9MB), run=119-119msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck