== sanity test 398c: run fio to test AIO ================= 23:00:58 (1746154858) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.236771 s, 177 MB/s osc.lustre-OST0000-osc-ffff8800b601b000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=22939: Thu May 1 23:01:17 2025 write: IOPS=158, BW=636KiB/s (651kB/s)(10.0MiB/16104msec) slat (usec): min=12, max=4075, avg=154.84, stdev=338.02 clat (msec): min=12, max=191, avg=100.18, stdev=22.13 lat (msec): min=13, max=191, avg=100.34, stdev=22.13 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 70], 10.00th=[ 75], 20.00th=[ 83], | 30.00th=[ 88], 40.00th=[ 93], 50.00th=[ 99], 60.00th=[ 105], | 70.00th=[ 110], 80.00th=[ 116], 90.00th=[ 130], 95.00th=[ 138], | 99.00th=[ 171], 99.50th=[ 178], 99.90th=[ 184], 99.95th=[ 188], | 99.99th=[ 192] bw ( KiB/s): min= 431, max= 856, per=25.01%, avg=633.72, stdev=99.82, samples=32 iops : min= 107, max= 214, avg=158.28, stdev=24.96, samples=32 lat (msec) : 20=0.04%, 50=0.47%, 100=52.89%, 250=46.60% cpu : usr=0.42%, sys=2.94%, ctx=1451, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=22940: Thu May 1 23:01:17 2025 write: IOPS=158, BW=634KiB/s (650kB/s)(10.0MiB/16142msec) slat (usec): min=11, max=4201, avg=119.41, stdev=296.99 clat (msec): min=9, max=191, avg=100.31, stdev=22.24 lat (msec): min=9, max=191, avg=100.43, stdev=22.24 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 69], 10.00th=[ 77], 20.00th=[ 83], | 30.00th=[ 89], 40.00th=[ 94], 50.00th=[ 99], 60.00th=[ 105], | 70.00th=[ 109], 80.00th=[ 117], 90.00th=[ 129], 95.00th=[ 138], | 99.00th=[ 174], 99.50th=[ 180], 99.90th=[ 186], 99.95th=[ 186], | 99.99th=[ 192] bw ( KiB/s): min= 439, max= 816, per=24.89%, avg=630.63, stdev=94.90, samples=32 iops : min= 109, max= 204, avg=157.50, stdev=23.72, samples=32 lat (msec) : 10=0.04%, 50=0.74%, 100=53.59%, 250=45.62% cpu : usr=0.36%, sys=2.32%, ctx=848, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=22941: Thu May 1 23:01:17 2025 write: IOPS=158, BW=634KiB/s (649kB/s)(10.0MiB/16159msec) slat (usec): min=10, max=4566, avg=116.38, stdev=295.73 clat (msec): min=17, max=178, avg=100.35, stdev=22.10 lat (msec): min=18, max=178, avg=100.47, stdev=22.09 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 70], 10.00th=[ 75], 20.00th=[ 83], | 30.00th=[ 89], 40.00th=[ 92], 50.00th=[ 99], 60.00th=[ 106], | 70.00th=[ 109], 80.00th=[ 116], 90.00th=[ 132], 95.00th=[ 140], | 99.00th=[ 169], 99.50th=[ 176], 99.90th=[ 180], 99.95th=[ 180], | 99.99th=[ 180] bw ( KiB/s): min= 440, max= 840, per=24.76%, avg=627.38, stdev=97.11, samples=32 iops : min= 110, max= 210, avg=156.75, stdev=24.27, samples=32 lat (msec) : 20=0.04%, 50=0.47%, 100=51.80%, 250=47.70% cpu : usr=0.42%, sys=2.23%, ctx=912, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=22942: Thu May 1 23:01:17 2025 write: IOPS=158, BW=634KiB/s (650kB/s)(10.0MiB/16143msec) slat (usec): min=11, max=4290, avg=126.47, stdev=311.50 clat (msec): min=25, max=188, avg=100.37, stdev=21.93 lat (msec): min=25, max=188, avg=100.50, stdev=21.93 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 70], 10.00th=[ 77], 20.00th=[ 82], | 30.00th=[ 88], 40.00th=[ 94], 50.00th=[ 99], 60.00th=[ 106], | 70.00th=[ 111], 80.00th=[ 117], 90.00th=[ 128], 95.00th=[ 138], | 99.00th=[ 176], 99.50th=[ 188], 99.90th=[ 188], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min= 416, max= 856, per=24.84%, avg=629.37, stdev=100.91, samples=32 iops : min= 104, max= 214, avg=157.25, stdev=25.23, samples=32 lat (msec) : 50=0.70%, 100=52.07%, 250=47.23% cpu : usr=0.27%, sys=2.55%, ctx=930, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=2535KiB/s (2596kB/s), 634KiB/s-636KiB/s (649kB/s-651kB/s), io=40.0MiB (41.9MB), run=16104-16159msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=22965: Thu May 1 23:01:35 2025 read: IOPS=79, BW=318KiB/s (325kB/s)(5048KiB/15885msec) slat (usec): min=17, max=30694, avg=6568.17, stdev=2985.86 clat (msec): min=4, max=271, avg=93.78, stdev=27.20 lat (msec): min=4, max=280, avg=100.35, stdev=28.23 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 66], 20.00th=[ 73], | 30.00th=[ 79], 40.00th=[ 84], 50.00th=[ 89], 60.00th=[ 95], | 70.00th=[ 103], 80.00th=[ 112], 90.00th=[ 127], 95.00th=[ 142], | 99.00th=[ 184], 99.50th=[ 207], 99.90th=[ 264], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 168, max= 423, per=25.33%, avg=317.65, stdev=64.71, samples=31 iops : min= 42, max= 105, avg=79.06, stdev=16.19, samples=31 write: IOPS=81, BW=327KiB/s (335kB/s)(5192KiB/15885msec) slat (usec): min=13, max=6722, avg=186.71, stdev=491.21 clat (msec): min=27, max=258, avg=97.72, stdev=27.62 lat (msec): min=27, max=258, avg=97.91, stdev=27.67 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 67], 10.00th=[ 71], 20.00th=[ 78], | 30.00th=[ 83], 40.00th=[ 88], 50.00th=[ 93], 60.00th=[ 100], | 70.00th=[ 106], 80.00th=[ 114], 90.00th=[ 130], 95.00th=[ 144], | 99.00th=[ 213], 99.50th=[ 222], 99.90th=[ 247], 99.95th=[ 259], | 99.99th=[ 259] bw ( KiB/s): min= 191, max= 471, per=25.13%, avg=328.00, stdev=62.09, samples=31 iops : min= 47, max= 117, avg=81.65, stdev=15.51, samples=31 lat (msec) : 10=0.04%, 20=0.04%, 50=0.43%, 100=64.26%, 250=35.12% lat (msec) : 500=0.12% cpu : usr=0.59%, sys=5.50%, ctx=2275, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22966: Thu May 1 23:01:35 2025 read: IOPS=77, BW=311KiB/s (318kB/s)(4976KiB/15999msec) slat (usec): min=1892, max=25792, avg=6648.04, stdev=2963.14 clat (msec): min=19, max=235, avg=94.22, stdev=25.76 lat (msec): min=24, max=241, avg=100.88, stdev=26.56 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 74], | 30.00th=[ 80], 40.00th=[ 86], 50.00th=[ 90], 60.00th=[ 96], | 70.00th=[ 104], 80.00th=[ 112], 90.00th=[ 126], 95.00th=[ 138], | 99.00th=[ 190], 99.50th=[ 203], 99.90th=[ 218], 99.95th=[ 236], | 99.99th=[ 236] bw ( KiB/s): min= 184, max= 440, per=24.85%, avg=311.61, stdev=69.01, samples=31 iops : min= 46, max= 110, avg=77.65, stdev=17.34, samples=31 write: IOPS=82, BW=329KiB/s (337kB/s)(5264KiB/15999msec) slat (usec): min=12, max=5003, avg=158.70, stdev=369.44 clat (msec): min=38, max=241, avg=97.99, stdev=27.42 lat (msec): min=38, max=242, avg=98.15, stdev=27.42 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 67], 10.00th=[ 71], 20.00th=[ 78], | 30.00th=[ 84], 40.00th=[ 88], 50.00th=[ 93], 60.00th=[ 99], | 70.00th=[ 105], 80.00th=[ 114], 90.00th=[ 133], 95.00th=[ 150], | 99.00th=[ 203], 99.50th=[ 220], 99.90th=[ 243], 99.95th=[ 243], | 99.99th=[ 243] bw ( KiB/s): min= 191, max= 432, per=25.26%, avg=329.65, stdev=63.89, samples=31 iops : min= 47, max= 108, avg=82.16, stdev=16.12, samples=31 lat (msec) : 20=0.04%, 50=0.35%, 100=63.83%, 250=35.78% cpu : usr=0.90%, sys=4.96%, ctx=2297, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22967: Thu May 1 23:01:35 2025 read: IOPS=77, BW=312KiB/s (319kB/s)(4984KiB/16000msec) slat (usec): min=1662, max=39782, avg=6751.34, stdev=3228.96 clat (msec): min=44, max=227, avg=94.54, stdev=25.25 lat (msec): min=48, max=240, avg=101.30, stdev=26.37 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 74], | 30.00th=[ 81], 40.00th=[ 85], 50.00th=[ 91], 60.00th=[ 96], | 70.00th=[ 104], 80.00th=[ 111], 90.00th=[ 126], 95.00th=[ 140], | 99.00th=[ 188], 99.50th=[ 201], 99.90th=[ 226], 99.95th=[ 228], | 99.99th=[ 228] bw ( KiB/s): min= 175, max= 472, per=24.91%, avg=312.32, stdev=76.10, samples=31 iops : min= 43, max= 118, avg=77.84, stdev=19.09, samples=31 write: IOPS=82, BW=329KiB/s (336kB/s)(5256KiB/16000msec) slat (usec): min=12, max=6264, avg=165.52, stdev=377.50 clat (msec): min=51, max=222, avg=97.18, stdev=24.77 lat (msec): min=51, max=222, avg=97.35, stdev=24.78 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 67], 10.00th=[ 71], 20.00th=[ 78], | 30.00th=[ 83], 40.00th=[ 88], 50.00th=[ 93], 60.00th=[ 100], | 70.00th=[ 106], 80.00th=[ 114], 90.00th=[ 125], 95.00th=[ 140], | 99.00th=[ 201], 99.50th=[ 207], 99.90th=[ 220], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 184, max= 432, per=25.02%, avg=326.52, stdev=59.80, samples=31 iops : min= 46, max= 108, avg=81.39, stdev=15.03, samples=31 lat (msec) : 50=0.16%, 100=63.01%, 250=36.84% cpu : usr=0.91%, sys=4.78%, ctx=2307, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=22968: Thu May 1 23:01:35 2025 read: IOPS=79, BW=317KiB/s (325kB/s)(5060KiB/15954msec) slat (usec): min=1449, max=31070, avg=6685.63, stdev=3203.69 clat (msec): min=47, max=212, avg=94.33, stdev=25.77 lat (msec): min=54, max=227, avg=101.02, stdev=26.76 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 62], 10.00th=[ 66], 20.00th=[ 73], | 30.00th=[ 80], 40.00th=[ 86], 50.00th=[ 91], 60.00th=[ 96], | 70.00th=[ 104], 80.00th=[ 112], 90.00th=[ 125], 95.00th=[ 142], | 99.00th=[ 188], 99.50th=[ 199], 99.90th=[ 213], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min= 208, max= 456, per=25.33%, avg=317.68, stdev=63.03, samples=31 iops : min= 52, max= 114, avg=79.23, stdev=15.77, samples=31 write: IOPS=81, BW=325KiB/s (332kB/s)(5180KiB/15954msec) slat (usec): min=12, max=5845, avg=151.15, stdev=360.12 clat (msec): min=51, max=233, avg=97.26, stdev=25.87 lat (msec): min=51, max=233, avg=97.42, stdev=25.87 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 67], 10.00th=[ 72], 20.00th=[ 77], | 30.00th=[ 83], 40.00th=[ 89], 50.00th=[ 93], 60.00th=[ 99], | 70.00th=[ 105], 80.00th=[ 114], 90.00th=[ 127], 95.00th=[ 144], | 99.00th=[ 209], 99.50th=[ 224], 99.90th=[ 232], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 192, max= 464, per=24.84%, avg=324.13, stdev=63.21, samples=31 iops : min= 48, max= 116, avg=80.84, stdev=15.82, samples=31 lat (msec) : 50=0.16%, 100=64.14%, 250=35.70% cpu : usr=0.71%, sys=5.05%, ctx=2349, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1254KiB/s (1284kB/s), 311KiB/s-318KiB/s (318kB/s-325kB/s), io=19.6MiB (20.5MB), run=15885-16000msec WRITE: bw=1306KiB/s (1337kB/s), 325KiB/s-329KiB/s (332kB/s-337kB/s), io=20.4MiB (21.4MB), run=15885-16000msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=22985: Thu May 1 23:01:38 2025 read: IOPS=2, BW=96.4MiB/s (101MB/s)(40.0MiB/415msec) slat (nsec): min=10683k, max=10683k, avg=10682584.00, stdev= 0.00 clat (nsec): min=387372k, max=387372k, avg=387372113.00, stdev= 0.00 lat (nsec): min=398089k, max=398089k, avg=398088948.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 388], 10.00th=[ 388], 20.00th=[ 388], | 30.00th=[ 388], 40.00th=[ 388], 50.00th=[ 388], 60.00th=[ 388], | 70.00th=[ 388], 80.00th=[ 388], 90.00th=[ 388], 95.00th=[ 388], | 99.00th=[ 388], 99.50th=[ 388], 99.90th=[ 388], 99.95th=[ 388], | 99.99th=[ 388] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.45%, ctx=4, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=96.4MiB/s (101MB/s), 96.4MiB/s-96.4MiB/s (101MB/s-101MB/s), io=40.0MiB (41.9MB), run=415-415msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout