== sanity test 398c: run fio to test AIO ================= 04:10:29 (1733649029) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0797449 s, 526 MB/s osc.lustre-OST0000-osc-ffff8800b5a06000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=2125: Sun Dec 8 04:10:54 2024 write: IOPS=108, BW=435KiB/s (446kB/s)(10.0MiB/23519msec) slat (usec): min=13, max=636, avg=59.89, stdev=37.59 clat (msec): min=11, max=222, avg=146.77, stdev=27.58 lat (msec): min=11, max=222, avg=146.84, stdev=27.58 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 106], 10.00th=[ 116], 20.00th=[ 128], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 148], 60.00th=[ 155], | 70.00th=[ 163], 80.00th=[ 167], 90.00th=[ 180], 95.00th=[ 186], | 99.00th=[ 207], 99.50th=[ 213], 99.90th=[ 222], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 336, max= 720, per=24.82%, avg=432.11, stdev=62.28, samples=47 iops : min= 84, max= 180, avg=107.98, stdev=15.58, samples=47 lat (msec) : 20=0.27%, 50=0.31%, 100=3.79%, 250=95.62% cpu : usr=0.14%, sys=0.74%, ctx=2390, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=2126: Sun Dec 8 04:10:54 2024 write: IOPS=108, BW=436KiB/s (446kB/s)(10.0MiB/23505msec) slat (usec): min=14, max=353, avg=60.44, stdev=36.06 clat (msec): min=15, max=301, avg=146.74, stdev=28.19 lat (msec): min=15, max=301, avg=146.80, stdev=28.19 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 106], 10.00th=[ 116], 20.00th=[ 127], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 148], 60.00th=[ 155], | 70.00th=[ 161], 80.00th=[ 167], 90.00th=[ 178], 95.00th=[ 188], | 99.00th=[ 211], 99.50th=[ 218], 99.90th=[ 268], 99.95th=[ 268], | 99.99th=[ 300] bw ( KiB/s): min= 312, max= 784, per=24.88%, avg=433.11, stdev=69.61, samples=47 iops : min= 78, max= 196, avg=108.21, stdev=17.43, samples=47 lat (msec) : 20=0.35%, 50=0.31%, 100=3.44%, 250=95.74%, 500=0.16% cpu : usr=0.18%, sys=0.70%, ctx=2518, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=2127: Sun Dec 8 04:10:54 2024 write: IOPS=108, BW=435KiB/s (446kB/s)(10.0MiB/23520msec) slat (usec): min=13, max=563, avg=59.39, stdev=37.57 clat (msec): min=38, max=233, avg=146.84, stdev=27.07 lat (msec): min=38, max=233, avg=146.90, stdev=27.07 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 104], 10.00th=[ 116], 20.00th=[ 128], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 150], 60.00th=[ 157], | 70.00th=[ 161], 80.00th=[ 169], 90.00th=[ 180], 95.00th=[ 188], | 99.00th=[ 205], 99.50th=[ 211], 99.90th=[ 226], 99.95th=[ 234], | 99.99th=[ 234] bw ( KiB/s): min= 336, max= 768, per=24.88%, avg=433.17, stdev=65.67, samples=47 iops : min= 84, max= 192, avg=108.28, stdev=16.43, samples=47 lat (msec) : 50=0.62%, 100=3.44%, 250=95.94% cpu : usr=0.10%, sys=0.77%, ctx=2350, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=2128: Sun Dec 8 04:10:54 2024 write: IOPS=109, BW=436KiB/s (446kB/s)(10.0MiB/23486msec) slat (usec): min=13, max=536, avg=61.42, stdev=38.95 clat (msec): min=35, max=320, avg=146.56, stdev=27.81 lat (msec): min=35, max=321, avg=146.62, stdev=27.81 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 106], 10.00th=[ 116], 20.00th=[ 127], | 30.00th=[ 136], 40.00th=[ 142], 50.00th=[ 148], 60.00th=[ 155], | 70.00th=[ 161], 80.00th=[ 167], 90.00th=[ 178], 95.00th=[ 188], | 99.00th=[ 207], 99.50th=[ 213], 99.90th=[ 284], 99.95th=[ 292], | 99.99th=[ 321] bw ( KiB/s): min= 344, max= 776, per=25.06%, avg=436.24, stdev=67.15, samples=46 iops : min= 86, max= 194, avg=108.98, stdev=16.80, samples=46 lat (msec) : 50=0.66%, 100=3.36%, 250=95.78%, 500=0.20% cpu : usr=0.14%, sys=0.75%, ctx=2447, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1741KiB/s (1783kB/s), 435KiB/s-436KiB/s (446kB/s-446kB/s), io=40.0MiB (41.9MB), run=23486-23520msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=2157: Sun Dec 8 04:11:09 2024 read: IOPS=86, BW=347KiB/s (355kB/s)(5048KiB/14546msec) slat (usec): min=675, max=3088, avg=1086.18, stdev=248.81 clat (msec): min=2, max=109, avg=59.67, stdev=10.22 lat (msec): min=3, max=111, avg=60.76, stdev=10.27 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 46], 10.00th=[ 50], 20.00th=[ 52], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 62], | 70.00th=[ 65], 80.00th=[ 67], 90.00th=[ 71], 95.00th=[ 74], | 99.00th=[ 92], 99.50th=[ 100], 99.90th=[ 110], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 224, max= 480, per=25.16%, avg=346.72, stdev=63.05, samples=29 iops : min= 56, max= 120, avg=86.66, stdev=15.80, samples=29 write: IOPS=89, BW=357KiB/s (366kB/s)(5192KiB/14546msec) slat (usec): min=14, max=1164, avg=51.91, stdev=56.37 clat (msec): min=49, max=201, avg=119.73, stdev=24.13 lat (msec): min=49, max=201, avg=119.78, stdev=24.13 clat percentiles (msec): | 1.00th=[ 71], 5.00th=[ 82], 10.00th=[ 88], 20.00th=[ 100], | 30.00th=[ 106], 40.00th=[ 113], 50.00th=[ 120], 60.00th=[ 125], | 70.00th=[ 133], 80.00th=[ 140], 90.00th=[ 150], 95.00th=[ 163], | 99.00th=[ 182], 99.50th=[ 186], 99.90th=[ 201], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min= 288, max= 424, per=24.78%, avg=355.28, stdev=31.49, samples=29 iops : min= 72, max= 106, avg=88.79, stdev= 7.84, samples=29 lat (msec) : 4=0.04%, 10=0.12%, 20=0.12%, 50=6.25%, 100=52.93% lat (msec) : 250=40.55% cpu : usr=0.21%, sys=1.60%, ctx=2907, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2158: Sun Dec 8 04:11:09 2024 read: IOPS=85, BW=343KiB/s (351kB/s)(4976KiB/14506msec) slat (usec): min=24, max=2945, avg=1082.92, stdev=266.96 clat (usec): min=1167, max=161577, avg=59305.53, stdev=12402.49 lat (msec): min=2, max=163, avg=60.39, stdev=12.44 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 43], 10.00th=[ 48], 20.00th=[ 52], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 62], | 70.00th=[ 64], 80.00th=[ 67], 90.00th=[ 71], 95.00th=[ 74], | 99.00th=[ 102], 99.50th=[ 111], 99.90th=[ 161], 99.95th=[ 163], | 99.99th=[ 163] bw ( KiB/s): min= 240, max= 464, per=24.76%, avg=341.17, stdev=52.73, samples=29 iops : min= 60, max= 116, avg=85.24, stdev=13.22, samples=29 write: IOPS=90, BW=363KiB/s (372kB/s)(5264KiB/14506msec) slat (usec): min=15, max=246, avg=50.28, stdev=24.13 clat (msec): min=24, max=219, avg=119.07, stdev=25.31 lat (msec): min=24, max=219, avg=119.12, stdev=25.31 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 80], 10.00th=[ 87], 20.00th=[ 100], | 30.00th=[ 106], 40.00th=[ 113], 50.00th=[ 118], 60.00th=[ 125], | 70.00th=[ 132], 80.00th=[ 140], 90.00th=[ 153], 95.00th=[ 163], | 99.00th=[ 182], 99.50th=[ 190], 99.90th=[ 215], 99.95th=[ 220], | 99.99th=[ 220] bw ( KiB/s): min= 320, max= 464, per=25.16%, avg=360.76, stdev=30.89, samples=29 iops : min= 80, max= 116, avg=90.14, stdev= 7.75, samples=29 lat (msec) : 2=0.04%, 4=0.16%, 10=0.08%, 20=0.16%, 50=7.54% lat (msec) : 100=51.48%, 250=40.55% cpu : usr=0.22%, sys=1.52%, ctx=2865, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2159: Sun Dec 8 04:11:09 2024 read: IOPS=85, BW=342KiB/s (351kB/s)(4984KiB/14560msec) slat (usec): min=659, max=3329, avg=1094.95, stdev=255.97 clat (usec): min=1868, max=109448, avg=59618.71, stdev=10924.93 lat (msec): min=2, max=111, avg=60.72, stdev=10.97 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 45], 10.00th=[ 50], 20.00th=[ 52], | 30.00th=[ 55], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 72], 95.00th=[ 75], | 99.00th=[ 93], 99.50th=[ 103], 99.90th=[ 108], 99.95th=[ 110], | 99.99th=[ 110] bw ( KiB/s): min= 255, max= 472, per=24.84%, avg=342.24, stdev=60.59, samples=29 iops : min= 63, max= 118, avg=85.48, stdev=15.16, samples=29 write: IOPS=90, BW=361KiB/s (370kB/s)(5256KiB/14560msec) slat (usec): min=14, max=1282, avg=49.87, stdev=40.12 clat (msec): min=33, max=192, avg=119.35, stdev=24.52 lat (msec): min=33, max=192, avg=119.40, stdev=24.51 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 82], 10.00th=[ 88], 20.00th=[ 99], | 30.00th=[ 107], 40.00th=[ 113], 50.00th=[ 120], 60.00th=[ 125], | 70.00th=[ 132], 80.00th=[ 140], 90.00th=[ 153], 95.00th=[ 165], | 99.00th=[ 180], 99.50th=[ 182], 99.90th=[ 192], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min= 288, max= 400, per=24.89%, avg=356.86, stdev=31.16, samples=29 iops : min= 72, max= 100, avg=89.14, stdev= 7.85, samples=29 lat (msec) : 2=0.04%, 10=0.12%, 20=0.27%, 50=6.41%, 100=52.19% lat (msec) : 250=40.98% cpu : usr=0.21%, sys=1.55%, ctx=2874, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=2160: Sun Dec 8 04:11:09 2024 read: IOPS=87, BW=348KiB/s (357kB/s)(5060KiB/14526msec) slat (usec): min=659, max=2571, avg=1099.14, stdev=251.62 clat (msec): min=9, max=141, avg=59.46, stdev=10.80 lat (msec): min=10, max=142, avg=60.57, stdev=10.84 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 44], 10.00th=[ 48], 20.00th=[ 52], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 62], | 70.00th=[ 64], 80.00th=[ 67], 90.00th=[ 72], 95.00th=[ 74], | 99.00th=[ 97], 99.50th=[ 102], 99.90th=[ 120], 99.95th=[ 142], | 99.99th=[ 142] bw ( KiB/s): min= 216, max= 464, per=25.24%, avg=347.83, stdev=65.47, samples=29 iops : min= 54, max= 116, avg=86.93, stdev=16.37, samples=29 write: IOPS=89, BW=357KiB/s (365kB/s)(5180KiB/14526msec) slat (usec): min=15, max=206, avg=48.16, stdev=21.48 clat (msec): min=49, max=221, avg=119.75, stdev=24.77 lat (msec): min=49, max=221, avg=119.80, stdev=24.77 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 82], 10.00th=[ 89], 20.00th=[ 100], | 30.00th=[ 106], 40.00th=[ 112], 50.00th=[ 120], 60.00th=[ 126], | 70.00th=[ 133], 80.00th=[ 140], 90.00th=[ 153], 95.00th=[ 161], | 99.00th=[ 182], 99.50th=[ 184], 99.90th=[ 222], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 280, max= 432, per=24.70%, avg=354.17, stdev=41.06, samples=29 iops : min= 70, max= 108, avg=88.52, stdev=10.26, samples=29 lat (msec) : 10=0.08%, 20=0.12%, 50=7.30%, 100=52.54%, 250=39.96% cpu : usr=0.26%, sys=1.51%, ctx=2909, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1378KiB/s (1411kB/s), 342KiB/s-348KiB/s (351kB/s-357kB/s), io=19.6MiB (20.5MB), run=14506-14560msec WRITE: bw=1435KiB/s (1469kB/s), 357KiB/s-363KiB/s (365kB/s-372kB/s), io=20.4MiB (21.4MB), run=14506-14560msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=2175: Sun Dec 8 04:11:11 2024 read: IOPS=10, BW=435MiB/s (456MB/s)(40.0MiB/92msec) slat (nsec): min=5635.4k, max=5635.4k, avg=5635426.00, stdev= 0.00 clat (nsec): min=80052k, max=80052k, avg=80051802.00, stdev= 0.00 lat (nsec): min=85697k, max=85697k, avg=85696944.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[80217], 5.00th=[80217], 10.00th=[80217], 20.00th=[80217], | 30.00th=[80217], 40.00th=[80217], 50.00th=[80217], 60.00th=[80217], | 70.00th=[80217], 80.00th=[80217], 90.00th=[80217], 95.00th=[80217], | 99.00th=[80217], 99.50th=[80217], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=5.49%, ctx=12, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=435MiB/s (456MB/s), 435MiB/s-435MiB/s (456MB/s-456MB/s), io=40.0MiB (41.9MB), run=92-92msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout