== sanity test 398c: run fio to test AIO ================= 11:36:22 (1736267782) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.0955582 s, 439 MB/s osc.lustre-OST0000-osc-ffff8800871be000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=25492: Tue Jan 7 11:36:50 2025 write: IOPS=96, BW=384KiB/s (393kB/s)(10.0MiB/26658msec) slat (usec): min=11, max=455, avg=69.96, stdev=44.68 clat (msec): min=57, max=270, avg=166.42, stdev=33.25 lat (msec): min=57, max=270, avg=166.49, stdev=33.25 clat percentiles (msec): | 1.00th=[ 81], 5.00th=[ 112], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 192], 90.00th=[ 209], 95.00th=[ 224], | 99.00th=[ 245], 99.50th=[ 249], 99.90th=[ 259], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 272, max= 640, per=24.97%, avg=382.83, stdev=60.42, samples=53 iops : min= 68, max= 160, avg=95.62, stdev=15.11, samples=53 lat (msec) : 100=3.55%, 250=96.05%, 500=0.39% cpu : usr=0.20%, sys=0.78%, ctx=1996, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=25493: Tue Jan 7 11:36:50 2025 write: IOPS=95, BW=383KiB/s (393kB/s)(10.0MiB/26711msec) slat (usec): min=11, max=524, avg=71.39, stdev=47.62 clat (msec): min=57, max=270, avg=166.76, stdev=33.54 lat (msec): min=57, max=270, avg=166.83, stdev=33.55 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 112], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 182], 80.00th=[ 192], 90.00th=[ 209], 95.00th=[ 226], | 99.00th=[ 247], 99.50th=[ 251], 99.90th=[ 268], 99.95th=[ 271], | 99.99th=[ 271] bw ( KiB/s): min= 272, max= 640, per=24.86%, avg=381.17, stdev=59.23, samples=53 iops : min= 68, max= 160, avg=95.21, stdev=14.81, samples=53 lat (msec) : 100=3.91%, 250=95.51%, 500=0.59% cpu : usr=0.14%, sys=0.81%, ctx=2169, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=25494: Tue Jan 7 11:36:50 2025 write: IOPS=96, BW=384KiB/s (393kB/s)(10.0MiB/26648msec) slat (usec): min=10, max=640, avg=73.32, stdev=46.86 clat (msec): min=17, max=283, avg=166.40, stdev=34.99 lat (msec): min=17, max=283, avg=166.47, stdev=34.99 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 112], 10.00th=[ 127], 20.00th=[ 142], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 182], 80.00th=[ 194], 90.00th=[ 211], 95.00th=[ 226], | 99.00th=[ 251], 99.50th=[ 257], 99.90th=[ 275], 99.95th=[ 284], | 99.99th=[ 284] bw ( KiB/s): min= 288, max= 728, per=25.04%, avg=383.92, stdev=69.41, samples=53 iops : min= 72, max= 182, avg=95.92, stdev=17.37, samples=53 lat (msec) : 20=0.04%, 50=0.04%, 100=3.75%, 250=95.16%, 500=1.02% cpu : usr=0.17%, sys=0.82%, ctx=2240, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=25495: Tue Jan 7 11:36:50 2025 write: IOPS=95, BW=383KiB/s (393kB/s)(10.0MiB/26711msec) slat (usec): min=12, max=408, avg=70.32, stdev=43.63 clat (msec): min=43, max=269, avg=166.76, stdev=32.87 lat (msec): min=43, max=269, avg=166.83, stdev=32.87 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 113], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 159], 50.00th=[ 167], 60.00th=[ 176], | 70.00th=[ 184], 80.00th=[ 192], 90.00th=[ 209], 95.00th=[ 224], | 99.00th=[ 245], 99.50th=[ 249], 99.90th=[ 262], 99.95th=[ 268], | 99.99th=[ 271] bw ( KiB/s): min= 272, max= 648, per=24.86%, avg=381.17, stdev=60.05, samples=53 iops : min= 68, max= 162, avg=95.21, stdev=15.02, samples=53 lat (msec) : 50=0.04%, 100=3.36%, 250=96.21%, 500=0.39% cpu : usr=0.13%, sys=0.82%, ctx=2150, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1533KiB/s (1570kB/s), 383KiB/s-384KiB/s (393kB/s-393kB/s), io=40.0MiB (41.9MB), run=26648-26711msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=25526: Tue Jan 7 11:37:05 2025 read: IOPS=89, BW=360KiB/s (369kB/s)(5048KiB/14026msec) slat (usec): min=463, max=2670, avg=819.99, stdev=225.46 clat (msec): min=7, max=127, avg=60.15, stdev=11.58 lat (msec): min=7, max=128, avg=60.97, stdev=11.62 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 44], 10.00th=[ 47], 20.00th=[ 52], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 65], 80.00th=[ 68], 90.00th=[ 75], 95.00th=[ 82], | 99.00th=[ 91], 99.50th=[ 94], 99.90th=[ 117], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min= 215, max= 456, per=25.04%, avg=358.14, stdev=70.15, samples=28 iops : min= 53, max= 114, avg=89.43, stdev=17.56, samples=28 write: IOPS=92, BW=370KiB/s (379kB/s)(5192KiB/14026msec) slat (usec): min=11, max=1359, avg=57.92, stdev=55.91 clat (msec): min=23, max=206, avg=113.38, stdev=26.59 lat (msec): min=23, max=207, avg=113.44, stdev=26.60 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 73], 10.00th=[ 82], 20.00th=[ 91], | 30.00th=[ 99], 40.00th=[ 106], 50.00th=[ 112], 60.00th=[ 118], | 70.00th=[ 127], 80.00th=[ 136], 90.00th=[ 148], 95.00th=[ 159], | 99.00th=[ 190], 99.50th=[ 194], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 271, max= 448, per=24.76%, avg=368.71, stdev=45.81, samples=28 iops : min= 67, max= 112, avg=92.07, stdev=11.55, samples=28 lat (msec) : 10=0.08%, 20=0.08%, 50=8.59%, 100=56.95%, 250=34.30% cpu : usr=0.21%, sys=1.83%, ctx=2913, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25527: Tue Jan 7 11:37:05 2025 read: IOPS=88, BW=355KiB/s (363kB/s)(4976KiB/14026msec) slat (usec): min=434, max=3449, avg=834.08, stdev=257.81 clat (usec): min=975, max=117086, avg=60195.25, stdev=11911.02 lat (usec): min=1732, max=117865, avg=61030.95, stdev=11950.05 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 44], 10.00th=[ 48], 20.00th=[ 52], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 65], 80.00th=[ 69], 90.00th=[ 74], 95.00th=[ 81], | 99.00th=[ 92], 99.50th=[ 100], 99.90th=[ 116], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min= 239, max= 464, per=24.67%, avg=352.71, stdev=56.97, samples=28 iops : min= 59, max= 116, avg=88.07, stdev=14.28, samples=28 write: IOPS=93, BW=375KiB/s (384kB/s)(5264KiB/14026msec) slat (usec): min=11, max=564, avg=57.99, stdev=37.94 clat (msec): min=42, max=206, avg=112.63, stdev=26.05 lat (msec): min=42, max=206, avg=112.69, stdev=26.06 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 73], 10.00th=[ 82], 20.00th=[ 91], | 30.00th=[ 97], 40.00th=[ 106], 50.00th=[ 112], 60.00th=[ 118], | 70.00th=[ 126], 80.00th=[ 136], 90.00th=[ 146], 95.00th=[ 157], | 99.00th=[ 180], 99.50th=[ 190], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 240, max= 488, per=25.13%, avg=374.14, stdev=49.68, samples=28 iops : min= 60, max= 122, avg=93.43, stdev=12.46, samples=28 lat (usec) : 1000=0.04% lat (msec) : 10=0.12%, 20=0.20%, 50=8.20%, 100=57.23%, 250=34.22% cpu : usr=0.29%, sys=1.78%, ctx=2873, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25528: Tue Jan 7 11:37:05 2025 read: IOPS=88, BW=355KiB/s (364kB/s)(4984KiB/14026msec) slat (usec): min=22, max=2875, avg=822.14, stdev=240.03 clat (usec): min=1543, max=103051, avg=60012.48, stdev=11928.69 lat (msec): min=2, max=104, avg=60.84, stdev=11.98 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 51], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 69], 90.00th=[ 75], 95.00th=[ 82], | 99.00th=[ 90], 99.50th=[ 91], 99.90th=[ 99], 99.95th=[ 104], | 99.99th=[ 104] bw ( KiB/s): min= 223, max= 496, per=24.85%, avg=355.32, stdev=69.21, samples=28 iops : min= 55, max= 124, avg=88.75, stdev=17.36, samples=28 write: IOPS=93, BW=375KiB/s (384kB/s)(5256KiB/14026msec) slat (usec): min=11, max=450, avg=58.17, stdev=35.77 clat (msec): min=11, max=206, avg=112.94, stdev=26.83 lat (msec): min=11, max=207, avg=113.00, stdev=26.84 clat percentiles (msec): | 1.00th=[ 59], 5.00th=[ 73], 10.00th=[ 83], 20.00th=[ 91], | 30.00th=[ 99], 40.00th=[ 106], 50.00th=[ 111], 60.00th=[ 117], | 70.00th=[ 126], 80.00th=[ 136], 90.00th=[ 148], 95.00th=[ 159], | 99.00th=[ 188], 99.50th=[ 192], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 272, max= 448, per=24.96%, avg=371.61, stdev=39.47, samples=28 iops : min= 68, max= 112, avg=92.82, stdev= 9.88, samples=28 lat (msec) : 2=0.04%, 10=0.08%, 20=0.23%, 50=9.22%, 100=56.13% lat (msec) : 250=34.30% cpu : usr=0.32%, sys=1.73%, ctx=2997, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=25529: Tue Jan 7 11:37:05 2025 read: IOPS=90, BW=363KiB/s (372kB/s)(5060KiB/13940msec) slat (usec): min=25, max=3096, avg=818.14, stdev=239.80 clat (usec): min=713, max=154221, avg=59764.66, stdev=13280.50 lat (usec): min=767, max=154896, avg=60584.51, stdev=13331.60 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 43], 10.00th=[ 47], 20.00th=[ 51], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 62], | 70.00th=[ 65], 80.00th=[ 69], 90.00th=[ 75], 95.00th=[ 82], | 99.00th=[ 91], 99.50th=[ 105], 99.90th=[ 155], 99.95th=[ 155], | 99.99th=[ 155] bw ( KiB/s): min= 224, max= 440, per=25.40%, avg=363.19, stdev=62.32, samples=27 iops : min= 56, max= 110, avg=90.74, stdev=15.60, samples=27 write: IOPS=92, BW=372KiB/s (381kB/s)(5180KiB/13940msec) slat (usec): min=11, max=408, avg=57.23, stdev=33.46 clat (msec): min=11, max=207, avg=112.86, stdev=27.21 lat (msec): min=12, max=207, avg=112.91, stdev=27.21 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 73], 10.00th=[ 82], 20.00th=[ 92], | 30.00th=[ 100], 40.00th=[ 105], 50.00th=[ 111], 60.00th=[ 117], | 70.00th=[ 126], 80.00th=[ 134], 90.00th=[ 146], 95.00th=[ 159], | 99.00th=[ 188], 99.50th=[ 192], 99.90th=[ 207], 99.95th=[ 207], | 99.99th=[ 207] bw ( KiB/s): min= 272, max= 432, per=24.83%, avg=369.70, stdev=41.81, samples=27 iops : min= 68, max= 108, avg=92.37, stdev=10.48, samples=27 lat (usec) : 750=0.08%, 1000=0.08% lat (msec) : 2=0.12%, 10=0.04%, 20=0.31%, 50=8.95%, 100=56.02% lat (msec) : 250=34.41% cpu : usr=0.28%, sys=1.76%, ctx=3009, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1431KiB/s (1465kB/s), 355KiB/s-363KiB/s (363kB/s-372kB/s), io=19.6MiB (20.5MB), run=13940-14026msec WRITE: bw=1490KiB/s (1525kB/s), 370KiB/s-375KiB/s (379kB/s-384kB/s), io=20.4MiB (21.4MB), run=13940-14026msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=25544: Tue Jan 7 11:37:07 2025 read: IOPS=13, BW=556MiB/s (583MB/s)(40.0MiB/72msec) slat (nsec): min=7510.4k, max=7510.4k, avg=7510401.00, stdev= 0.00 clat (nsec): min=61917k, max=61917k, avg=61916715.00, stdev= 0.00 lat (nsec): min=69443k, max=69443k, avg=69443261.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[62129], 5.00th=[62129], 10.00th=[62129], 20.00th=[62129], | 30.00th=[62129], 40.00th=[62129], 50.00th=[62129], 60.00th=[62129], | 70.00th=[62129], 80.00th=[62129], 90.00th=[62129], 95.00th=[62129], | 99.00th=[62129], 99.50th=[62129], 99.90th=[62129], 99.95th=[62129], | 99.99th=[62129] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=9.86%, ctx=6, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=556MiB/s (583MB/s), 556MiB/s-556MiB/s (583MB/s-583MB/s), io=40.0MiB (41.9MB), run=72-72msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck