== sanity test 398c: run fio to test AIO ================= 23:53:16 (1746157996) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.196298 s, 214 MB/s osc.lustre-OST0000-osc-ffff8800a997f000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=32259: Thu May 1 23:55:01 2025 write: IOPS=24, BW=99.3KiB/s (102kB/s)(10.0MiB/103143msec) slat (usec): min=13, max=5398, avg=137.04, stdev=326.60 clat (msec): min=129, max=1240, avg=643.52, stdev=173.97 lat (msec): min=129, max=1240, avg=643.65, stdev=173.99 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 376], 10.00th=[ 426], 20.00th=[ 489], | 30.00th=[ 542], 40.00th=[ 584], 50.00th=[ 625], 60.00th=[ 676], | 70.00th=[ 735], 80.00th=[ 802], 90.00th=[ 885], 95.00th=[ 944], | 99.00th=[ 1036], 99.50th=[ 1053], 99.90th=[ 1167], 99.95th=[ 1183], | 99.99th=[ 1234] bw ( KiB/s): min= 15, max= 256, per=24.87%, avg=98.47, stdev=34.71, samples=206 iops : min= 3, max= 64, avg=24.35, stdev= 8.72, samples=206 lat (msec) : 250=0.62%, 500=21.84%, 750=50.08%, 1000=25.86% cpu : usr=0.12%, sys=0.46%, ctx=1350, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=32260: Thu May 1 23:55:01 2025 write: IOPS=24, BW=99.2KiB/s (102kB/s)(10.0MiB/103246msec) slat (usec): min=14, max=5350, avg=134.68, stdev=316.23 clat (msec): min=78, max=1201, avg=644.52, stdev=177.10 lat (msec): min=78, max=1201, avg=644.66, stdev=177.11 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 380], 10.00th=[ 426], 20.00th=[ 485], | 30.00th=[ 550], 40.00th=[ 584], 50.00th=[ 634], 60.00th=[ 684], | 70.00th=[ 735], 80.00th=[ 810], 90.00th=[ 885], 95.00th=[ 944], | 99.00th=[ 1045], 99.50th=[ 1099], 99.90th=[ 1183], 99.95th=[ 1200], | 99.99th=[ 1200] bw ( KiB/s): min= 8, max= 239, per=24.99%, avg=98.95, stdev=34.80, samples=205 iops : min= 2, max= 59, avg=24.47, stdev= 8.68, samples=205 lat (msec) : 100=0.59%, 250=0.08%, 500=21.95%, 750=49.45%, 1000=25.82% cpu : usr=0.06%, sys=0.52%, ctx=1404, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=32261: Thu May 1 23:55:01 2025 write: IOPS=24, BW=99.1KiB/s (102kB/s)(10.0MiB/103280msec) slat (usec): min=13, max=5889, avg=146.82, stdev=380.92 clat (msec): min=100, max=1233, avg=644.36, stdev=172.78 lat (msec): min=100, max=1235, avg=644.51, stdev=172.79 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 376], 10.00th=[ 426], 20.00th=[ 485], | 30.00th=[ 542], 40.00th=[ 592], 50.00th=[ 634], 60.00th=[ 676], | 70.00th=[ 743], 80.00th=[ 802], 90.00th=[ 877], 95.00th=[ 936], | 99.00th=[ 1053], 99.50th=[ 1099], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 15, max= 192, per=24.84%, avg=98.35, stdev=33.09, samples=206 iops : min= 3, max= 48, avg=24.37, stdev= 8.31, samples=206 lat (msec) : 250=0.39%, 500=21.41%, 750=50.04%, 1000=26.05% cpu : usr=0.04%, sys=0.57%, ctx=1423, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=32262: Thu May 1 23:55:01 2025 write: IOPS=24, BW=99.1KiB/s (102kB/s)(10.0MiB/103301msec) slat (usec): min=12, max=4646, avg=142.42, stdev=343.20 clat (msec): min=161, max=1240, avg=644.56, stdev=172.42 lat (msec): min=162, max=1241, avg=644.71, stdev=172.44 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 380], 10.00th=[ 422], 20.00th=[ 485], | 30.00th=[ 550], 40.00th=[ 600], 50.00th=[ 634], 60.00th=[ 676], | 70.00th=[ 735], 80.00th=[ 802], 90.00th=[ 885], 95.00th=[ 936], | 99.00th=[ 1053], 99.50th=[ 1116], 99.90th=[ 1217], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 32, max= 255, per=24.79%, avg=98.18, stdev=33.06, samples=206 iops : min= 8, max= 63, avg=24.26, stdev= 8.24, samples=206 lat (msec) : 250=0.12%, 500=22.62%, 750=50.35%, 1000=25.08% cpu : usr=0.09%, sys=0.51%, ctx=1469, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=397KiB/s (406kB/s), 99.1KiB/s-99.3KiB/s (102kB/s-102kB/s), io=40.0MiB (41.9MB), run=103143-103301msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=32354: Thu May 1 23:55:57 2025 read: IOPS=23, BW=93.3KiB/s (95.5kB/s)(5048KiB/54125msec) slat (usec): min=1069, max=18022, avg=4539.80, stdev=2476.36 clat (msec): min=23, max=465, avg=227.86, stdev=81.61 lat (msec): min=26, max=471, avg=232.41, stdev=83.03 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 100], 10.00th=[ 116], 20.00th=[ 144], | 30.00th=[ 176], 40.00th=[ 207], 50.00th=[ 239], 60.00th=[ 257], | 70.00th=[ 275], 80.00th=[ 300], 90.00th=[ 326], 95.00th=[ 363], | 99.00th=[ 414], 99.50th=[ 443], 99.90th=[ 456], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 15, max= 216, per=25.03%, avg=92.59, stdev=40.89, samples=108 iops : min= 3, max= 54, avg=23.03, stdev=10.27, samples=108 write: IOPS=23, BW=95.9KiB/s (98.2kB/s)(5192KiB/54125msec) slat (usec): min=12, max=2863, avg=130.12, stdev=239.70 clat (msec): min=98, max=910, avg=440.30, stdev=171.37 lat (msec): min=98, max=910, avg=440.43, stdev=171.40 clat percentiles (msec): | 1.00th=[ 150], 5.00th=[ 194], 10.00th=[ 222], 20.00th=[ 279], | 30.00th=[ 313], 40.00th=[ 372], 50.00th=[ 435], 60.00th=[ 498], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 659], 95.00th=[ 735], | 99.00th=[ 844], 99.50th=[ 877], 99.90th=[ 911], 99.95th=[ 911], | 99.99th=[ 911] bw ( KiB/s): min= 24, max= 224, per=24.76%, avg=95.33, stdev=41.79, samples=108 iops : min= 6, max= 56, avg=23.69, stdev=10.47, samples=108 lat (msec) : 50=0.08%, 100=2.62%, 250=32.70%, 500=44.57%, 750=17.77% lat (msec) : 1000=2.27% cpu : usr=0.37%, sys=1.26%, ctx=2800, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32355: Thu May 1 23:55:57 2025 read: IOPS=22, BW=91.9KiB/s (94.1kB/s)(4976KiB/54124msec) slat (usec): min=20, max=24493, avg=4624.67, stdev=2606.05 clat (msec): min=3, max=571, avg=226.27, stdev=83.34 lat (msec): min=3, max=579, avg=230.90, stdev=84.68 clat percentiles (msec): | 1.00th=[ 66], 5.00th=[ 101], 10.00th=[ 116], 20.00th=[ 144], | 30.00th=[ 174], 40.00th=[ 205], 50.00th=[ 234], 60.00th=[ 253], | 70.00th=[ 275], 80.00th=[ 296], 90.00th=[ 330], 95.00th=[ 355], | 99.00th=[ 443], 99.50th=[ 456], 99.90th=[ 567], 99.95th=[ 575], | 99.99th=[ 575] bw ( KiB/s): min= 15, max= 224, per=24.62%, avg=91.11, stdev=42.52, samples=108 iops : min= 3, max= 56, avg=22.60, stdev=10.69, samples=108 write: IOPS=24, BW=97.3KiB/s (99.6kB/s)(5264KiB/54124msec) slat (usec): min=13, max=2870, avg=132.54, stdev=237.02 clat (msec): min=46, max=924, avg=439.20, stdev=174.55 lat (msec): min=46, max=925, avg=439.34, stdev=174.58 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 186], 10.00th=[ 222], 20.00th=[ 266], | 30.00th=[ 309], 40.00th=[ 372], 50.00th=[ 430], 60.00th=[ 498], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 667], 95.00th=[ 735], | 99.00th=[ 844], 99.50th=[ 869], 99.90th=[ 902], 99.95th=[ 927], | 99.99th=[ 927] bw ( KiB/s): min= 15, max= 224, per=25.11%, avg=96.69, stdev=44.96, samples=108 iops : min= 3, max= 56, avg=23.99, stdev=11.28, samples=108 lat (msec) : 4=0.08%, 10=0.04%, 50=0.27%, 100=2.54%, 250=33.91% lat (msec) : 500=42.97%, 750=18.16%, 1000=2.03% cpu : usr=0.41%, sys=1.19%, ctx=2935, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32356: Thu May 1 23:55:57 2025 read: IOPS=23, BW=92.3KiB/s (94.5kB/s)(4984KiB/54024msec) slat (usec): min=953, max=21993, avg=4450.75, stdev=2575.74 clat (msec): min=6, max=477, avg=220.56, stdev=82.57 lat (msec): min=10, max=481, avg=225.01, stdev=83.95 clat percentiles (msec): | 1.00th=[ 75], 5.00th=[ 100], 10.00th=[ 111], 20.00th=[ 134], | 30.00th=[ 167], 40.00th=[ 194], 50.00th=[ 226], 60.00th=[ 247], | 70.00th=[ 268], 80.00th=[ 292], 90.00th=[ 326], 95.00th=[ 355], | 99.00th=[ 414], 99.50th=[ 439], 99.90th=[ 477], 99.95th=[ 477], | 99.99th=[ 477] bw ( KiB/s): min= 23, max= 287, per=24.87%, avg=92.04, stdev=48.70, samples=107 iops : min= 5, max= 71, avg=22.73, stdev=12.22, samples=107 write: IOPS=24, BW=97.3KiB/s (99.6kB/s)(5256KiB/54024msec) slat (usec): min=12, max=3150, avg=132.43, stdev=237.57 clat (msec): min=98, max=902, avg=443.43, stdev=167.79 lat (msec): min=99, max=902, avg=443.57, stdev=167.81 clat percentiles (msec): | 1.00th=[ 153], 5.00th=[ 201], 10.00th=[ 236], 20.00th=[ 275], | 30.00th=[ 317], 40.00th=[ 376], 50.00th=[ 439], 60.00th=[ 498], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 667], 95.00th=[ 735], | 99.00th=[ 818], 99.50th=[ 869], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 32, max= 232, per=25.07%, avg=96.52, stdev=38.46, samples=107 iops : min= 8, max= 58, avg=23.85, stdev= 9.69, samples=107 lat (msec) : 10=0.08%, 20=0.04%, 50=0.08%, 100=2.50%, 250=33.83% lat (msec) : 500=43.36%, 750=18.12%, 1000=1.99% cpu : usr=0.06%, sys=1.58%, ctx=3083, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=32357: Thu May 1 23:55:57 2025 read: IOPS=23, BW=94.1KiB/s (96.4kB/s)(5060KiB/53757msec) slat (usec): min=1010, max=17859, avg=4572.36, stdev=2485.13 clat (msec): min=7, max=502, avg=228.46, stdev=82.09 lat (msec): min=9, max=510, avg=233.04, stdev=83.48 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 100], 10.00th=[ 116], 20.00th=[ 144], | 30.00th=[ 176], 40.00th=[ 211], 50.00th=[ 236], 60.00th=[ 253], | 70.00th=[ 275], 80.00th=[ 305], 90.00th=[ 330], 95.00th=[ 359], | 99.00th=[ 414], 99.50th=[ 447], 99.90th=[ 464], 99.95th=[ 502], | 99.99th=[ 502] bw ( KiB/s): min= 24, max= 224, per=25.35%, avg=93.79, stdev=42.83, samples=107 iops : min= 6, max= 56, avg=23.31, stdev=10.72, samples=107 write: IOPS=24, BW=96.4KiB/s (98.7kB/s)(5180KiB/53757msec) slat (usec): min=11, max=3907, avg=130.80, stdev=247.33 clat (msec): min=48, max=886, avg=435.60, stdev=167.89 lat (msec): min=49, max=886, avg=435.73, stdev=167.90 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 199], 10.00th=[ 230], 20.00th=[ 275], | 30.00th=[ 313], 40.00th=[ 368], 50.00th=[ 426], 60.00th=[ 485], | 70.00th=[ 542], 80.00th=[ 592], 90.00th=[ 659], 95.00th=[ 718], | 99.00th=[ 827], 99.50th=[ 835], 99.90th=[ 877], 99.95th=[ 885], | 99.99th=[ 885] bw ( KiB/s): min= 32, max= 240, per=24.96%, avg=96.11, stdev=41.17, samples=107 iops : min= 8, max= 60, avg=23.89, stdev=10.31, samples=107 lat (msec) : 10=0.04%, 20=0.04%, 50=0.08%, 100=2.85%, 250=33.09% lat (msec) : 500=45.12%, 750=16.80%, 1000=1.99% cpu : usr=0.19%, sys=1.47%, ctx=2940, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=371KiB/s (380kB/s), 91.9KiB/s-94.1KiB/s (94.1kB/s-96.4kB/s), io=19.6MiB (20.5MB), run=53757-54125msec WRITE: bw=386KiB/s (395kB/s), 95.9KiB/s-97.3KiB/s (98.2kB/s-99.6kB/s), io=20.4MiB (21.4MB), run=53757-54125msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=32405: Thu May 1 23:55:59 2025 read: IOPS=2, BW=103MiB/s (108MB/s)(40.0MiB/389msec) slat (nsec): min=9783.8k, max=9783.8k, avg=9783806.00, stdev= 0.00 clat (nsec): min=369838k, max=369838k, avg=369837914.00, stdev= 0.00 lat (nsec): min=379644k, max=379644k, avg=379643966.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 372], 10.00th=[ 372], 20.00th=[ 372], | 30.00th=[ 372], 40.00th=[ 372], 50.00th=[ 372], 60.00th=[ 372], | 70.00th=[ 372], 80.00th=[ 372], 90.00th=[ 372], 95.00th=[ 372], | 99.00th=[ 372], 99.50th=[ 372], 99.90th=[ 372], 99.95th=[ 372], | 99.99th=[ 372] lat (msec) : 500=100.00% cpu : usr=0.52%, sys=0.77%, ctx=14, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=103MiB/s (108MB/s), 103MiB/s-103MiB/s (108MB/s-108MB/s), io=40.0MiB (41.9MB), run=389-389msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout