== sanity test 398c: run fio to test AIO ================= 21:05:58 (1763345158) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.337638 s, 124 MB/s osc.lustre-OST0000-osc-ffff8dfc85261000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=149275: Sun Nov 16 21:08:41 2025 write: IOPS=16, BW=65.4KiB/s (66.0kB/s)(10.0MiB/156554msec); 0 zone resets slat (usec): min=27, max=8886, avg=173.03, stdev=453.04 clat (msec): min=231, max=2074, avg=977.55, stdev=246.53 lat (msec): min=231, max=2076, avg=977.72, stdev=246.52 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 575], 10.00th=[ 701], 20.00th=[ 810], | 30.00th=[ 869], 40.00th=[ 927], 50.00th=[ 978], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1250], 95.00th=[ 1334], | 99.00th=[ 1552], 99.50th=[ 1787], 99.90th=[ 2072], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 254, per=24.79%, avg=64.69, stdev=31.34, samples=311 iops : min= 1, max= 63, avg=15.69, stdev= 7.84, samples=311 lat (msec) : 250=0.74%, 500=3.75%, 750=9.38%, 1000=40.16%, 2000=45.66% lat (msec) : >=2000=0.31% cpu : usr=0.05%, sys=0.27%, ctx=1838, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149276: Sun Nov 16 21:08:41 2025 write: IOPS=16, BW=65.4KiB/s (66.9kB/s)(10.0MiB/156654msec); 0 zone resets slat (usec): min=27, max=9633, avg=168.12, stdev=467.25 clat (msec): min=195, max=2077, avg=978.10, stdev=246.76 lat (msec): min=195, max=2078, avg=978.27, stdev=246.77 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 584], 10.00th=[ 693], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 936], 50.00th=[ 986], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1167], 90.00th=[ 1250], 95.00th=[ 1351], | 99.00th=[ 1552], 99.50th=[ 1620], 99.90th=[ 2072], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 255, per=24.86%, avg=64.89, stdev=29.88, samples=310 iops : min= 1, max= 63, avg=15.73, stdev= 7.46, samples=310 lat (msec) : 250=0.98%, 500=3.48%, 750=10.70%, 1000=37.66%, 2000=46.88% lat (msec) : >=2000=0.31% cpu : usr=0.04%, sys=0.26%, ctx=1762, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149277: Sun Nov 16 21:08:41 2025 write: IOPS=16, BW=65.5KiB/s (67.1kB/s)(10.0MiB/156366msec); 0 zone resets slat (usec): min=28, max=9385, avg=170.03, stdev=406.04 clat (msec): min=101, max=2016, avg=976.60, stdev=251.80 lat (msec): min=101, max=2016, avg=976.77, stdev=251.79 clat percentiles (msec): | 1.00th=[ 243], 5.00th=[ 558], 10.00th=[ 693], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 927], 50.00th=[ 978], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1167], 90.00th=[ 1267], 95.00th=[ 1351], | 99.00th=[ 1552], 99.50th=[ 1787], 99.90th=[ 2005], 99.95th=[ 2005], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 223, per=24.87%, avg=64.92, stdev=29.76, samples=310 iops : min= 1, max= 55, avg=15.74, stdev= 7.41, samples=310 lat (msec) : 250=1.05%, 500=3.59%, 750=9.73%, 1000=39.22%, 2000=46.37% lat (msec) : >=2000=0.04% cpu : usr=0.04%, sys=0.26%, ctx=1901, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149278: Sun Nov 16 21:08:41 2025 write: IOPS=16, BW=65.4KiB/s (66.0kB/s)(10.0MiB/156544msec); 0 zone resets slat (usec): min=28, max=5357, avg=159.03, stdev=362.47 clat (msec): min=223, max=2078, avg=977.52, stdev=245.66 lat (msec): min=223, max=2078, avg=977.68, stdev=245.65 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 600], 10.00th=[ 693], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 927], 50.00th=[ 986], 60.00th=[ 1036], | 70.00th=[ 1099], 80.00th=[ 1167], 90.00th=[ 1267], 95.00th=[ 1351], | 99.00th=[ 1569], 99.50th=[ 1720], 99.90th=[ 2072], 99.95th=[ 2072], | 99.99th=[ 2072] bw ( KiB/s): min= 7, max= 254, per=24.80%, avg=64.72, stdev=27.98, samples=311 iops : min= 1, max= 63, avg=15.69, stdev= 6.99, samples=311 lat (msec) : 250=0.86%, 500=3.59%, 750=9.45%, 1000=38.79%, 2000=47.03% lat (msec) : >=2000=0.27% cpu : usr=0.04%, sys=0.26%, ctx=1827, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=261KiB/s (268kB/s), 65.4KiB/s-65.5KiB/s (66.9kB/s-67.1kB/s), io=40.0MiB (41.9MB), run=156366-156654msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=149309: Sun Nov 16 21:10:21 2025 read: IOPS=13, BW=53.4KiB/s (54.7kB/s)(5048KiB/94492msec) slat (usec): min=3103, max=41369, avg=10974.13, stdev=4778.73 clat (msec): min=24, max=654, avg=378.02, stdev=82.03 lat (msec): min=30, max=670, avg=389.01, stdev=82.75 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 257], 10.00th=[ 284], 20.00th=[ 313], | 30.00th=[ 334], 40.00th=[ 355], 50.00th=[ 376], 60.00th=[ 401], | 70.00th=[ 418], 80.00th=[ 443], 90.00th=[ 477], 95.00th=[ 518], | 99.00th=[ 575], 99.50th=[ 609], 99.90th=[ 642], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 8, max= 119, per=24.95%, avg=52.89, stdev=20.52, samples=187 iops : min= 2, max= 29, avg=12.68, stdev= 5.19, samples=187 write: IOPS=13, BW=54.9KiB/s (56.3kB/s)(5192KiB/94492msec); 0 zone resets slat (usec): min=32, max=12975, avg=240.39, stdev=681.88 clat (msec): min=176, max=1412, avg=784.17, stdev=194.01 lat (msec): min=179, max=1412, avg=784.42, stdev=194.02 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 460], 10.00th=[ 550], 20.00th=[ 625], | 30.00th=[ 676], 40.00th=[ 726], 50.00th=[ 785], 60.00th=[ 835], | 70.00th=[ 885], 80.00th=[ 944], 90.00th=[ 1028], 95.00th=[ 1099], | 99.00th=[ 1267], 99.50th=[ 1318], 99.90th=[ 1401], 99.95th=[ 1418], | 99.99th=[ 1418] bw ( KiB/s): min= 7, max= 119, per=24.70%, avg=54.33, stdev=20.52, samples=187 iops : min= 1, max= 29, avg=13.04, stdev= 5.15, samples=187 lat (msec) : 50=0.12%, 100=0.20%, 250=1.76%, 500=47.15%, 750=22.46% lat (msec) : 1000=22.03%, 2000=6.29% cpu : usr=0.10%, sys=0.70%, ctx=4735, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149310: Sun Nov 16 21:10:21 2025 read: IOPS=13, BW=52.7KiB/s (53.9kB/s)(4976KiB/94495msec) slat (usec): min=2613, max=41314, avg=10705.89, stdev=4661.92 clat (msec): min=14, max=698, avg=374.67, stdev=79.81 lat (msec): min=21, max=710, avg=385.39, stdev=80.71 clat percentiles (msec): | 1.00th=[ 184], 5.00th=[ 255], 10.00th=[ 279], 20.00th=[ 309], | 30.00th=[ 334], 40.00th=[ 351], 50.00th=[ 376], 60.00th=[ 397], | 70.00th=[ 418], 80.00th=[ 439], 90.00th=[ 472], 95.00th=[ 498], | 99.00th=[ 567], 99.50th=[ 609], 99.90th=[ 693], 99.95th=[ 701], | 99.99th=[ 701] bw ( KiB/s): min= 7, max= 127, per=24.54%, avg=52.02, stdev=23.02, samples=187 iops : min= 1, max= 31, avg=12.47, stdev= 5.82, samples=187 write: IOPS=13, BW=55.7KiB/s (57.0kB/s)(5264KiB/94495msec); 0 zone resets slat (usec): min=33, max=11311, avg=226.71, stdev=531.78 clat (msec): min=182, max=1457, avg=782.40, stdev=196.66 lat (msec): min=182, max=1457, avg=782.63, stdev=196.65 clat percentiles (msec): | 1.00th=[ 292], 5.00th=[ 472], 10.00th=[ 550], 20.00th=[ 617], | 30.00th=[ 676], 40.00th=[ 726], 50.00th=[ 776], 60.00th=[ 844], | 70.00th=[ 885], 80.00th=[ 944], 90.00th=[ 1028], 95.00th=[ 1099], | 99.00th=[ 1250], 99.50th=[ 1301], 99.90th=[ 1401], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 15, max= 111, per=24.99%, avg=54.97, stdev=20.54, samples=187 iops : min= 3, max= 27, avg=13.20, stdev= 5.14, samples=187 lat (msec) : 20=0.08%, 50=0.04%, 100=0.16%, 250=2.34%, 500=47.03% lat (msec) : 750=21.84%, 1000=21.80%, 2000=6.72% cpu : usr=0.10%, sys=0.70%, ctx=4901, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149311: Sun Nov 16 21:10:21 2025 read: IOPS=13, BW=52.7KiB/s (53.0kB/s)(4984KiB/94550msec) slat (usec): min=2804, max=39086, avg=11064.54, stdev=4808.18 clat (msec): min=9, max=687, avg=373.74, stdev=83.31 lat (msec): min=17, max=705, avg=384.81, stdev=83.93 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 249], 10.00th=[ 275], 20.00th=[ 309], | 30.00th=[ 334], 40.00th=[ 355], 50.00th=[ 376], 60.00th=[ 397], | 70.00th=[ 418], 80.00th=[ 439], 90.00th=[ 472], 95.00th=[ 514], | 99.00th=[ 558], 99.50th=[ 584], 99.90th=[ 642], 99.95th=[ 684], | 99.99th=[ 684] bw ( KiB/s): min= 7, max= 135, per=24.85%, avg=52.68, stdev=24.07, samples=186 iops : min= 1, max= 33, avg=12.62, stdev= 6.07, samples=186 write: IOPS=13, BW=55.6KiB/s (56.9kB/s)(5256KiB/94550msec); 0 zone resets slat (usec): min=34, max=8219, avg=225.84, stdev=499.55 clat (msec): min=179, max=1456, avg=783.98, stdev=197.16 lat (msec): min=179, max=1456, avg=784.21, stdev=197.16 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 456], 10.00th=[ 542], 20.00th=[ 617], | 30.00th=[ 676], 40.00th=[ 743], 50.00th=[ 785], 60.00th=[ 835], | 70.00th=[ 885], 80.00th=[ 944], 90.00th=[ 1036], 95.00th=[ 1099], | 99.00th=[ 1267], 99.50th=[ 1301], 99.90th=[ 1351], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 15, max= 127, per=24.82%, avg=54.61, stdev=20.42, samples=188 iops : min= 3, max= 31, avg=13.11, stdev= 5.12, samples=188 lat (msec) : 10=0.04%, 50=0.16%, 100=0.31%, 250=2.19%, 500=46.56% lat (msec) : 750=21.29%, 1000=22.50%, 2000=6.95% cpu : usr=0.10%, sys=0.73%, ctx=4643, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149312: Sun Nov 16 21:10:21 2025 read: IOPS=13, BW=53.8KiB/s (55.1kB/s)(5060KiB/94013msec) slat (usec): min=54, max=40332, avg=10867.43, stdev=4915.52 clat (msec): min=14, max=829, avg=376.67, stdev=81.64 lat (msec): min=22, max=843, avg=387.54, stdev=82.51 clat percentiles (msec): | 1.00th=[ 182], 5.00th=[ 257], 10.00th=[ 284], 20.00th=[ 313], | 30.00th=[ 334], 40.00th=[ 355], 50.00th=[ 376], 60.00th=[ 393], | 70.00th=[ 418], 80.00th=[ 443], 90.00th=[ 477], 95.00th=[ 506], | 99.00th=[ 575], 99.50th=[ 609], 99.90th=[ 709], 99.95th=[ 827], | 99.99th=[ 827] bw ( KiB/s): min= 7, max= 119, per=25.03%, avg=53.06, stdev=22.04, samples=187 iops : min= 1, max= 29, avg=12.73, stdev= 5.51, samples=187 write: IOPS=13, BW=55.1KiB/s (56.4kB/s)(5180KiB/94013msec); 0 zone resets slat (usec): min=31, max=11123, avg=245.28, stdev=620.29 clat (msec): min=86, max=1467, avg=781.92, stdev=199.31 lat (msec): min=86, max=1467, avg=782.17, stdev=199.30 clat percentiles (msec): | 1.00th=[ 266], 5.00th=[ 456], 10.00th=[ 535], 20.00th=[ 617], | 30.00th=[ 684], 40.00th=[ 743], 50.00th=[ 785], 60.00th=[ 835], | 70.00th=[ 885], 80.00th=[ 936], 90.00th=[ 1020], 95.00th=[ 1099], | 99.00th=[ 1284], 99.50th=[ 1318], 99.90th=[ 1452], 99.95th=[ 1469], | 99.99th=[ 1469] bw ( KiB/s): min= 7, max= 109, per=24.74%, avg=54.43, stdev=19.72, samples=186 iops : min= 1, max= 27, avg=13.07, stdev= 4.94, samples=186 lat (msec) : 20=0.04%, 50=0.27%, 100=0.12%, 250=2.03%, 500=48.16% lat (msec) : 750=19.92%, 1000=23.36%, 2000=6.09% cpu : usr=0.13%, sys=0.69%, ctx=4743, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=212KiB/s (217kB/s), 52.7KiB/s-53.8KiB/s (53.9kB/s-55.1kB/s), io=19.6MiB (20.5MB), run=94013-94550msec WRITE: bw=221KiB/s (226kB/s), 54.9KiB/s-55.7KiB/s (56.3kB/s-57.0kB/s), io=20.4MiB (21.4MB), run=94013-94550msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=149329: Sun Nov 16 21:10:27 2025 read: IOPS=2, BW=96.4MiB/s (101MB/s)(40.0MiB/415msec) slat (nsec): min=36507k, max=36507k, avg=36507100.00, stdev= 0.00 clat (nsec): min=360321k, max=360321k, avg=360321390.00, stdev= 0.00 lat (nsec): min=396834k, max=396834k, avg=396833762.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 359], 10.00th=[ 359], 20.00th=[ 359], | 30.00th=[ 359], 40.00th=[ 359], 50.00th=[ 359], 60.00th=[ 359], | 70.00th=[ 359], 80.00th=[ 359], 90.00th=[ 359], 95.00th=[ 359], | 99.00th=[ 359], 99.50th=[ 359], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] lat (msec) : 500=100.00% cpu : usr=0.48%, sys=1.93%, ctx=6, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=96.4MiB/s (101MB/s), 96.4MiB/s-96.4MiB/s (101MB/s-101MB/s), io=40.0MiB (41.9MB), run=415-415msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout