== sanity test 398c: run fio to test AIO ================= 15:46:07 (1789501567) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.484957 s, 86.5 MB/s osc.lustre-OST0000-osc-ffff919c2cd28800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146120: Tue Sep 15 15:48:37 2026 write: IOPS=18, BW=73.3KiB/s (75.0kB/s)(10.0MiB/139785msec); 0 zone resets slat (usec): min=23, max=9811, avg=182.78, stdev=602.23 clat (msec): min=217, max=1432, avg=872.39, stdev=193.55 lat (msec): min=217, max=1432, avg=872.57, stdev=193.54 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 550], 10.00th=[ 634], 20.00th=[ 726], | 30.00th=[ 776], 40.00th=[ 827], 50.00th=[ 877], 60.00th=[ 927], | 70.00th=[ 978], 80.00th=[ 1028], 90.00th=[ 1116], 95.00th=[ 1183], | 99.00th=[ 1284], 99.50th=[ 1334], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1435] bw ( KiB/s): min= 7, max= 256, per=25.03%, avg=73.35, stdev=34.64, samples=275 iops : min= 1, max= 64, avg=17.89, stdev= 8.69, samples=275 lat (msec) : 250=0.62%, 500=2.46%, 750=22.07%, 1000=50.35%, 2000=24.49% cpu : usr=0.05%, sys=0.21%, ctx=1980, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146121: Tue Sep 15 15:48:37 2026 write: IOPS=18, BW=73.3KiB/s (75.0kB/s)(10.0MiB/139738msec); 0 zone resets slat (usec): min=23, max=16505, avg=192.29, stdev=732.32 clat (msec): min=142, max=1406, avg=872.78, stdev=198.25 lat (msec): min=142, max=1406, avg=872.97, stdev=198.25 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 550], 10.00th=[ 634], 20.00th=[ 726], | 30.00th=[ 785], 40.00th=[ 827], 50.00th=[ 869], 60.00th=[ 927], | 70.00th=[ 978], 80.00th=[ 1036], 90.00th=[ 1116], 95.00th=[ 1183], | 99.00th=[ 1318], 99.50th=[ 1368], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 7, max= 218, per=24.74%, avg=72.47, stdev=28.11, samples=278 iops : min= 1, max= 54, avg=17.67, stdev= 7.04, samples=278 lat (msec) : 250=0.78%, 500=2.46%, 750=21.56%, 1000=49.45%, 2000=25.74% cpu : usr=0.04%, sys=0.23%, ctx=2140, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146122: Tue Sep 15 15:48:37 2026 write: IOPS=18, BW=73.3KiB/s (75.1kB/s)(10.0MiB/139630msec); 0 zone resets slat (usec): min=27, max=14424, avg=182.32, stdev=711.78 clat (msec): min=220, max=1400, avg=871.38, stdev=191.03 lat (msec): min=220, max=1400, avg=871.57, stdev=191.03 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 558], 10.00th=[ 634], 20.00th=[ 726], | 30.00th=[ 785], 40.00th=[ 827], 50.00th=[ 869], 60.00th=[ 927], | 70.00th=[ 978], 80.00th=[ 1028], 90.00th=[ 1116], 95.00th=[ 1167], | 99.00th=[ 1284], 99.50th=[ 1334], 99.90th=[ 1351], 99.95th=[ 1351], | 99.99th=[ 1401] bw ( KiB/s): min= 7, max= 256, per=25.12%, avg=73.61, stdev=32.49, samples=274 iops : min= 1, max= 64, avg=17.96, stdev= 8.16, samples=274 lat (msec) : 250=0.62%, 500=2.66%, 750=21.29%, 1000=50.00%, 2000=25.43% cpu : usr=0.04%, sys=0.22%, ctx=1536, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146123: Tue Sep 15 15:48:37 2026 write: IOPS=18, BW=73.3KiB/s (75.0kB/s)(10.0MiB/139788msec); 0 zone resets slat (usec): min=28, max=13634, avg=188.94, stdev=709.15 clat (msec): min=223, max=1395, avg=872.48, stdev=195.94 lat (msec): min=223, max=1396, avg=872.67, stdev=195.93 clat percentiles (msec): | 1.00th=[ 326], 5.00th=[ 550], 10.00th=[ 634], 20.00th=[ 726], | 30.00th=[ 776], 40.00th=[ 827], 50.00th=[ 877], 60.00th=[ 927], | 70.00th=[ 978], 80.00th=[ 1028], 90.00th=[ 1099], 95.00th=[ 1200], | 99.00th=[ 1334], 99.50th=[ 1351], 99.90th=[ 1401], 99.95th=[ 1401], | 99.99th=[ 1401] bw ( KiB/s): min= 7, max= 200, per=24.75%, avg=72.50, stdev=26.52, samples=278 iops : min= 1, max= 50, avg=17.68, stdev= 6.64, samples=278 lat (msec) : 250=0.90%, 500=2.27%, 750=21.21%, 1000=49.69%, 2000=25.94% cpu : usr=0.05%, sys=0.23%, ctx=1879, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=293KiB/s (300kB/s), 73.3KiB/s-73.3KiB/s (75.0kB/s-75.1kB/s), io=40.0MiB (41.9MB), run=139630-139788msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146154: Tue Sep 15 15:50:27 2026 read: IOPS=12, BW=49.7KiB/s (50.9kB/s)(5048KiB/101503msec) slat (usec): min=2173, max=65972, avg=12968.02, stdev=6975.39 clat (msec): min=10, max=1099, avg=396.38, stdev=108.05 lat (msec): min=16, max=1127, avg=409.35, stdev=109.34 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 255], 10.00th=[ 288], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 363], 50.00th=[ 384], 60.00th=[ 409], | 70.00th=[ 430], 80.00th=[ 464], 90.00th=[ 527], 95.00th=[ 575], | 99.00th=[ 793], 99.50th=[ 827], 99.90th=[ 995], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 8, max= 112, per=25.25%, avg=49.48, stdev=22.56, samples=201 iops : min= 2, max= 28, avg=12.00, stdev= 5.64, samples=201 write: IOPS=12, BW=51.2KiB/s (52.4kB/s)(5192KiB/101503msec); 0 zone resets slat (usec): min=29, max=17657, avg=225.22, stdev=741.04 clat (msec): min=271, max=1685, avg=850.73, stdev=227.34 lat (msec): min=272, max=1685, avg=850.96, stdev=227.34 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 502], 10.00th=[ 567], 20.00th=[ 659], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 827], 60.00th=[ 885], | 70.00th=[ 961], 80.00th=[ 1036], 90.00th=[ 1150], 95.00th=[ 1234], | 99.00th=[ 1502], 99.50th=[ 1552], 99.90th=[ 1653], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 112, per=25.00%, avg=50.99, stdev=22.47, samples=200 iops : min= 1, max= 28, avg=12.38, stdev= 5.61, samples=200 lat (msec) : 20=0.04%, 50=0.16%, 100=0.12%, 250=1.76%, 500=43.05% lat (msec) : 750=21.29%, 1000=21.25%, 2000=12.34% cpu : usr=0.08%, sys=0.56%, ctx=4433, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146155: Tue Sep 15 15:50:27 2026 read: IOPS=12, BW=48.8KiB/s (49.0kB/s)(4976KiB/101913msec) slat (usec): min=368, max=45005, avg=12922.07, stdev=6544.53 clat (msec): min=17, max=1121, avg=391.39, stdev=104.95 lat (msec): min=27, max=1160, avg=404.32, stdev=106.14 clat percentiles (msec): | 1.00th=[ 100], 5.00th=[ 253], 10.00th=[ 279], 20.00th=[ 309], | 30.00th=[ 338], 40.00th=[ 359], 50.00th=[ 380], 60.00th=[ 409], | 70.00th=[ 435], 80.00th=[ 468], 90.00th=[ 523], 95.00th=[ 558], | 99.00th=[ 684], 99.50th=[ 709], 99.90th=[ 1083], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 112, per=24.76%, avg=48.53, stdev=22.05, samples=201 iops : min= 1, max= 28, avg=11.77, stdev= 5.51, samples=201 write: IOPS=12, BW=51.7KiB/s (52.9kB/s)(5264KiB/101913msec); 0 zone resets slat (usec): min=32, max=17546, avg=295.84, stdev=1152.37 clat (msec): min=152, max=1709, avg=856.01, stdev=235.55 lat (msec): min=152, max=1709, avg=856.30, stdev=235.52 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 502], 10.00th=[ 567], 20.00th=[ 676], | 30.00th=[ 735], 40.00th=[ 785], 50.00th=[ 835], 60.00th=[ 894], | 70.00th=[ 961], 80.00th=[ 1028], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1536], 99.50th=[ 1569], 99.90th=[ 1603], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 111, per=25.01%, avg=51.01, stdev=20.39, samples=202 iops : min= 1, max= 27, avg=12.39, stdev= 5.09, samples=202 lat (msec) : 20=0.04%, 50=0.23%, 100=0.23%, 250=1.95%, 500=42.15% lat (msec) : 750=20.51%, 1000=22.30%, 2000=12.58% cpu : usr=0.11%, sys=0.60%, ctx=4803, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146156: Tue Sep 15 15:50:27 2026 read: IOPS=12, BW=48.9KiB/s (50.1kB/s)(4984KiB/101919msec) slat (usec): min=2942, max=52937, avg=12730.37, stdev=6620.86 clat (msec): min=16, max=980, avg=398.05, stdev=104.00 lat (msec): min=28, max=1004, avg=410.79, stdev=105.08 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 259], 10.00th=[ 284], 20.00th=[ 317], | 30.00th=[ 347], 40.00th=[ 368], 50.00th=[ 393], 60.00th=[ 418], | 70.00th=[ 443], 80.00th=[ 477], 90.00th=[ 523], 95.00th=[ 567], | 99.00th=[ 693], 99.50th=[ 793], 99.90th=[ 852], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min= 7, max= 126, per=24.87%, avg=48.74, stdev=21.42, samples=201 iops : min= 1, max= 31, avg=11.82, stdev= 5.36, samples=201 write: IOPS=12, BW=51.6KiB/s (52.8kB/s)(5256KiB/101919msec); 0 zone resets slat (usec): min=31, max=11446, avg=226.97, stdev=657.13 clat (msec): min=243, max=1675, avg=849.29, stdev=231.94 lat (msec): min=243, max=1676, avg=849.52, stdev=231.93 clat percentiles (msec): | 1.00th=[ 359], 5.00th=[ 506], 10.00th=[ 567], 20.00th=[ 651], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 835], 60.00th=[ 885], | 70.00th=[ 961], 80.00th=[ 1036], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1485], 99.50th=[ 1536], 99.90th=[ 1620], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 111, per=24.84%, avg=50.67, stdev=20.33, samples=202 iops : min= 1, max= 27, avg=12.30, stdev= 5.06, samples=202 lat (msec) : 20=0.08%, 50=0.16%, 100=0.20%, 250=1.68%, 500=42.11% lat (msec) : 750=22.15%, 1000=20.94%, 2000=12.70% cpu : usr=0.10%, sys=0.58%, ctx=4855, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146157: Tue Sep 15 15:50:27 2026 read: IOPS=12, BW=49.7KiB/s (50.9kB/s)(5060KiB/101710msec) slat (usec): min=2541, max=43655, avg=12850.59, stdev=6608.13 clat (msec): min=17, max=1098, avg=394.75, stdev=104.56 lat (msec): min=24, max=1128, avg=407.61, stdev=105.66 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 255], 10.00th=[ 284], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 363], 50.00th=[ 388], 60.00th=[ 409], | 70.00th=[ 435], 80.00th=[ 472], 90.00th=[ 523], 95.00th=[ 558], | 99.00th=[ 701], 99.50th=[ 793], 99.90th=[ 969], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 103, per=25.21%, avg=49.41, stdev=22.28, samples=202 iops : min= 1, max= 25, avg=11.99, stdev= 5.57, samples=202 write: IOPS=12, BW=50.9KiB/s (52.2kB/s)(5180KiB/101710msec); 0 zone resets slat (usec): min=32, max=9987, avg=258.10, stdev=745.27 clat (msec): min=148, max=1594, avg=856.37, stdev=232.96 lat (msec): min=149, max=1595, avg=856.63, stdev=232.97 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 502], 10.00th=[ 575], 20.00th=[ 651], | 30.00th=[ 726], 40.00th=[ 776], 50.00th=[ 844], 60.00th=[ 902], | 70.00th=[ 969], 80.00th=[ 1062], 90.00th=[ 1167], 95.00th=[ 1267], | 99.00th=[ 1469], 99.50th=[ 1485], 99.90th=[ 1586], 99.95th=[ 1603], | 99.99th=[ 1603] bw ( KiB/s): min= 7, max= 104, per=24.79%, avg=50.56, stdev=20.09, samples=201 iops : min= 1, max= 26, avg=12.27, stdev= 5.05, samples=201 lat (msec) : 20=0.04%, 50=0.27%, 100=0.20%, 250=1.80%, 500=43.05% lat (msec) : 750=21.37%, 1000=20.35%, 2000=12.93% cpu : usr=0.12%, sys=0.54%, ctx=4822, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=197KiB/s (202kB/s), 48.8KiB/s-49.7KiB/s (49.0kB/s-50.9kB/s), io=19.6MiB (20.5MB), run=101503-101919msec WRITE: bw=205KiB/s (210kB/s), 50.9KiB/s-51.7KiB/s (52.2kB/s-52.9kB/s), io=20.4MiB (21.4MB), run=101503-101919msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146175: Tue Sep 15 15:50:36 2026 read: IOPS=2, BW=90.3MiB/s (94.7MB/s)(40.0MiB/443msec) slat (nsec): min=26796k, max=26796k, avg=26796315.00, stdev= 0.00 clat (nsec): min=393061k, max=393061k, avg=393061271.00, stdev= 0.00 lat (nsec): min=419864k, max=419864k, avg=419863575.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 393], 10.00th=[ 393], 20.00th=[ 393], | 30.00th=[ 393], 40.00th=[ 393], 50.00th=[ 393], 60.00th=[ 393], | 70.00th=[ 393], 80.00th=[ 393], 90.00th=[ 393], 95.00th=[ 393], | 99.00th=[ 393], 99.50th=[ 393], 99.90th=[ 393], 99.95th=[ 393], | 99.99th=[ 393] lat (msec) : 500=100.00% cpu : usr=0.23%, sys=2.71%, ctx=7, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=90.3MiB/s (94.7MB/s), 90.3MiB/s-90.3MiB/s (94.7MB/s-94.7MB/s), io=40.0MiB (41.9MB), run=443-443msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout