== sanity test 398c: run fio to test AIO ================= 13:33:25 (1781285605) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.434898 s, 96.4 MB/s osc.lustre-OST0000-osc-ffff8b70d178a800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=143573: Fri Jun 12 13:36:12 2026 write: IOPS=16, BW=65.6KiB/s (67.2kB/s)(10.0MiB/156150msec); 0 zone resets slat (usec): min=23, max=23166, avg=331.82, stdev=1187.70 clat (msec): min=316, max=1836, avg=974.61, stdev=248.29 lat (msec): min=316, max=1836, avg=974.95, stdev=248.33 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 609], 10.00th=[ 693], 20.00th=[ 768], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 961], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1167], 90.00th=[ 1284], 95.00th=[ 1385], | 99.00th=[ 1687], 99.50th=[ 1770], 99.90th=[ 1821], 99.95th=[ 1821], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 256, per=25.07%, avg=65.68, stdev=29.56, samples=307 iops : min= 1, max= 64, avg=15.96, stdev= 7.42, samples=307 lat (msec) : 500=2.30%, 750=15.70%, 1000=39.10%, 2000=42.89% cpu : usr=0.05%, sys=0.30%, ctx=1758, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=143574: Fri Jun 12 13:36:12 2026 write: IOPS=16, BW=65.7KiB/s (67.2kB/s)(10.0MiB/155926msec); 0 zone resets slat (usec): min=21, max=18314, avg=308.65, stdev=1048.97 clat (msec): min=314, max=1807, avg=973.41, stdev=246.02 lat (msec): min=314, max=1807, avg=973.73, stdev=246.03 clat percentiles (msec): | 1.00th=[ 321], 5.00th=[ 625], 10.00th=[ 693], 20.00th=[ 776], | 30.00th=[ 844], 40.00th=[ 894], 50.00th=[ 961], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1167], 90.00th=[ 1284], 95.00th=[ 1385], | 99.00th=[ 1670], 99.50th=[ 1737], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 1804] bw ( KiB/s): min= 8, max= 256, per=24.90%, avg=65.24, stdev=29.64, samples=309 iops : min= 2, max= 64, avg=15.85, stdev= 7.41, samples=309 lat (msec) : 500=2.46%, 750=13.67%, 1000=40.51%, 2000=43.36% cpu : usr=0.05%, sys=0.28%, ctx=1554, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=143575: Fri Jun 12 13:36:12 2026 write: IOPS=16, BW=65.6KiB/s (67.2kB/s)(10.0MiB/156149msec); 0 zone resets slat (usec): min=24, max=19910, avg=284.16, stdev=1059.22 clat (msec): min=172, max=1894, avg=974.89, stdev=250.91 lat (msec): min=172, max=1894, avg=975.17, stdev=250.94 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 634], 10.00th=[ 684], 20.00th=[ 768], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 961], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1284], 95.00th=[ 1401], | 99.00th=[ 1703], 99.50th=[ 1787], 99.90th=[ 1871], 99.95th=[ 1871], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 256, per=25.15%, avg=65.89, stdev=28.71, samples=306 iops : min= 1, max= 64, avg=16.02, stdev= 7.20, samples=306 lat (msec) : 250=0.04%, 500=2.11%, 750=15.16%, 1000=40.08%, 2000=42.62% cpu : usr=0.05%, sys=0.30%, ctx=1573, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=143576: Fri Jun 12 13:36:12 2026 write: IOPS=16, BW=65.7KiB/s (67.3kB/s)(10.0MiB/155788msec); 0 zone resets slat (usec): min=26, max=19351, avg=325.24, stdev=1077.86 clat (msec): min=106, max=1890, avg=972.89, stdev=252.44 lat (msec): min=107, max=1890, avg=973.21, stdev=252.45 clat percentiles (msec): | 1.00th=[ 330], 5.00th=[ 600], 10.00th=[ 693], 20.00th=[ 776], | 30.00th=[ 844], 40.00th=[ 894], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1301], 95.00th=[ 1401], | 99.00th=[ 1670], 99.50th=[ 1737], 99.90th=[ 1838], 99.95th=[ 1854], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 200, per=25.05%, avg=65.64, stdev=33.10, samples=307 iops : min= 1, max= 50, avg=15.95, stdev= 8.29, samples=307 lat (msec) : 250=0.47%, 500=2.70%, 750=13.32%, 1000=40.51%, 2000=43.01% cpu : usr=0.06%, sys=0.30%, ctx=1584, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=262KiB/s (269kB/s), 65.6KiB/s-65.7KiB/s (67.2kB/s-67.3kB/s), io=40.0MiB (41.9MB), run=155788-156150msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=143609: Fri Jun 12 13:38:02 2026 read: IOPS=12, BW=49.5KiB/s (50.7kB/s)(5048KiB/102050msec) slat (usec): min=2512, max=65006, avg=12583.78, stdev=6879.38 clat (msec): min=60, max=774, avg=400.85, stdev=92.25 lat (msec): min=67, max=787, avg=413.44, stdev=92.88 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 264], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 351], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 414], | 70.00th=[ 443], 80.00th=[ 472], 90.00th=[ 527], 95.00th=[ 558], | 99.00th=[ 651], 99.50th=[ 676], 99.90th=[ 768], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 7, max= 112, per=24.93%, avg=48.87, stdev=21.51, samples=201 iops : min= 1, max= 28, avg=11.67, stdev= 5.35, samples=201 write: IOPS=12, BW=50.9KiB/s (52.1kB/s)(5192KiB/102050msec); 0 zone resets slat (usec): min=31, max=20070, avg=401.02, stdev=1289.57 clat (msec): min=122, max=1755, avg=853.94, stdev=231.75 lat (msec): min=122, max=1755, avg=854.35, stdev=231.72 clat percentiles (msec): | 1.00th=[ 388], 5.00th=[ 531], 10.00th=[ 592], 20.00th=[ 667], | 30.00th=[ 718], 40.00th=[ 768], 50.00th=[ 827], 60.00th=[ 885], | 70.00th=[ 961], 80.00th=[ 1036], 90.00th=[ 1133], 95.00th=[ 1284], | 99.00th=[ 1519], 99.50th=[ 1603], 99.90th=[ 1703], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 7, max= 111, per=24.49%, avg=49.96, stdev=20.33, samples=202 iops : min= 1, max= 27, avg=11.95, stdev= 5.10, samples=202 lat (msec) : 100=0.12%, 250=1.80%, 500=42.27%, 750=23.24%, 1000=19.84% lat (msec) : 2000=12.73% cpu : usr=0.12%, sys=0.80%, ctx=3363, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=143610: Fri Jun 12 13:38:02 2026 read: IOPS=12, BW=48.8KiB/s (49.9kB/s)(4976KiB/102047msec) slat (usec): min=195, max=48760, avg=12350.67, stdev=6815.63 clat (msec): min=11, max=765, avg=398.82, stdev=94.77 lat (msec): min=40, max=786, avg=411.18, stdev=95.23 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 268], 10.00th=[ 292], 20.00th=[ 326], | 30.00th=[ 351], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 414], | 70.00th=[ 439], 80.00th=[ 477], 90.00th=[ 514], 95.00th=[ 558], | 99.00th=[ 676], 99.50th=[ 684], 99.90th=[ 760], 99.95th=[ 768], | 99.99th=[ 768] bw ( KiB/s): min= 7, max= 127, per=24.49%, avg=48.00, stdev=22.76, samples=202 iops : min= 1, max= 31, avg=11.45, stdev= 5.69, samples=202 write: IOPS=12, BW=51.6KiB/s (52.8kB/s)(5264KiB/102047msec); 0 zone resets slat (usec): min=28, max=15697, avg=429.89, stdev=1168.42 clat (msec): min=115, max=1791, avg=851.03, stdev=243.56 lat (msec): min=117, max=1791, avg=851.47, stdev=243.50 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 498], 10.00th=[ 575], 20.00th=[ 659], | 30.00th=[ 709], 40.00th=[ 768], 50.00th=[ 835], 60.00th=[ 885], | 70.00th=[ 961], 80.00th=[ 1045], 90.00th=[ 1167], 95.00th=[ 1301], | 99.00th=[ 1536], 99.50th=[ 1636], 99.90th=[ 1754], 99.95th=[ 1787], | 99.99th=[ 1787] bw ( KiB/s): min= 8, max= 119, per=24.83%, avg=50.66, stdev=20.90, samples=202 iops : min= 2, max= 29, avg=12.12, stdev= 5.22, samples=202 lat (msec) : 20=0.04%, 50=0.08%, 100=0.23%, 250=1.52%, 500=43.09% lat (msec) : 750=22.15%, 1000=19.88%, 2000=13.01% cpu : usr=0.12%, sys=0.76%, ctx=3283, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=143611: Fri Jun 12 13:38:02 2026 read: IOPS=12, BW=48.9KiB/s (50.0kB/s)(4984KiB/102001msec) slat (usec): min=2182, max=46390, avg=12548.25, stdev=7163.32 clat (msec): min=9, max=762, avg=397.47, stdev=96.06 lat (msec): min=15, max=786, avg=410.03, stdev=96.56 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 264], 10.00th=[ 292], 20.00th=[ 326], | 30.00th=[ 347], 40.00th=[ 368], 50.00th=[ 388], 60.00th=[ 414], | 70.00th=[ 439], 80.00th=[ 477], 90.00th=[ 518], 95.00th=[ 558], | 99.00th=[ 651], 99.50th=[ 684], 99.90th=[ 760], 99.95th=[ 760], | 99.99th=[ 760] bw ( KiB/s): min= 7, max= 111, per=24.81%, avg=48.63, stdev=22.02, samples=201 iops : min= 1, max= 27, avg=11.61, stdev= 5.48, samples=201 write: IOPS=12, BW=51.5KiB/s (52.8kB/s)(5256KiB/102001msec); 0 zone resets slat (usec): min=29, max=18181, avg=417.69, stdev=1239.35 clat (msec): min=238, max=1707, avg=851.44, stdev=228.24 lat (msec): min=244, max=1707, avg=851.86, stdev=228.27 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 506], 10.00th=[ 584], 20.00th=[ 667], | 30.00th=[ 726], 40.00th=[ 776], 50.00th=[ 827], 60.00th=[ 885], | 70.00th=[ 961], 80.00th=[ 1036], 90.00th=[ 1150], 95.00th=[ 1250], | 99.00th=[ 1502], 99.50th=[ 1569], 99.90th=[ 1687], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 15, max= 110, per=24.84%, avg=50.68, stdev=17.69, samples=202 iops : min= 3, max= 27, avg=12.12, stdev= 4.46, samples=202 lat (msec) : 10=0.04%, 50=0.16%, 100=0.31%, 250=1.41%, 500=42.30% lat (msec) : 750=22.54%, 1000=20.94%, 2000=12.30% cpu : usr=0.13%, sys=0.79%, ctx=3372, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=143612: Fri Jun 12 13:38:02 2026 read: IOPS=12, BW=49.7KiB/s (50.9kB/s)(5060KiB/101734msec) slat (usec): min=2336, max=46945, avg=12300.16, stdev=6803.78 clat (msec): min=18, max=813, avg=398.65, stdev=95.66 lat (msec): min=22, max=819, avg=410.96, stdev=96.31 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 264], 10.00th=[ 288], 20.00th=[ 326], | 30.00th=[ 351], 40.00th=[ 372], 50.00th=[ 393], 60.00th=[ 414], | 70.00th=[ 439], 80.00th=[ 477], 90.00th=[ 518], 95.00th=[ 558], | 99.00th=[ 676], 99.50th=[ 693], 99.90th=[ 785], 99.95th=[ 818], | 99.99th=[ 818] bw ( KiB/s): min= 7, max= 126, per=25.04%, avg=49.08, stdev=22.14, samples=202 iops : min= 1, max= 31, avg=11.72, stdev= 5.54, samples=202 write: IOPS=12, BW=50.9KiB/s (52.1kB/s)(5180KiB/101734msec); 0 zone resets slat (usec): min=28, max=20138, avg=386.85, stdev=1105.88 clat (msec): min=143, max=1753, avg=853.77, stdev=231.61 lat (msec): min=144, max=1754, avg=854.16, stdev=231.62 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 518], 10.00th=[ 584], 20.00th=[ 659], | 30.00th=[ 726], 40.00th=[ 776], 50.00th=[ 827], 60.00th=[ 885], | 70.00th=[ 953], 80.00th=[ 1028], 90.00th=[ 1150], 95.00th=[ 1284], | 99.00th=[ 1552], 99.50th=[ 1636], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 1754] bw ( KiB/s): min= 15, max= 110, per=24.55%, avg=50.08, stdev=19.42, samples=202 iops : min= 3, max= 27, avg=11.97, stdev= 4.86, samples=202 lat (msec) : 20=0.04%, 50=0.12%, 100=0.12%, 250=1.64%, 500=43.20% lat (msec) : 750=21.48%, 1000=21.52%, 2000=11.88% cpu : usr=0.11%, sys=0.76%, ctx=3432, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=197KiB/s (201kB/s), 48.8KiB/s-49.7KiB/s (49.9kB/s-50.9kB/s), io=19.6MiB (20.5MB), run=101734-102050msec WRITE: bw=205KiB/s (210kB/s), 50.9KiB/s-51.6KiB/s (52.1kB/s-52.8kB/s), io=20.4MiB (21.4MB), run=101734-102050msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=143631: Fri Jun 12 13:38:11 2026 read: IOPS=1, BW=71.6MiB/s (75.0MB/s)(40.0MiB/559msec) slat (nsec): min=47109k, max=47109k, avg=47109165.00, stdev= 0.00 clat (nsec): min=472409k, max=472409k, avg=472408837.00, stdev= 0.00 lat (nsec): min=519524k, max=519524k, avg=519523727.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 472], 10.00th=[ 472], 20.00th=[ 472], | 30.00th=[ 472], 40.00th=[ 472], 50.00th=[ 472], 60.00th=[ 472], | 70.00th=[ 472], 80.00th=[ 472], 90.00th=[ 472], 95.00th=[ 472], | 99.00th=[ 472], 99.50th=[ 472], 99.90th=[ 472], 99.95th=[ 472], | 99.99th=[ 472] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.61%, ctx=5, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=71.6MiB/s (75.0MB/s), 71.6MiB/s-71.6MiB/s (75.0MB/s-75.0MB/s), io=40.0MiB (41.9MB), run=559-559msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck