== sanity test 398c: run fio to test AIO ================= 11:17:57 (1787584677) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.435446 s, 96.3 MB/s osc.lustre-OST0000-osc-ffff941ac7cc9000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145574: Mon Aug 24 11:20:38 2026 write: IOPS=16, BW=67.5KiB/s (69.1kB/s)(10.0MiB/151645msec); 0 zone resets slat (usec): min=27, max=12162, avg=174.42, stdev=569.64 clat (msec): min=213, max=1997, avg=946.85, stdev=256.47 lat (msec): min=213, max=1997, avg=947.03, stdev=256.49 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 592], 10.00th=[ 684], 20.00th=[ 760], | 30.00th=[ 827], 40.00th=[ 869], 50.00th=[ 919], 60.00th=[ 978], | 70.00th=[ 1036], 80.00th=[ 1133], 90.00th=[ 1267], 95.00th=[ 1385], | 99.00th=[ 1720], 99.50th=[ 1804], 99.90th=[ 1955], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 256, per=25.30%, avg=68.05, stdev=32.12, samples=296 iops : min= 1, max= 64, avg=16.54, stdev= 8.07, samples=296 lat (msec) : 250=0.62%, 500=3.01%, 750=15.27%, 1000=44.88%, 2000=36.21% cpu : usr=0.04%, sys=0.22%, ctx=1548, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145575: Mon Aug 24 11:20:38 2026 write: IOPS=16, BW=67.5KiB/s (69.1kB/s)(10.0MiB/151652msec); 0 zone resets slat (usec): min=28, max=11531, avg=188.06, stdev=671.75 clat (msec): min=164, max=2047, avg=947.17, stdev=258.50 lat (msec): min=164, max=2047, avg=947.36, stdev=258.52 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 592], 10.00th=[ 684], 20.00th=[ 768], | 30.00th=[ 827], 40.00th=[ 877], 50.00th=[ 919], 60.00th=[ 969], | 70.00th=[ 1036], 80.00th=[ 1133], 90.00th=[ 1267], 95.00th=[ 1385], | 99.00th=[ 1787], 99.50th=[ 1854], 99.90th=[ 1989], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 223, per=24.94%, avg=67.09, stdev=28.92, samples=300 iops : min= 1, max= 55, avg=16.30, stdev= 7.29, samples=300 lat (msec) : 250=0.62%, 500=3.12%, 750=13.83%, 1000=47.58%, 2000=34.77% lat (msec) : >=2000=0.08% cpu : usr=0.04%, sys=0.21%, ctx=1605, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145576: Mon Aug 24 11:20:38 2026 write: IOPS=16, BW=67.5KiB/s (69.1kB/s)(10.0MiB/151652msec); 0 zone resets slat (usec): min=27, max=13918, avg=174.12, stdev=532.41 clat (msec): min=71, max=2015, avg=946.70, stdev=254.13 lat (msec): min=72, max=2015, avg=946.87, stdev=254.12 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 592], 10.00th=[ 676], 20.00th=[ 768], | 30.00th=[ 827], 40.00th=[ 877], 50.00th=[ 927], 60.00th=[ 978], | 70.00th=[ 1036], 80.00th=[ 1133], 90.00th=[ 1267], 95.00th=[ 1385], | 99.00th=[ 1770], 99.50th=[ 1821], 99.90th=[ 2005], 99.95th=[ 2022], | 99.99th=[ 2022] bw ( KiB/s): min= 7, max= 248, per=25.19%, avg=67.76, stdev=31.20, samples=299 iops : min= 1, max= 62, avg=16.47, stdev= 7.83, samples=299 lat (msec) : 100=0.04%, 250=0.27%, 500=3.32%, 750=13.87%, 1000=47.07% lat (msec) : 2000=35.27%, >=2000=0.16% cpu : usr=0.04%, sys=0.20%, ctx=1927, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145577: Mon Aug 24 11:20:38 2026 write: IOPS=16, BW=67.5KiB/s (69.1kB/s)(10.0MiB/151709msec); 0 zone resets slat (usec): min=28, max=12276, avg=182.58, stdev=639.44 clat (msec): min=276, max=2005, avg=947.08, stdev=256.34 lat (msec): min=276, max=2005, avg=947.27, stdev=256.33 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 567], 10.00th=[ 676], 20.00th=[ 760], | 30.00th=[ 827], 40.00th=[ 869], 50.00th=[ 919], 60.00th=[ 986], | 70.00th=[ 1036], 80.00th=[ 1133], 90.00th=[ 1267], 95.00th=[ 1368], | 99.00th=[ 1770], 99.50th=[ 1854], 99.90th=[ 2005], 99.95th=[ 2005], | 99.99th=[ 2005] bw ( KiB/s): min= 7, max= 256, per=24.96%, avg=67.14, stdev=28.23, samples=300 iops : min= 1, max= 64, avg=16.32, stdev= 7.12, samples=300 lat (msec) : 500=3.36%, 750=14.65%, 1000=44.57%, 2000=37.27%, >=2000=0.16% cpu : usr=0.04%, sys=0.22%, ctx=1720, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=270KiB/s (276kB/s), 67.5KiB/s-67.5KiB/s (69.1kB/s-69.1kB/s), io=40.0MiB (41.9MB), run=151645-151709msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145608: Mon Aug 24 11:22:31 2026 read: IOPS=11, BW=47.5KiB/s (48.6kB/s)(5048KiB/106368msec) slat (usec): min=26, max=55766, avg=13382.76, stdev=7902.10 clat (msec): min=20, max=1238, avg=402.88, stdev=115.85 lat (msec): min=36, max=1249, avg=416.27, stdev=117.15 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 255], 10.00th=[ 284], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 368], 50.00th=[ 397], 60.00th=[ 422], | 70.00th=[ 447], 80.00th=[ 485], 90.00th=[ 535], 95.00th=[ 575], | 99.00th=[ 751], 99.50th=[ 818], 99.90th=[ 1167], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 7, max= 120, per=25.19%, avg=47.10, stdev=22.25, samples=210 iops : min= 1, max= 30, avg=11.29, stdev= 5.64, samples=210 write: IOPS=12, BW=48.8KiB/s (49.0kB/s)(5192KiB/106368msec); 0 zone resets slat (usec): min=27, max=9340, avg=216.31, stdev=670.32 clat (msec): min=110, max=1708, avg=905.13, stdev=237.88 lat (msec): min=110, max=1708, avg=905.35, stdev=237.87 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 498], 10.00th=[ 592], 20.00th=[ 693], | 30.00th=[ 785], 40.00th=[ 852], 50.00th=[ 927], 60.00th=[ 986], | 70.00th=[ 1028], 80.00th=[ 1099], 90.00th=[ 1200], 95.00th=[ 1267], | 99.00th=[ 1435], 99.50th=[ 1519], 99.90th=[ 1653], 99.95th=[ 1703], | 99.99th=[ 1703] bw ( KiB/s): min= 7, max= 116, per=24.67%, avg=48.11, stdev=20.39, samples=211 iops : min= 1, max= 29, avg=11.54, stdev= 5.15, samples=211 lat (msec) : 50=0.20%, 100=0.23%, 250=1.91%, 500=41.13%, 750=18.32% lat (msec) : 1000=20.16%, 2000=18.05% cpu : usr=0.08%, sys=0.50%, ctx=3741, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145609: Mon Aug 24 11:22:31 2026 read: IOPS=11, BW=46.6KiB/s (47.7kB/s)(4976KiB/106804msec) slat (usec): min=2374, max=59160, avg=13856.67, stdev=8239.72 clat (msec): min=15, max=897, avg=406.21, stdev=105.05 lat (msec): min=20, max=924, avg=420.07, stdev=106.39 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 262], 10.00th=[ 284], 20.00th=[ 321], | 30.00th=[ 347], 40.00th=[ 372], 50.00th=[ 401], 60.00th=[ 426], | 70.00th=[ 451], 80.00th=[ 498], 90.00th=[ 542], 95.00th=[ 584], | 99.00th=[ 676], 99.50th=[ 701], 99.90th=[ 835], 99.95th=[ 894], | 99.99th=[ 894] bw ( KiB/s): min= 7, max= 110, per=24.59%, avg=45.98, stdev=21.20, samples=212 iops : min= 1, max= 27, avg=11.01, stdev= 5.37, samples=212 write: IOPS=12, BW=49.3KiB/s (50.5kB/s)(5264KiB/106804msec); 0 zone resets slat (usec): min=29, max=7202, avg=192.27, stdev=487.41 clat (msec): min=140, max=1690, avg=898.97, stdev=235.12 lat (msec): min=140, max=1690, avg=899.16, stdev=235.11 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 518], 10.00th=[ 584], 20.00th=[ 693], | 30.00th=[ 776], 40.00th=[ 844], 50.00th=[ 911], 60.00th=[ 969], | 70.00th=[ 1028], 80.00th=[ 1099], 90.00th=[ 1183], 95.00th=[ 1267], | 99.00th=[ 1469], 99.50th=[ 1502], 99.90th=[ 1670], 99.95th=[ 1687], | 99.99th=[ 1687] bw ( KiB/s): min= 7, max= 111, per=24.93%, avg=48.62, stdev=20.38, samples=212 iops : min= 1, max= 27, avg=11.67, stdev= 5.16, samples=212 lat (msec) : 20=0.04%, 100=0.31%, 250=1.48%, 500=39.38%, 750=20.74% lat (msec) : 1000=20.43%, 2000=17.62% cpu : usr=0.08%, sys=0.54%, ctx=4507, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145610: Mon Aug 24 11:22:31 2026 read: IOPS=11, BW=46.7KiB/s (47.8kB/s)(4984KiB/106807msec) slat (usec): min=2608, max=73609, avg=13683.64, stdev=7879.12 clat (msec): min=9, max=806, avg=404.66, stdev=107.27 lat (msec): min=21, max=817, avg=418.34, stdev=108.34 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 255], 10.00th=[ 284], 20.00th=[ 317], | 30.00th=[ 342], 40.00th=[ 372], 50.00th=[ 397], 60.00th=[ 426], | 70.00th=[ 456], 80.00th=[ 493], 90.00th=[ 542], 95.00th=[ 592], | 99.00th=[ 684], 99.50th=[ 709], 99.90th=[ 776], 99.95th=[ 810], | 99.99th=[ 810] bw ( KiB/s): min= 7, max= 126, per=24.72%, avg=46.22, stdev=21.45, samples=212 iops : min= 1, max= 31, avg=11.07, stdev= 5.46, samples=212 write: IOPS=12, BW=49.2KiB/s (50.4kB/s)(5256KiB/106807msec); 0 zone resets slat (usec): min=32, max=21664, avg=244.41, stdev=1107.00 clat (msec): min=126, max=1584, avg=901.64, stdev=235.05 lat (msec): min=126, max=1584, avg=901.88, stdev=235.11 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 510], 10.00th=[ 600], 20.00th=[ 701], | 30.00th=[ 776], 40.00th=[ 835], 50.00th=[ 902], 60.00th=[ 978], | 70.00th=[ 1036], 80.00th=[ 1099], 90.00th=[ 1200], 95.00th=[ 1267], | 99.00th=[ 1435], 99.50th=[ 1485], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 1586] bw ( KiB/s): min= 7, max= 116, per=24.80%, avg=48.36, stdev=19.72, samples=212 iops : min= 1, max= 29, avg=11.61, stdev= 4.95, samples=212 lat (msec) : 10=0.04%, 50=0.16%, 100=0.20%, 250=2.11%, 500=39.30% lat (msec) : 750=20.08%, 1000=19.57%, 2000=18.55% cpu : usr=0.09%, sys=0.54%, ctx=4291, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145611: Mon Aug 24 11:22:31 2026 read: IOPS=11, BW=47.4KiB/s (48.5kB/s)(5060KiB/106808msec) slat (usec): min=2425, max=56965, avg=13408.13, stdev=7538.66 clat (msec): min=19, max=835, avg=398.32, stdev=104.81 lat (msec): min=26, max=860, avg=411.73, stdev=105.82 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 257], 10.00th=[ 279], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 363], 50.00th=[ 388], 60.00th=[ 418], | 70.00th=[ 443], 80.00th=[ 481], 90.00th=[ 531], 95.00th=[ 575], | 99.00th=[ 693], 99.50th=[ 735], 99.90th=[ 827], 99.95th=[ 835], | 99.99th=[ 835] bw ( KiB/s): min= 7, max= 132, per=25.64%, avg=47.95, stdev=22.47, samples=207 iops : min= 1, max= 33, avg=11.51, stdev= 5.69, samples=207 write: IOPS=12, BW=48.5KiB/s (49.7kB/s)(5180KiB/106808msec); 0 zone resets slat (usec): min=30, max=9009, avg=199.92, stdev=533.30 clat (msec): min=199, max=1666, avg=914.85, stdev=232.16 lat (msec): min=199, max=1666, avg=915.05, stdev=232.15 clat percentiles (msec): | 1.00th=[ 372], 5.00th=[ 542], 10.00th=[ 625], 20.00th=[ 718], | 30.00th=[ 776], 40.00th=[ 852], 50.00th=[ 927], 60.00th=[ 978], | 70.00th=[ 1036], 80.00th=[ 1116], 90.00th=[ 1217], 95.00th=[ 1301], | 99.00th=[ 1452], 99.50th=[ 1552], 99.90th=[ 1620], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 103, per=24.52%, avg=47.81, stdev=18.64, samples=212 iops : min= 1, max= 25, avg=11.47, stdev= 4.68, samples=212 lat (msec) : 20=0.04%, 50=0.16%, 100=0.16%, 250=1.95%, 500=41.21% lat (msec) : 750=18.48%, 1000=20.00%, 2000=18.01% cpu : usr=0.08%, sys=0.51%, ctx=4069, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=188KiB/s (192kB/s), 46.6KiB/s-47.5KiB/s (47.7kB/s-48.6kB/s), io=19.6MiB (20.5MB), run=106368-106808msec WRITE: bw=196KiB/s (200kB/s), 48.5KiB/s-49.3KiB/s (49.7kB/s-50.5kB/s), io=20.4MiB (21.4MB), run=106368-106808msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145630: Mon Aug 24 11:22:40 2026 read: IOPS=1, BW=72.2MiB/s (75.7MB/s)(40.0MiB/554msec) slat (nsec): min=28577k, max=28577k, avg=28576741.00, stdev= 0.00 clat (nsec): min=504043k, max=504043k, avg=504043163.00, stdev= 0.00 lat (nsec): min=532624k, max=532624k, avg=532624090.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 506], 5.00th=[ 506], 10.00th=[ 506], 20.00th=[ 506], | 30.00th=[ 506], 40.00th=[ 506], 50.00th=[ 506], 60.00th=[ 506], | 70.00th=[ 506], 80.00th=[ 506], 90.00th=[ 506], 95.00th=[ 506], | 99.00th=[ 506], 99.50th=[ 506], 99.90th=[ 506], 99.95th=[ 506], | 99.99th=[ 506] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.35%, ctx=5, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=72.2MiB/s (75.7MB/s), 72.2MiB/s-72.2MiB/s (75.7MB/s-75.7MB/s), io=40.0MiB (41.9MB), run=554-554msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout