== sanity test 398c: run fio to test AIO ================= 09:58:02 (1761314282) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.362292 s, 116 MB/s osc.lustre-OST0000-osc-ffff9a2e075d6800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=123240: Fri Oct 24 10:00:39 2025 write: IOPS=17, BW=69.5KiB/s (71.1kB/s)(10.0MiB/147377msec); 0 zone resets slat (usec): min=25, max=11085, avg=249.73, stdev=705.23 clat (msec): min=312, max=1650, avg=920.45, stdev=207.94 lat (msec): min=312, max=1650, avg=920.70, stdev=207.94 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 743], | 30.00th=[ 818], 40.00th=[ 869], 50.00th=[ 911], 60.00th=[ 961], | 70.00th=[ 1028], 80.00th=[ 1099], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1401], 99.50th=[ 1435], 99.90th=[ 1636], 99.95th=[ 1653], | 99.99th=[ 1653] bw ( KiB/s): min= 7, max= 128, per=25.51%, avg=70.67, stdev=28.95, samples=285 iops : min= 1, max= 32, avg=17.13, stdev= 7.26, samples=285 lat (msec) : 500=1.13%, 750=19.77%, 1000=45.27%, 2000=33.83% cpu : usr=0.06%, sys=0.30%, ctx=1459, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123241: Fri Oct 24 10:00:39 2025 write: IOPS=17, BW=69.8KiB/s (71.4kB/s)(10.0MiB/146807msec); 0 zone resets slat (usec): min=26, max=13773, avg=287.94, stdev=853.95 clat (msec): min=90, max=1670, avg=916.89, stdev=216.03 lat (msec): min=90, max=1670, avg=917.18, stdev=216.04 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 584], 10.00th=[ 642], 20.00th=[ 743], | 30.00th=[ 810], 40.00th=[ 860], 50.00th=[ 902], 60.00th=[ 953], | 70.00th=[ 1011], 80.00th=[ 1099], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1502], 99.50th=[ 1586], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 159, per=24.97%, avg=69.17, stdev=31.65, samples=291 iops : min= 1, max= 39, avg=16.76, stdev= 7.90, samples=291 lat (msec) : 100=0.04%, 250=0.12%, 500=1.80%, 750=19.65%, 1000=46.02% lat (msec) : 2000=32.38% cpu : usr=0.05%, sys=0.32%, ctx=1597, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123242: Fri Oct 24 10:00:39 2025 write: IOPS=17, BW=69.5KiB/s (71.1kB/s)(10.0MiB/147379msec); 0 zone resets slat (usec): min=25, max=13625, avg=249.00, stdev=711.47 clat (msec): min=68, max=1663, avg=920.15, stdev=209.13 lat (msec): min=68, max=1663, avg=920.40, stdev=209.14 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 584], 10.00th=[ 651], 20.00th=[ 751], | 30.00th=[ 810], 40.00th=[ 860], 50.00th=[ 902], 60.00th=[ 961], | 70.00th=[ 1020], 80.00th=[ 1099], 90.00th=[ 1200], 95.00th=[ 1267], | 99.00th=[ 1519], 99.50th=[ 1552], 99.90th=[ 1620], 99.95th=[ 1636], | 99.99th=[ 1670] bw ( KiB/s): min= 8, max= 128, per=25.07%, avg=69.44, stdev=25.88, samples=290 iops : min= 2, max= 32, avg=16.83, stdev= 6.48, samples=290 lat (msec) : 100=0.04%, 500=1.21%, 750=18.83%, 1000=46.76%, 2000=33.16% cpu : usr=0.05%, sys=0.31%, ctx=1520, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=123243: Fri Oct 24 10:00:39 2025 write: IOPS=17, BW=69.5KiB/s (71.2kB/s)(10.0MiB/147304msec); 0 zone resets slat (usec): min=25, max=13626, avg=259.31, stdev=796.99 clat (msec): min=246, max=1662, avg=919.76, stdev=206.85 lat (msec): min=246, max=1662, avg=920.03, stdev=206.87 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 592], 10.00th=[ 651], 20.00th=[ 751], | 30.00th=[ 818], 40.00th=[ 869], 50.00th=[ 911], 60.00th=[ 953], | 70.00th=[ 1020], 80.00th=[ 1099], 90.00th=[ 1200], 95.00th=[ 1284], | 99.00th=[ 1435], 99.50th=[ 1485], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 1670] bw ( KiB/s): min= 7, max= 128, per=25.16%, avg=69.69, stdev=27.92, samples=289 iops : min= 1, max= 32, avg=16.89, stdev= 6.98, samples=289 lat (msec) : 250=0.04%, 500=1.48%, 750=19.14%, 1000=47.38%, 2000=31.95% cpu : usr=0.05%, sys=0.32%, ctx=1543, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=278KiB/s (285kB/s), 69.5KiB/s-69.8KiB/s (71.1kB/s-71.4kB/s), io=40.0MiB (41.9MB), run=146807-147379msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=123274: Fri Oct 24 10:02:22 2025 read: IOPS=13, BW=52.9KiB/s (54.2kB/s)(5048KiB/95433msec) slat (usec): min=2409, max=41732, avg=11073.16, stdev=5156.07 clat (msec): min=15, max=738, avg=387.56, stdev=100.13 lat (msec): min=22, max=748, avg=398.64, stdev=100.82 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 247], 10.00th=[ 275], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 359], 50.00th=[ 376], 60.00th=[ 401], | 70.00th=[ 430], 80.00th=[ 464], 90.00th=[ 510], 95.00th=[ 575], | 99.00th=[ 651], 99.50th=[ 693], 99.90th=[ 735], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min= 7, max= 127, per=24.97%, avg=52.19, stdev=23.40, samples=189 iops : min= 1, max= 31, avg=12.49, stdev= 5.93, samples=189 write: IOPS=13, BW=54.4KiB/s (55.7kB/s)(5192KiB/95433msec); 0 zone resets slat (usec): min=31, max=11520, avg=320.22, stdev=867.02 clat (msec): min=232, max=1444, avg=787.12, stdev=209.65 lat (msec): min=234, max=1445, avg=787.45, stdev=209.65 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 422], 10.00th=[ 506], 20.00th=[ 625], | 30.00th=[ 693], 40.00th=[ 743], 50.00th=[ 793], 60.00th=[ 835], | 70.00th=[ 894], 80.00th=[ 944], 90.00th=[ 1036], 95.00th=[ 1133], | 99.00th=[ 1351], 99.50th=[ 1368], 99.90th=[ 1435], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 7, max= 152, per=24.72%, avg=53.88, stdev=20.91, samples=189 iops : min= 1, max= 38, avg=12.91, stdev= 5.27, samples=189 lat (msec) : 20=0.08%, 50=0.20%, 100=0.27%, 250=2.34%, 500=45.43% lat (msec) : 750=22.54%, 1000=22.34%, 2000=6.80% cpu : usr=0.10%, sys=0.81%, ctx=3061, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123275: Fri Oct 24 10:02:22 2025 read: IOPS=13, BW=52.0KiB/s (53.3kB/s)(4976KiB/95627msec) slat (usec): min=1580, max=82154, avg=11368.40, stdev=5834.26 clat (msec): min=14, max=771, avg=388.99, stdev=96.37 lat (msec): min=27, max=788, avg=400.37, stdev=97.42 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 251], 10.00th=[ 284], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 359], 50.00th=[ 380], 60.00th=[ 401], | 70.00th=[ 430], 80.00th=[ 464], 90.00th=[ 514], 95.00th=[ 558], | 99.00th=[ 642], 99.50th=[ 701], 99.90th=[ 768], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 7, max= 120, per=24.62%, avg=51.46, stdev=21.45, samples=189 iops : min= 1, max= 30, avg=12.31, stdev= 5.41, samples=189 write: IOPS=13, BW=55.0KiB/s (56.4kB/s)(5264KiB/95627msec); 0 zone resets slat (usec): min=27, max=12580, avg=301.62, stdev=738.30 clat (msec): min=232, max=1556, avg=783.00, stdev=212.45 lat (msec): min=232, max=1556, avg=783.30, stdev=212.51 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 414], 10.00th=[ 489], 20.00th=[ 617], | 30.00th=[ 684], 40.00th=[ 735], 50.00th=[ 785], 60.00th=[ 835], | 70.00th=[ 885], 80.00th=[ 953], 90.00th=[ 1036], 95.00th=[ 1133], | 99.00th=[ 1318], 99.50th=[ 1385], 99.90th=[ 1552], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 15, max= 136, per=24.91%, avg=54.31, stdev=20.68, samples=190 iops : min= 3, max= 34, avg=13.02, stdev= 5.21, samples=190 lat (msec) : 20=0.08%, 50=0.04%, 100=0.23%, 250=2.15%, 500=46.25% lat (msec) : 750=21.88%, 1000=23.01%, 2000=6.37% cpu : usr=0.12%, sys=0.78%, ctx=3257, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123276: Fri Oct 24 10:02:22 2025 read: IOPS=13, BW=52.1KiB/s (53.4kB/s)(4984KiB/95624msec) slat (usec): min=2213, max=45175, avg=11272.11, stdev=5374.52 clat (msec): min=9, max=771, avg=392.39, stdev=101.41 lat (msec): min=17, max=788, avg=403.67, stdev=102.25 clat percentiles (msec): | 1.00th=[ 186], 5.00th=[ 251], 10.00th=[ 279], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 359], 50.00th=[ 380], 60.00th=[ 405], | 70.00th=[ 439], 80.00th=[ 468], 90.00th=[ 527], 95.00th=[ 575], | 99.00th=[ 667], 99.50th=[ 709], 99.90th=[ 768], 99.95th=[ 776], | 99.99th=[ 776] bw ( KiB/s): min= 7, max= 120, per=24.90%, avg=52.03, stdev=23.07, samples=188 iops : min= 1, max= 30, avg=12.45, stdev= 5.80, samples=188 write: IOPS=13, BW=54.0KiB/s (56.3kB/s)(5256KiB/95624msec); 0 zone resets slat (usec): min=26, max=11455, avg=304.72, stdev=726.35 clat (msec): min=266, max=1545, avg=779.99, stdev=207.90 lat (msec): min=266, max=1546, avg=780.30, stdev=207.90 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 409], 10.00th=[ 485], 20.00th=[ 617], | 30.00th=[ 684], 40.00th=[ 735], 50.00th=[ 785], 60.00th=[ 835], | 70.00th=[ 894], 80.00th=[ 944], 90.00th=[ 1028], 95.00th=[ 1116], | 99.00th=[ 1284], 99.50th=[ 1351], 99.90th=[ 1435], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 15, max= 152, per=24.91%, avg=54.31, stdev=20.66, samples=189 iops : min= 3, max= 38, avg=13.02, stdev= 5.20, samples=189 lat (msec) : 10=0.04%, 50=0.08%, 100=0.23%, 250=2.03%, 500=45.59% lat (msec) : 750=22.50%, 1000=22.77%, 2000=6.76% cpu : usr=0.11%, sys=0.80%, ctx=3014, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=123277: Fri Oct 24 10:02:22 2025 read: IOPS=13, BW=53.4KiB/s (54.7kB/s)(5060KiB/94778msec) slat (usec): min=34, max=42688, avg=11133.58, stdev=5325.45 clat (msec): min=34, max=844, avg=389.54, stdev=96.67 lat (msec): min=46, max=857, avg=400.68, stdev=97.51 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 245], 10.00th=[ 275], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 359], 50.00th=[ 384], 60.00th=[ 409], | 70.00th=[ 435], 80.00th=[ 468], 90.00th=[ 514], 95.00th=[ 558], | 99.00th=[ 642], 99.50th=[ 701], 99.90th=[ 827], 99.95th=[ 844], | 99.99th=[ 844] bw ( KiB/s): min= 7, max= 120, per=25.27%, avg=52.81, stdev=23.14, samples=188 iops : min= 1, max= 30, avg=12.64, stdev= 5.86, samples=188 write: IOPS=13, BW=54.7KiB/s (55.0kB/s)(5180KiB/94778msec); 0 zone resets slat (usec): min=25, max=10258, avg=307.26, stdev=712.93 clat (msec): min=108, max=1548, avg=778.41, stdev=220.60 lat (msec): min=109, max=1549, avg=778.73, stdev=220.62 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 414], 10.00th=[ 493], 20.00th=[ 600], | 30.00th=[ 676], 40.00th=[ 726], 50.00th=[ 776], 60.00th=[ 827], | 70.00th=[ 885], 80.00th=[ 953], 90.00th=[ 1045], 95.00th=[ 1167], | 99.00th=[ 1351], 99.50th=[ 1401], 99.90th=[ 1502], 99.95th=[ 1552], | 99.99th=[ 1552] bw ( KiB/s): min= 7, max= 144, per=24.81%, avg=54.09, stdev=21.44, samples=187 iops : min= 1, max= 36, avg=12.96, stdev= 5.38, samples=187 lat (msec) : 50=0.12%, 100=0.16%, 250=2.97%, 500=45.86%, 750=22.66% lat (msec) : 1000=21.25%, 2000=6.99% cpu : usr=0.11%, sys=0.75%, ctx=3079, majf=0, minf=16 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=210KiB/s (215kB/s), 52.0KiB/s-53.4KiB/s (53.3kB/s-54.7kB/s), io=19.6MiB (20.5MB), run=94778-95627msec WRITE: bw=218KiB/s (224kB/s), 54.4KiB/s-55.0KiB/s (55.7kB/s-56.4kB/s), io=20.4MiB (21.4MB), run=94778-95627msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=123310: Fri Oct 24 10:02:31 2025 read: IOPS=1, BW=74.5MiB/s (78.1MB/s)(40.0MiB/537msec) slat (nsec): min=62133k, max=62133k, avg=62133405.00, stdev= 0.00 clat (nsec): min=446176k, max=446176k, avg=446175945.00, stdev= 0.00 lat (nsec): min=508315k, max=508315k, avg=508314646.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 447], 10.00th=[ 447], 20.00th=[ 447], | 30.00th=[ 447], 40.00th=[ 447], 50.00th=[ 447], 60.00th=[ 447], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] lat (msec) : 500=100.00% cpu : usr=0.37%, sys=2.99%, ctx=4, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=74.5MiB/s (78.1MB/s), 74.5MiB/s-74.5MiB/s (78.1MB/s-78.1MB/s), io=40.0MiB (41.9MB), run=537-537msec debug=super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck