== sanity test 398c: run fio to test AIO ================= 00:12:55 (1790741575) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.825525 s, 50.8 MB/s osc.lustre-OST0000-osc-ffff94b902f6c800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146277: Wed Sep 30 00:16:12 2026 write: IOPS=13, BW=55.8KiB/s (57.1kB/s)(10.0MiB/183571msec); 0 zone resets slat (usec): min=30, max=33471, avg=310.91, stdev=1157.18 clat (msec): min=159, max=2488, avg=1145.90, stdev=340.61 lat (msec): min=159, max=2488, avg=1146.21, stdev=340.62 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 625], 10.00th=[ 810], 20.00th=[ 919], | 30.00th=[ 986], 40.00th=[ 1053], 50.00th=[ 1133], 60.00th=[ 1200], | 70.00th=[ 1284], 80.00th=[ 1418], 90.00th=[ 1569], 95.00th=[ 1703], | 99.00th=[ 2022], 99.50th=[ 2333], 99.90th=[ 2467], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 255, per=25.07%, avg=55.65, stdev=27.13, samples=361 iops : min= 1, max= 63, avg=13.39, stdev= 6.80, samples=361 lat (msec) : 250=1.91%, 500=1.84%, 750=3.79%, 1000=24.77%, 2000=66.37% lat (msec) : >=2000=1.33% cpu : usr=0.05%, sys=0.23%, ctx=1879, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146278: Wed Sep 30 00:16:12 2026 write: IOPS=13, BW=55.7KiB/s (57.0kB/s)(10.0MiB/183805msec); 0 zone resets slat (usec): min=29, max=14327, avg=315.96, stdev=1084.38 clat (msec): min=182, max=2465, avg=1147.72, stdev=333.99 lat (msec): min=188, max=2466, avg=1148.04, stdev=333.95 clat percentiles (msec): | 1.00th=[ 218], 5.00th=[ 651], 10.00th=[ 810], 20.00th=[ 919], | 30.00th=[ 995], 40.00th=[ 1053], 50.00th=[ 1116], 60.00th=[ 1200], | 70.00th=[ 1284], 80.00th=[ 1385], 90.00th=[ 1569], 95.00th=[ 1703], | 99.00th=[ 1989], 99.50th=[ 2232], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 255, per=26.56%, avg=58.95, stdev=31.74, samples=341 iops : min= 1, max= 63, avg=14.21, stdev= 7.97, samples=341 lat (msec) : 250=1.91%, 500=1.84%, 750=3.83%, 1000=22.97%, 2000=68.52% lat (msec) : >=2000=0.94% cpu : usr=0.05%, sys=0.24%, ctx=1690, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146279: Wed Sep 30 00:16:12 2026 write: IOPS=13, BW=55.7KiB/s (57.0kB/s)(10.0MiB/183809msec); 0 zone resets slat (usec): min=28, max=21777, avg=335.18, stdev=1170.96 clat (msec): min=97, max=2464, avg=1148.01, stdev=331.37 lat (msec): min=108, max=2464, avg=1148.35, stdev=331.28 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 651], 10.00th=[ 810], 20.00th=[ 936], | 30.00th=[ 995], 40.00th=[ 1053], 50.00th=[ 1133], 60.00th=[ 1200], | 70.00th=[ 1284], 80.00th=[ 1401], 90.00th=[ 1552], 95.00th=[ 1653], | 99.00th=[ 2123], 99.50th=[ 2333], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 255, per=25.13%, avg=55.80, stdev=29.71, samples=360 iops : min= 1, max= 63, avg=13.42, stdev= 7.44, samples=360 lat (msec) : 100=0.04%, 250=1.41%, 500=1.84%, 750=4.10%, 1000=23.36% lat (msec) : 2000=67.77%, >=2000=1.48% cpu : usr=0.05%, sys=0.28%, ctx=1978, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146280: Wed Sep 30 00:16:12 2026 write: IOPS=13, BW=55.8KiB/s (57.1kB/s)(10.0MiB/183635msec); 0 zone resets slat (usec): min=30, max=19515, avg=320.35, stdev=1148.27 clat (msec): min=76, max=2520, avg=1146.87, stdev=339.41 lat (msec): min=77, max=2520, avg=1147.19, stdev=339.34 clat percentiles (msec): | 1.00th=[ 209], 5.00th=[ 617], 10.00th=[ 793], 20.00th=[ 919], | 30.00th=[ 995], 40.00th=[ 1053], 50.00th=[ 1133], 60.00th=[ 1200], | 70.00th=[ 1284], 80.00th=[ 1385], 90.00th=[ 1552], 95.00th=[ 1687], | 99.00th=[ 2198], 99.50th=[ 2366], 99.90th=[ 2500], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 263, per=25.56%, avg=56.74, stdev=30.77, samples=354 iops : min= 1, max= 65, avg=13.66, stdev= 7.68, samples=354 lat (msec) : 100=0.04%, 250=2.19%, 500=1.33%, 750=3.67%, 1000=23.71% lat (msec) : 2000=67.54%, >=2000=1.52% cpu : usr=0.05%, sys=0.24%, ctx=1881, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=223KiB/s (228kB/s), 55.7KiB/s-55.8KiB/s (57.0kB/s-57.1kB/s), io=40.0MiB (41.9MB), run=183571-183809msec 3% of I/O are 1-page (PAGE_SIZE=4096), max 50% mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146318: Wed Sep 30 00:18:25 2026 read: IOPS=10, BW=40.0KiB/s (41.0kB/s)(5048KiB/123187msec) slat (usec): min=3529, max=62502, avg=16646.02, stdev=8906.71 clat (msec): min=87, max=1085, avg=480.68, stdev=124.54 lat (msec): min=102, max=1121, avg=497.33, stdev=126.41 clat percentiles (msec): | 1.00th=[ 255], 5.00th=[ 317], 10.00th=[ 347], 20.00th=[ 380], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 464], 60.00th=[ 489], | 70.00th=[ 531], 80.00th=[ 575], 90.00th=[ 634], 95.00th=[ 709], | 99.00th=[ 844], 99.50th=[ 894], 99.90th=[ 1083], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 135, per=25.43%, avg=41.20, stdev=20.52, samples=240 iops : min= 1, max= 33, avg= 9.77, stdev= 5.18, samples=240 write: IOPS=10, BW=42.1KiB/s (43.2kB/s)(5192KiB/123187msec); 0 zone resets slat (usec): min=34, max=28756, avg=381.33, stdev=1421.60 clat (msec): min=163, max=2228, avg=1030.95, stdev=282.15 lat (msec): min=163, max=2228, avg=1031.33, stdev=282.21 clat percentiles (msec): | 1.00th=[ 493], 5.00th=[ 617], 10.00th=[ 693], 20.00th=[ 810], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 1003], 60.00th=[ 1062], | 70.00th=[ 1150], 80.00th=[ 1250], 90.00th=[ 1385], 95.00th=[ 1519], | 99.00th=[ 1938], 99.50th=[ 2039], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 95, per=24.63%, avg=41.63, stdev=18.15, samples=243 iops : min= 1, max= 23, avg= 9.87, stdev= 4.55, samples=243 lat (msec) : 100=0.08%, 250=0.39%, 500=31.21%, 750=23.12%, 1000=19.53% lat (msec) : 2000=25.39%, >=2000=0.27% cpu : usr=0.11%, sys=0.72%, ctx=4856, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146319: Wed Sep 30 00:18:25 2026 read: IOPS=10, BW=40.4KiB/s (41.4kB/s)(4976KiB/123184msec) slat (usec): min=3480, max=65350, avg=16500.60, stdev=8604.85 clat (msec): min=32, max=1108, avg=477.72, stdev=129.81 lat (msec): min=60, max=1128, avg=494.23, stdev=131.21 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 309], 10.00th=[ 342], 20.00th=[ 376], | 30.00th=[ 405], 40.00th=[ 430], 50.00th=[ 460], 60.00th=[ 485], | 70.00th=[ 531], 80.00th=[ 567], 90.00th=[ 642], 95.00th=[ 718], | 99.00th=[ 860], 99.50th=[ 919], 99.90th=[ 1099], 99.95th=[ 1116], | 99.99th=[ 1116] bw ( KiB/s): min= 7, max= 88, per=25.04%, avg=40.57, stdev=19.26, samples=240 iops : min= 1, max= 22, avg= 9.61, stdev= 4.85, samples=240 write: IOPS=10, BW=42.7KiB/s (43.8kB/s)(5264KiB/123184msec); 0 zone resets slat (usec): min=30, max=17447, avg=333.00, stdev=1069.12 clat (msec): min=277, max=2273, avg=1027.34, stdev=285.69 lat (msec): min=277, max=2273, avg=1027.68, stdev=285.71 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 600], 10.00th=[ 684], 20.00th=[ 802], | 30.00th=[ 877], 40.00th=[ 944], 50.00th=[ 1003], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1250], 90.00th=[ 1385], 95.00th=[ 1485], | 99.00th=[ 1871], 99.50th=[ 2089], 99.90th=[ 2232], 99.95th=[ 2265], | 99.99th=[ 2265] bw ( KiB/s): min= 7, max= 96, per=24.86%, avg=42.02, stdev=18.82, samples=244 iops : min= 1, max= 24, avg= 9.97, stdev= 4.74, samples=244 lat (msec) : 50=0.04%, 100=0.23%, 250=0.47%, 500=31.02%, 750=22.46% lat (msec) : 1000=19.61%, 2000=25.82%, >=2000=0.35% cpu : usr=0.13%, sys=0.64%, ctx=4499, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146320: Wed Sep 30 00:18:25 2026 read: IOPS=10, BW=40.5KiB/s (41.4kB/s)(4984KiB/123136msec) slat (usec): min=33, max=63786, avg=16794.57, stdev=8960.94 clat (msec): min=11, max=1541, avg=471.47, stdev=138.29 lat (msec): min=21, max=1570, avg=488.27, stdev=139.92 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 292], 10.00th=[ 330], 20.00th=[ 372], | 30.00th=[ 397], 40.00th=[ 430], 50.00th=[ 460], 60.00th=[ 485], | 70.00th=[ 523], 80.00th=[ 567], 90.00th=[ 642], 95.00th=[ 709], | 99.00th=[ 835], 99.50th=[ 902], 99.90th=[ 1053], 99.95th=[ 1536], | 99.99th=[ 1536] bw ( KiB/s): min= 7, max= 120, per=24.70%, avg=40.01, stdev=21.76, samples=244 iops : min= 1, max= 30, avg= 9.47, stdev= 5.51, samples=244 write: IOPS=10, BW=42.7KiB/s (43.7kB/s)(5256KiB/123136msec); 0 zone resets slat (usec): min=34, max=31913, avg=414.90, stdev=1691.57 clat (msec): min=212, max=2290, avg=1034.86, stdev=283.61 lat (msec): min=213, max=2290, avg=1035.28, stdev=283.70 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 617], 10.00th=[ 693], 20.00th=[ 802], | 30.00th=[ 885], 40.00th=[ 944], 50.00th=[ 1020], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1385], 95.00th=[ 1502], | 99.00th=[ 1854], 99.50th=[ 1938], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 104, per=25.10%, avg=42.43, stdev=18.78, samples=241 iops : min= 1, max= 26, avg=10.07, stdev= 4.71, samples=241 lat (msec) : 20=0.08%, 50=0.12%, 100=0.39%, 250=1.09%, 500=30.90% lat (msec) : 750=21.25%, 1000=19.34%, 2000=26.68%, >=2000=0.16% cpu : usr=0.12%, sys=0.68%, ctx=4510, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146321: Wed Sep 30 00:18:25 2026 read: IOPS=10, BW=41.1KiB/s (42.1kB/s)(5060KiB/123059msec) slat (usec): min=3027, max=81185, avg=16692.77, stdev=9337.11 clat (msec): min=46, max=1446, avg=483.40, stdev=134.35 lat (msec): min=57, max=1455, avg=500.10, stdev=135.79 clat percentiles (msec): | 1.00th=[ 245], 5.00th=[ 305], 10.00th=[ 347], 20.00th=[ 384], | 30.00th=[ 409], 40.00th=[ 439], 50.00th=[ 460], 60.00th=[ 493], | 70.00th=[ 531], 80.00th=[ 575], 90.00th=[ 642], 95.00th=[ 718], | 99.00th=[ 894], 99.50th=[ 1020], 99.90th=[ 1351], 99.95th=[ 1452], | 99.99th=[ 1452] bw ( KiB/s): min= 7, max= 104, per=25.60%, avg=41.47, stdev=19.49, samples=239 iops : min= 1, max= 26, avg= 9.83, stdev= 4.91, samples=239 write: IOPS=10, BW=42.1KiB/s (43.1kB/s)(5180KiB/123059msec); 0 zone resets slat (usec): min=39, max=18725, avg=362.79, stdev=1102.55 clat (msec): min=344, max=2203, avg=1027.82, stdev=286.73 lat (msec): min=344, max=2218, avg=1028.19, stdev=286.77 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 609], 10.00th=[ 693], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 936], 50.00th=[ 995], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1250], 90.00th=[ 1385], 95.00th=[ 1485], | 99.00th=[ 2039], 99.50th=[ 2165], 99.90th=[ 2198], 99.95th=[ 2198], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 96, per=24.57%, avg=41.52, stdev=18.75, samples=243 iops : min= 1, max= 24, avg= 9.84, stdev= 4.71, samples=243 lat (msec) : 50=0.08%, 100=0.12%, 250=0.39%, 500=31.17%, 750=22.89% lat (msec) : 1000=20.16%, 2000=24.65%, >=2000=0.55% cpu : usr=0.14%, sys=0.72%, ctx=4867, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=163KiB/s (167kB/s), 40.4KiB/s-41.1KiB/s (41.4kB/s-42.1kB/s), io=19.6MiB (20.5MB), run=123059-123187msec WRITE: bw=170KiB/s (174kB/s), 42.1KiB/s-42.7KiB/s (43.1kB/s-43.8kB/s), io=20.4MiB (21.4MB), run=123059-123187msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146343: Wed Sep 30 00:18:38 2026 read: IOPS=1, BW=62.4MiB/s (65.4MB/s)(40.0MiB/641msec) slat (nsec): min=27358k, max=27358k, avg=27358212.00, stdev= 0.00 clat (nsec): min=594233k, max=594233k, avg=594233306.00, stdev= 0.00 lat (nsec): min=621596k, max=621596k, avg=621596230.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 592], 5.00th=[ 592], 10.00th=[ 592], 20.00th=[ 592], | 30.00th=[ 592], 40.00th=[ 592], 50.00th=[ 592], 60.00th=[ 592], | 70.00th=[ 592], 80.00th=[ 592], 90.00th=[ 592], 95.00th=[ 592], | 99.00th=[ 592], 99.50th=[ 592], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 592] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=2.34%, ctx=40, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=62.4MiB/s (65.4MB/s), 62.4MiB/s-62.4MiB/s (65.4MB/s-65.4MB/s), io=40.0MiB (41.9MB), run=641-641msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout