== sanity test 398c: run fio to test AIO ================= 18:27:49 (1787437669) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.67829 s, 61.8 MB/s osc.lustre-OST0000-osc-ffff91270fa61000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145327: Sat Aug 22 18:30:44 2026 write: IOPS=15, BW=62.2KiB/s (63.7kB/s)(10.0MiB/164632msec); 0 zone resets slat (usec): min=28, max=17006, avg=322.06, stdev=1054.66 clat (msec): min=180, max=2285, avg=1028.24, stdev=288.54 lat (msec): min=180, max=2285, avg=1028.57, stdev=288.57 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 575], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 961], 50.00th=[ 1020], 60.00th=[ 1083], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1401], 95.00th=[ 1502], | 99.00th=[ 1720], 99.50th=[ 1938], 99.90th=[ 2265], 99.95th=[ 2299], | 99.99th=[ 2299] bw ( KiB/s): min= 7, max= 254, per=25.58%, avg=63.43, stdev=35.04, samples=317 iops : min= 1, max= 63, avg=15.34, stdev= 8.80, samples=317 lat (msec) : 250=0.62%, 500=3.20%, 750=9.57%, 1000=32.54%, 2000=53.63% lat (msec) : >=2000=0.43% cpu : usr=0.08%, sys=0.27%, ctx=1953, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145328: Sat Aug 22 18:30:44 2026 write: IOPS=15, BW=62.1KiB/s (63.6kB/s)(10.0MiB/164818msec); 0 zone resets slat (usec): min=29, max=18021, avg=333.40, stdev=965.26 clat (msec): min=157, max=2203, avg=1028.94, stdev=289.87 lat (msec): min=157, max=2203, avg=1029.28, stdev=289.89 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 584], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 894], 40.00th=[ 953], 50.00th=[ 1020], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1234], 90.00th=[ 1368], 95.00th=[ 1536], | 99.00th=[ 1804], 99.50th=[ 1938], 99.90th=[ 2123], 99.95th=[ 2123], | 99.99th=[ 2198] bw ( KiB/s): min= 7, max= 184, per=25.43%, avg=63.06, stdev=30.23, samples=319 iops : min= 1, max= 46, avg=15.25, stdev= 7.57, samples=319 lat (msec) : 250=0.47%, 500=2.93%, 750=10.27%, 1000=33.48%, 2000=52.38% lat (msec) : >=2000=0.47% cpu : usr=0.05%, sys=0.30%, ctx=2053, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145329: Sat Aug 22 18:30:44 2026 write: IOPS=15, BW=62.1KiB/s (63.6kB/s)(10.0MiB/164789msec); 0 zone resets slat (usec): min=28, max=17505, avg=274.29, stdev=831.98 clat (msec): min=243, max=2128, avg=1028.50, stdev=283.30 lat (msec): min=243, max=2128, avg=1028.78, stdev=283.32 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 609], 10.00th=[ 709], 20.00th=[ 802], | 30.00th=[ 877], 40.00th=[ 961], 50.00th=[ 1020], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1368], 95.00th=[ 1519], | 99.00th=[ 1720], 99.50th=[ 1854], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 222, per=25.17%, avg=62.42, stdev=32.19, samples=322 iops : min= 1, max= 55, avg=15.08, stdev= 8.08, samples=322 lat (msec) : 250=0.62%, 500=2.66%, 750=10.70%, 1000=32.58%, 2000=53.32% lat (msec) : >=2000=0.12% cpu : usr=0.05%, sys=0.25%, ctx=1722, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145330: Sat Aug 22 18:30:44 2026 write: IOPS=15, BW=62.1KiB/s (63.6kB/s)(10.0MiB/164850msec); 0 zone resets slat (usec): min=23, max=14360, avg=322.52, stdev=984.45 clat (msec): min=255, max=2048, avg=1028.93, stdev=284.61 lat (msec): min=255, max=2048, avg=1029.26, stdev=284.63 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 584], 10.00th=[ 709], 20.00th=[ 810], | 30.00th=[ 885], 40.00th=[ 969], 50.00th=[ 1028], 60.00th=[ 1083], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1368], 95.00th=[ 1519], | 99.00th=[ 1804], 99.50th=[ 1921], 99.90th=[ 2022], 99.95th=[ 2022], | 99.99th=[ 2056] bw ( KiB/s): min= 7, max= 256, per=24.95%, avg=61.88, stdev=31.59, samples=325 iops : min= 1, max= 64, avg=14.95, stdev= 7.90, samples=325 lat (msec) : 500=3.40%, 750=9.84%, 1000=32.73%, 2000=53.83%, >=2000=0.20% cpu : usr=0.05%, sys=0.26%, ctx=1788, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=248KiB/s (254kB/s), 62.1KiB/s-62.2KiB/s (63.6kB/s-63.7kB/s), io=40.0MiB (41.9MB), run=164632-164850msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145364: Sat Aug 22 18:32:59 2026 read: IOPS=9, BW=39.1KiB/s (40.1kB/s)(5048KiB/128944msec) slat (usec): min=3676, max=80117, avg=15671.05, stdev=9347.53 clat (msec): min=13, max=1217, avg=489.47, stdev=173.25 lat (msec): min=32, max=1276, avg=505.15, stdev=176.69 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 279], 10.00th=[ 317], 20.00th=[ 351], | 30.00th=[ 388], 40.00th=[ 422], 50.00th=[ 456], 60.00th=[ 506], | 70.00th=[ 550], 80.00th=[ 600], 90.00th=[ 709], 95.00th=[ 818], | 99.00th=[ 1083], 99.50th=[ 1150], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 103, per=25.52%, avg=39.56, stdev=20.86, samples=249 iops : min= 1, max= 25, avg= 9.34, stdev= 5.28, samples=249 write: IOPS=10, BW=40.3KiB/s (41.2kB/s)(5192KiB/128944msec); 0 zone resets slat (usec): min=29, max=17341, avg=410.06, stdev=1124.27 clat (msec): min=278, max=3045, avg=1095.44, stdev=474.20 lat (msec): min=279, max=3045, avg=1095.86, stdev=474.25 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 550], 10.00th=[ 634], 20.00th=[ 726], | 30.00th=[ 827], 40.00th=[ 911], 50.00th=[ 986], 60.00th=[ 1070], | 70.00th=[ 1200], 80.00th=[ 1401], 90.00th=[ 1687], 95.00th=[ 2089], | 99.00th=[ 2702], 99.50th=[ 2836], 99.90th=[ 3004], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 7, max= 88, per=24.44%, avg=39.60, stdev=20.37, samples=255 iops : min= 1, max= 22, avg= 9.35, stdev= 5.16, samples=255 lat (msec) : 20=0.04%, 50=0.20%, 100=0.04%, 250=1.02%, 500=29.53% lat (msec) : 750=26.25%, 1000=17.77%, 2000=22.07%, >=2000=3.09% cpu : usr=0.09%, sys=0.63%, ctx=4336, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145365: Sat Aug 22 18:32:59 2026 read: IOPS=9, BW=38.6KiB/s (39.5kB/s)(4976KiB/128865msec) slat (usec): min=50, max=70842, avg=15842.77, stdev=9247.78 clat (msec): min=18, max=1217, avg=499.87, stdev=186.39 lat (msec): min=18, max=1245, avg=515.73, stdev=189.65 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 275], 10.00th=[ 309], 20.00th=[ 355], | 30.00th=[ 388], 40.00th=[ 418], 50.00th=[ 456], 60.00th=[ 502], | 70.00th=[ 558], 80.00th=[ 634], 90.00th=[ 743], 95.00th=[ 877], | 99.00th=[ 1133], 99.50th=[ 1183], 99.90th=[ 1217], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 112, per=24.95%, avg=38.68, stdev=21.47, samples=251 iops : min= 1, max= 28, avg= 9.12, stdev= 5.42, samples=251 write: IOPS=10, BW=40.8KiB/s (41.8kB/s)(5264KiB/128865msec); 0 zone resets slat (usec): min=30, max=21556, avg=381.22, stdev=1234.04 clat (msec): min=111, max=2971, avg=1078.08, stdev=465.42 lat (msec): min=111, max=2971, avg=1078.47, stdev=465.48 clat percentiles (msec): | 1.00th=[ 309], 5.00th=[ 542], 10.00th=[ 617], 20.00th=[ 718], | 30.00th=[ 810], 40.00th=[ 894], 50.00th=[ 969], 60.00th=[ 1062], | 70.00th=[ 1200], 80.00th=[ 1385], 90.00th=[ 1653], 95.00th=[ 2056], | 99.00th=[ 2668], 99.50th=[ 2769], 99.90th=[ 2937], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 7, max= 104, per=25.37%, avg=41.10, stdev=22.13, samples=249 iops : min= 1, max= 26, avg= 9.73, stdev= 5.59, samples=249 lat (msec) : 20=0.08%, 50=0.04%, 100=0.12%, 250=1.29%, 500=29.41% lat (msec) : 750=24.80%, 1000=19.18%, 2000=21.95%, >=2000=3.12% cpu : usr=0.11%, sys=0.62%, ctx=4396, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145366: Sat Aug 22 18:32:59 2026 read: IOPS=9, BW=38.7KiB/s (39.6kB/s)(4984KiB/128861msec) slat (usec): min=3315, max=79634, avg=16046.15, stdev=9906.26 clat (msec): min=44, max=1237, avg=506.20, stdev=191.46 lat (msec): min=66, max=1258, avg=522.26, stdev=194.64 clat percentiles (msec): | 1.00th=[ 190], 5.00th=[ 275], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 384], 40.00th=[ 422], 50.00th=[ 460], 60.00th=[ 518], | 70.00th=[ 584], 80.00th=[ 651], 90.00th=[ 768], 95.00th=[ 885], | 99.00th=[ 1099], 99.50th=[ 1167], 99.90th=[ 1234], 99.95th=[ 1234], | 99.99th=[ 1234] bw ( KiB/s): min= 7, max= 103, per=24.96%, avg=38.69, stdev=20.39, samples=252 iops : min= 1, max= 25, avg= 9.12, stdev= 5.12, samples=252 write: IOPS=10, BW=40.8KiB/s (41.8kB/s)(5256KiB/128861msec); 0 zone resets slat (usec): min=34, max=19529, avg=367.30, stdev=970.33 clat (msec): min=196, max=3045, avg=1071.49, stdev=464.23 lat (msec): min=196, max=3046, avg=1071.85, stdev=464.22 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 542], 10.00th=[ 634], 20.00th=[ 718], | 30.00th=[ 793], 40.00th=[ 885], 50.00th=[ 961], 60.00th=[ 1053], | 70.00th=[ 1183], 80.00th=[ 1351], 90.00th=[ 1636], 95.00th=[ 2089], | 99.00th=[ 2702], 99.50th=[ 2769], 99.90th=[ 2970], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min= 7, max= 111, per=25.08%, avg=40.64, stdev=21.31, samples=251 iops : min= 1, max= 27, avg= 9.60, stdev= 5.40, samples=251 lat (msec) : 50=0.04%, 100=0.16%, 250=1.25%, 500=28.32%, 750=26.17% lat (msec) : 1000=19.26%, 2000=21.80%, >=2000=3.01% cpu : usr=0.09%, sys=0.60%, ctx=4627, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145367: Sat Aug 22 18:32:59 2026 read: IOPS=9, BW=39.3KiB/s (40.3kB/s)(5060KiB/128701msec) slat (usec): min=2935, max=89037, avg=16173.45, stdev=10183.08 clat (msec): min=24, max=1246, avg=498.61, stdev=187.26 lat (msec): min=46, max=1283, avg=514.80, stdev=190.85 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 275], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 418], 50.00th=[ 460], 60.00th=[ 510], | 70.00th=[ 567], 80.00th=[ 625], 90.00th=[ 751], 95.00th=[ 877], | 99.00th=[ 1099], 99.50th=[ 1167], 99.90th=[ 1217], 99.95th=[ 1250], | 99.99th=[ 1250] bw ( KiB/s): min= 7, max= 136, per=25.69%, avg=39.82, stdev=21.42, samples=248 iops : min= 1, max= 34, avg= 9.40, stdev= 5.40, samples=248 write: IOPS=10, BW=40.2KiB/s (41.2kB/s)(5180KiB/128701msec); 0 zone resets slat (usec): min=32, max=18068, avg=359.81, stdev=1032.24 clat (msec): min=194, max=3013, avg=1084.97, stdev=469.31 lat (msec): min=195, max=3013, avg=1085.33, stdev=469.39 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 735], | 30.00th=[ 810], 40.00th=[ 894], 50.00th=[ 969], 60.00th=[ 1062], | 70.00th=[ 1200], 80.00th=[ 1368], 90.00th=[ 1670], 95.00th=[ 2106], | 99.00th=[ 2836], 99.50th=[ 2869], 99.90th=[ 2937], 99.95th=[ 3004], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 112, per=25.11%, avg=40.68, stdev=19.96, samples=248 iops : min= 1, max= 28, avg= 9.61, stdev= 5.07, samples=248 lat (msec) : 50=0.08%, 100=0.23%, 250=1.17%, 500=28.71%, 750=25.20% lat (msec) : 1000=19.88%, 2000=21.64%, >=2000=3.09% cpu : usr=0.10%, sys=0.61%, ctx=4382, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=156KiB/s (159kB/s), 38.6KiB/s-39.3KiB/s (39.5kB/s-40.3kB/s), io=19.6MiB (20.5MB), run=128701-128944msec WRITE: bw=162KiB/s (166kB/s), 40.2KiB/s-40.8KiB/s (41.2kB/s-41.8kB/s), io=20.4MiB (21.4MB), run=128701-128944msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145390: Sat Aug 22 18:33:09 2026 read: IOPS=2, BW=80.2MiB/s (84.1MB/s)(40.0MiB/499msec) slat (nsec): min=71010k, max=71010k, avg=71010369.00, stdev= 0.00 clat (nsec): min=412664k, max=412664k, avg=412664319.00, stdev= 0.00 lat (nsec): min=483680k, max=483680k, avg=483680204.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 414], 5.00th=[ 414], 10.00th=[ 414], 20.00th=[ 414], | 30.00th=[ 414], 40.00th=[ 414], 50.00th=[ 414], 60.00th=[ 414], | 70.00th=[ 414], 80.00th=[ 414], 90.00th=[ 414], 95.00th=[ 414], | 99.00th=[ 414], 99.50th=[ 414], 99.90th=[ 414], 99.95th=[ 414], | 99.99th=[ 414] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=1.61%, ctx=7, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=80.2MiB/s (84.1MB/s), 80.2MiB/s-80.2MiB/s (84.1MB/s-84.1MB/s), io=40.0MiB (41.9MB), run=499-499msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout