== sanity test 398c: run fio to test AIO ================= 03:47:19 (1787471239) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.536713 s, 78.1 MB/s osc.lustre-OST0000-osc-ffff8df22bf30000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145480: Sun Aug 23 03:50:34 2026 write: IOPS=13, BW=55.5KiB/s (56.8kB/s)(10.0MiB/184639msec); 0 zone resets slat (usec): min=35, max=18577, avg=310.64, stdev=1105.99 clat (msec): min=157, max=2787, avg=1153.27, stdev=364.25 lat (msec): min=157, max=2788, avg=1153.59, stdev=364.32 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 651], 10.00th=[ 743], 20.00th=[ 852], | 30.00th=[ 936], 40.00th=[ 1028], 50.00th=[ 1133], 60.00th=[ 1217], | 70.00th=[ 1318], 80.00th=[ 1435], 90.00th=[ 1620], 95.00th=[ 1804], | 99.00th=[ 2299], 99.50th=[ 2333], 99.90th=[ 2769], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 222, per=25.47%, avg=56.28, stdev=28.97, samples=357 iops : min= 1, max= 55, avg=13.59, stdev= 7.27, samples=357 lat (msec) : 250=0.62%, 500=0.62%, 750=9.45%, 1000=26.45%, 2000=60.27% lat (msec) : >=2000=2.58% cpu : usr=0.04%, sys=0.25%, ctx=1803, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145481: Sun Aug 23 03:50:34 2026 write: IOPS=13, BW=55.4KiB/s (56.7kB/s)(10.0MiB/184822msec); 0 zone resets slat (usec): min=36, max=15982, avg=268.45, stdev=743.21 clat (msec): min=147, max=2775, avg=1153.68, stdev=355.57 lat (msec): min=147, max=2775, avg=1153.95, stdev=355.59 clat percentiles (msec): | 1.00th=[ 435], 5.00th=[ 684], 10.00th=[ 751], 20.00th=[ 869], | 30.00th=[ 936], 40.00th=[ 1028], 50.00th=[ 1116], 60.00th=[ 1200], | 70.00th=[ 1301], 80.00th=[ 1418], 90.00th=[ 1620], 95.00th=[ 1787], | 99.00th=[ 2165], 99.50th=[ 2433], 99.90th=[ 2735], 99.95th=[ 2735], | 99.99th=[ 2769] bw ( KiB/s): min= 7, max= 136, per=25.34%, avg=56.01, stdev=28.09, samples=359 iops : min= 1, max= 34, avg=13.52, stdev= 7.06, samples=359 lat (msec) : 250=0.82%, 500=0.74%, 750=8.48%, 1000=25.70%, 2000=61.91% lat (msec) : >=2000=2.34% cpu : usr=0.04%, sys=0.24%, ctx=1829, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145482: Sun Aug 23 03:50:34 2026 write: IOPS=13, BW=55.5KiB/s (56.9kB/s)(10.0MiB/184372msec); 0 zone resets slat (usec): min=35, max=17665, avg=291.09, stdev=902.42 clat (msec): min=115, max=2857, avg=1150.91, stdev=364.28 lat (msec): min=115, max=2857, avg=1151.21, stdev=364.32 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 667], 10.00th=[ 726], 20.00th=[ 852], | 30.00th=[ 936], 40.00th=[ 1020], 50.00th=[ 1116], 60.00th=[ 1200], | 70.00th=[ 1301], 80.00th=[ 1435], 90.00th=[ 1620], 95.00th=[ 1770], | 99.00th=[ 2232], 99.50th=[ 2400], 99.90th=[ 2836], 99.95th=[ 2869], | 99.99th=[ 2869] bw ( KiB/s): min= 7, max= 128, per=25.35%, avg=56.02, stdev=27.44, samples=359 iops : min= 1, max= 32, avg=13.53, stdev= 6.89, samples=359 lat (msec) : 250=0.70%, 500=0.62%, 750=10.51%, 1000=25.47%, 2000=60.35% lat (msec) : >=2000=2.34% cpu : usr=0.04%, sys=0.27%, ctx=1912, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145483: Sun Aug 23 03:50:34 2026 write: IOPS=13, BW=55.4KiB/s (56.7kB/s)(10.0MiB/184811msec); 0 zone resets slat (usec): min=36, max=13105, avg=272.54, stdev=775.12 clat (msec): min=218, max=2817, avg=1153.57, stdev=361.17 lat (msec): min=218, max=2829, avg=1153.84, stdev=361.24 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 659], 10.00th=[ 735], 20.00th=[ 860], | 30.00th=[ 936], 40.00th=[ 1028], 50.00th=[ 1099], 60.00th=[ 1200], | 70.00th=[ 1318], 80.00th=[ 1418], 90.00th=[ 1603], 95.00th=[ 1770], | 99.00th=[ 2299], 99.50th=[ 2467], 99.90th=[ 2735], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 128, per=25.85%, avg=57.14, stdev=32.18, samples=352 iops : min= 1, max= 32, avg=13.81, stdev= 8.08, samples=352 lat (msec) : 250=0.62%, 750=10.00%, 1000=26.37%, 2000=60.51%, >=2000=2.50% cpu : usr=0.03%, sys=0.26%, ctx=1843, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=222KiB/s (227kB/s), 55.4KiB/s-55.5KiB/s (56.7kB/s-56.9kB/s), io=40.0MiB (41.9MB), run=184372-184822msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145520: Sun Aug 23 03:52:52 2026 read: IOPS=9, BW=39.5KiB/s (40.4kB/s)(5048KiB/127931msec) slat (usec): min=2844, max=61886, avg=16703.41, stdev=9252.73 clat (msec): min=18, max=962, avg=482.23, stdev=139.64 lat (msec): min=32, max=979, avg=498.95, stdev=141.01 clat percentiles (msec): | 1.00th=[ 148], 5.00th=[ 284], 10.00th=[ 334], 20.00th=[ 376], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 460], 60.00th=[ 493], | 70.00th=[ 542], 80.00th=[ 600], 90.00th=[ 667], 95.00th=[ 743], | 99.00th=[ 852], 99.50th=[ 902], 99.90th=[ 936], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 7, max= 167, per=25.32%, avg=39.50, stdev=20.24, samples=250 iops : min= 1, max= 41, avg= 9.39, stdev= 5.07, samples=250 write: IOPS=10, BW=40.6KiB/s (41.6kB/s)(5192KiB/127931msec); 0 zone resets slat (usec): min=41, max=19525, avg=417.12, stdev=1117.05 clat (msec): min=296, max=2105, avg=1087.17, stdev=310.13 lat (msec): min=297, max=2105, avg=1087.59, stdev=310.11 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 642], 10.00th=[ 726], 20.00th=[ 827], | 30.00th=[ 894], 40.00th=[ 969], 50.00th=[ 1045], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1334], 90.00th=[ 1552], 95.00th=[ 1670], | 99.00th=[ 1888], 99.50th=[ 2005], 99.90th=[ 2072], 99.95th=[ 2106], | 99.99th=[ 2106] bw ( KiB/s): min= 7, max= 95, per=24.91%, avg=40.35, stdev=18.05, samples=251 iops : min= 1, max= 23, avg= 9.60, stdev= 4.53, samples=251 lat (msec) : 20=0.04%, 50=0.12%, 100=0.12%, 250=1.17%, 500=29.57% lat (msec) : 750=22.03%, 1000=18.12%, 2000=28.52%, >=2000=0.31% cpu : usr=0.10%, sys=0.66%, ctx=4739, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145521: Sun Aug 23 03:52:52 2026 read: IOPS=9, BW=38.8KiB/s (39.7kB/s)(4976KiB/128238msec) slat (usec): min=3366, max=82163, avg=16272.69, stdev=9119.00 clat (msec): min=18, max=968, avg=487.33, stdev=141.58 lat (msec): min=24, max=975, avg=503.61, stdev=143.03 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 288], 10.00th=[ 330], 20.00th=[ 376], | 30.00th=[ 409], 40.00th=[ 439], 50.00th=[ 468], 60.00th=[ 498], | 70.00th=[ 550], 80.00th=[ 609], 90.00th=[ 684], 95.00th=[ 751], | 99.00th=[ 877], 99.50th=[ 894], 99.90th=[ 927], 99.95th=[ 969], | 99.99th=[ 969] bw ( KiB/s): min= 7, max= 119, per=24.74%, avg=38.60, stdev=19.70, samples=251 iops : min= 1, max= 29, avg= 9.16, stdev= 4.95, samples=251 write: IOPS=10, BW=41.0KiB/s (42.0kB/s)(5264KiB/128238msec); 0 zone resets slat (usec): min=39, max=24315, avg=406.28, stdev=1203.43 clat (msec): min=281, max=2086, avg=1080.07, stdev=317.79 lat (msec): min=282, max=2087, avg=1080.48, stdev=317.79 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 592], 10.00th=[ 709], 20.00th=[ 818], | 30.00th=[ 894], 40.00th=[ 969], 50.00th=[ 1036], 60.00th=[ 1116], | 70.00th=[ 1217], 80.00th=[ 1368], 90.00th=[ 1536], 95.00th=[ 1653], | 99.00th=[ 1838], 99.50th=[ 1972], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min= 7, max= 119, per=25.38%, avg=41.11, stdev=20.62, samples=251 iops : min= 1, max= 29, avg= 9.79, stdev= 5.18, samples=251 lat (msec) : 20=0.04%, 50=0.04%, 100=0.16%, 250=1.09%, 500=28.98% lat (msec) : 750=22.66%, 1000=18.16%, 2000=28.67%, >=2000=0.20% cpu : usr=0.09%, sys=0.63%, ctx=4133, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145522: Sun Aug 23 03:52:52 2026 read: IOPS=9, BW=38.9KiB/s (39.8kB/s)(4984KiB/128181msec) slat (usec): min=3457, max=73056, avg=16546.09, stdev=9319.29 clat (msec): min=34, max=904, avg=476.04, stdev=136.49 lat (msec): min=40, max=947, avg=492.60, stdev=138.15 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 284], 10.00th=[ 330], 20.00th=[ 372], | 30.00th=[ 401], 40.00th=[ 430], 50.00th=[ 460], 60.00th=[ 485], | 70.00th=[ 523], 80.00th=[ 600], 90.00th=[ 659], 95.00th=[ 735], | 99.00th=[ 852], 99.50th=[ 885], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 7, max= 143, per=25.24%, avg=39.37, stdev=21.76, samples=248 iops : min= 1, max= 35, avg= 9.35, stdev= 5.45, samples=248 write: IOPS=10, BW=41.0KiB/s (41.0kB/s)(5256KiB/128181msec); 0 zone resets slat (usec): min=44, max=15576, avg=417.05, stdev=1098.15 clat (msec): min=306, max=2118, avg=1089.23, stdev=322.32 lat (msec): min=311, max=2118, avg=1089.64, stdev=322.27 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 609], 10.00th=[ 709], 20.00th=[ 818], | 30.00th=[ 894], 40.00th=[ 969], 50.00th=[ 1053], 60.00th=[ 1133], | 70.00th=[ 1234], 80.00th=[ 1351], 90.00th=[ 1569], 95.00th=[ 1653], | 99.00th=[ 1905], 99.50th=[ 2056], 99.90th=[ 2089], 99.95th=[ 2123], | 99.99th=[ 2123] bw ( KiB/s): min= 7, max= 103, per=25.16%, avg=40.76, stdev=19.81, samples=251 iops : min= 1, max= 25, avg= 9.71, stdev= 4.98, samples=251 lat (msec) : 50=0.04%, 100=0.08%, 250=1.21%, 500=31.13%, 750=21.29% lat (msec) : 1000=17.11%, 2000=28.75%, >=2000=0.39% cpu : usr=0.08%, sys=0.65%, ctx=4191, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145523: Sun Aug 23 03:52:52 2026 read: IOPS=9, BW=39.7KiB/s (40.7kB/s)(5060KiB/127399msec) slat (usec): min=94, max=84794, avg=16862.80, stdev=9753.55 clat (msec): min=5, max=1564, avg=482.61, stdev=151.34 lat (msec): min=9, max=1575, avg=499.48, stdev=153.05 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 279], 10.00th=[ 330], 20.00th=[ 368], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 464], 60.00th=[ 493], | 70.00th=[ 542], 80.00th=[ 592], 90.00th=[ 676], 95.00th=[ 743], | 99.00th=[ 877], 99.50th=[ 911], 99.90th=[ 1536], 99.95th=[ 1569], | 99.99th=[ 1569] bw ( KiB/s): min= 7, max= 109, per=25.63%, avg=39.99, stdev=18.97, samples=248 iops : min= 1, max= 27, avg= 9.52, stdev= 4.76, samples=248 write: IOPS=10, BW=40.7KiB/s (41.6kB/s)(5180KiB/127399msec); 0 zone resets slat (usec): min=40, max=22595, avg=366.25, stdev=1051.17 clat (msec): min=232, max=2751, avg=1084.13, stdev=339.50 lat (msec): min=233, max=2752, avg=1084.51, stdev=339.52 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 600], 10.00th=[ 701], 20.00th=[ 802], | 30.00th=[ 894], 40.00th=[ 969], 50.00th=[ 1053], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1351], 90.00th=[ 1536], 95.00th=[ 1636], | 99.00th=[ 1905], 99.50th=[ 2500], 99.90th=[ 2769], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 7, max= 151, per=25.01%, avg=40.52, stdev=19.68, samples=249 iops : min= 1, max= 37, avg= 9.64, stdev= 4.92, samples=249 lat (msec) : 10=0.08%, 50=0.23%, 100=0.23%, 250=1.41%, 500=29.61% lat (msec) : 750=23.05%, 1000=16.76%, 2000=28.20%, >=2000=0.43% cpu : usr=0.10%, sys=0.66%, ctx=4577, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=156KiB/s (160kB/s), 38.8KiB/s-39.7KiB/s (39.7kB/s-40.7kB/s), io=19.6MiB (20.5MB), run=127399-128238msec WRITE: bw=163KiB/s (167kB/s), 40.6KiB/s-41.0KiB/s (41.6kB/s-42.0kB/s), io=20.4MiB (21.4MB), run=127399-128238msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145545: Sun Aug 23 03:53:02 2026 read: IOPS=1, BW=60.7MiB/s (63.6MB/s)(40.0MiB/659msec) slat (nsec): min=41659k, max=41659k, avg=41658570.00, stdev= 0.00 clat (nsec): min=550807k, max=550807k, avg=550806699.00, stdev= 0.00 lat (nsec): min=592469k, max=592469k, avg=592469421.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 550], 5.00th=[ 550], 10.00th=[ 550], 20.00th=[ 550], | 30.00th=[ 550], 40.00th=[ 550], 50.00th=[ 550], 60.00th=[ 550], | 70.00th=[ 550], 80.00th=[ 550], 90.00th=[ 550], 95.00th=[ 550], | 99.00th=[ 550], 99.50th=[ 550], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.82%, ctx=8, majf=0, minf=8 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=60.7MiB/s (63.6MB/s), 60.7MiB/s-60.7MiB/s (63.6MB/s-63.6MB/s), io=40.0MiB (41.9MB), run=659-659msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout