== sanity test 398c: run fio to test AIO ================= 10:02:35 (1774879355) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.605498 s, 69.3 MB/s osc.lustre-OST0000-osc-ffff8e1d90127800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=149608: Mon Mar 30 10:05:44 2026 write: IOPS=14, BW=57.4KiB/s (58.7kB/s)(10.0MiB/178494msec); 0 zone resets slat (usec): min=50, max=18131, avg=360.32, stdev=996.47 clat (msec): min=124, max=2544, avg=1114.80, stdev=315.88 lat (msec): min=124, max=2545, avg=1115.16, stdev=315.88 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 693], 10.00th=[ 776], 20.00th=[ 877], | 30.00th=[ 944], 40.00th=[ 1020], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1351], 90.00th=[ 1485], 95.00th=[ 1653], | 99.00th=[ 2022], 99.50th=[ 2165], 99.90th=[ 2433], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 223, per=25.34%, avg=58.03, stdev=28.18, samples=346 iops : min= 1, max= 55, avg=14.00, stdev= 7.07, samples=346 lat (msec) : 250=0.62%, 500=1.88%, 750=5.66%, 1000=29.65%, 2000=61.02% lat (msec) : >=2000=1.17% cpu : usr=0.05%, sys=0.27%, ctx=2176, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149609: Mon Mar 30 10:05:44 2026 write: IOPS=14, BW=57.3KiB/s (58.7kB/s)(10.0MiB/178656msec); 0 zone resets slat (usec): min=40, max=39822, avg=405.12, stdev=1344.39 clat (msec): min=86, max=2551, avg=1115.35, stdev=310.76 lat (msec): min=87, max=2552, avg=1115.76, stdev=310.72 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 693], 10.00th=[ 776], 20.00th=[ 869], | 30.00th=[ 961], 40.00th=[ 1020], 50.00th=[ 1083], 60.00th=[ 1167], | 70.00th=[ 1217], 80.00th=[ 1334], 90.00th=[ 1485], 95.00th=[ 1653], | 99.00th=[ 2022], 99.50th=[ 2198], 99.90th=[ 2433], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 7, max= 148, per=24.92%, avg=57.08, stdev=27.67, samples=352 iops : min= 1, max= 37, avg=13.76, stdev= 6.94, samples=352 lat (msec) : 100=0.04%, 250=0.39%, 500=1.68%, 750=6.48%, 1000=27.42% lat (msec) : 2000=62.97%, >=2000=1.02% cpu : usr=0.05%, sys=0.31%, ctx=2283, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149610: Mon Mar 30 10:05:44 2026 write: IOPS=14, BW=57.4KiB/s (58.8kB/s)(10.0MiB/178398msec); 0 zone resets slat (usec): min=35, max=24758, avg=420.98, stdev=1276.53 clat (msec): min=151, max=2445, avg=1113.84, stdev=314.09 lat (msec): min=151, max=2445, avg=1114.27, stdev=314.12 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 701], 10.00th=[ 768], 20.00th=[ 877], | 30.00th=[ 944], 40.00th=[ 1020], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1485], 95.00th=[ 1703], | 99.00th=[ 2039], 99.50th=[ 2198], 99.90th=[ 2333], 99.95th=[ 2400], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 255, per=25.07%, avg=57.41, stdev=26.79, samples=350 iops : min= 1, max= 63, avg=13.84, stdev= 6.75, samples=350 lat (msec) : 250=0.62%, 500=1.41%, 750=6.68%, 1000=28.20%, 2000=61.91% lat (msec) : >=2000=1.17% cpu : usr=0.05%, sys=0.29%, ctx=2051, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=149611: Mon Mar 30 10:05:44 2026 write: IOPS=14, BW=57.3KiB/s (58.7kB/s)(10.0MiB/178732msec); 0 zone resets slat (usec): min=36, max=20502, avg=380.90, stdev=1062.69 clat (msec): min=270, max=2528, avg=1115.71, stdev=309.01 lat (msec): min=270, max=2534, avg=1116.10, stdev=309.04 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 701], 10.00th=[ 776], 20.00th=[ 877], | 30.00th=[ 953], 40.00th=[ 1020], 50.00th=[ 1083], 60.00th=[ 1150], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1502], 95.00th=[ 1653], | 99.00th=[ 2022], 99.50th=[ 2089], 99.90th=[ 2433], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 128, per=25.36%, avg=58.08, stdev=28.20, samples=346 iops : min= 1, max= 32, avg=14.01, stdev= 7.09, samples=346 lat (msec) : 500=1.88%, 750=6.17%, 1000=29.14%, 2000=61.60%, >=2000=1.21% cpu : usr=0.05%, sys=0.27%, ctx=2062, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=229KiB/s (235kB/s), 57.3KiB/s-57.4KiB/s (58.7kB/s-58.8kB/s), io=40.0MiB (41.9MB), run=178398-178732msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=149648: Mon Mar 30 10:07:57 2026 read: IOPS=10, BW=40.7KiB/s (41.7kB/s)(5048KiB/123960msec) slat (usec): min=71, max=75494, avg=15710.45, stdev=8800.68 clat (msec): min=14, max=1321, avg=489.84, stdev=167.06 lat (msec): min=20, max=1344, avg=505.56, stdev=169.31 clat percentiles (msec): | 1.00th=[ 83], 5.00th=[ 296], 10.00th=[ 330], 20.00th=[ 376], | 30.00th=[ 401], 40.00th=[ 430], 50.00th=[ 468], 60.00th=[ 498], | 70.00th=[ 531], 80.00th=[ 592], 90.00th=[ 676], 95.00th=[ 802], | 99.00th=[ 1116], 99.50th=[ 1200], 99.90th=[ 1301], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 7, max= 103, per=25.79%, avg=41.52, stdev=20.43, samples=238 iops : min= 1, max= 25, avg= 9.87, stdev= 5.16, samples=238 write: IOPS=10, BW=41.9KiB/s (42.9kB/s)(5192KiB/123960msec); 0 zone resets slat (usec): min=56, max=12687, avg=488.76, stdev=1074.33 clat (msec): min=132, max=3134, avg=1034.83, stdev=369.91 lat (msec): min=135, max=3134, avg=1035.32, stdev=369.92 clat percentiles (msec): | 1.00th=[ 351], 5.00th=[ 575], 10.00th=[ 651], 20.00th=[ 743], | 30.00th=[ 827], 40.00th=[ 919], 50.00th=[ 995], 60.00th=[ 1070], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1452], 95.00th=[ 1687], | 99.00th=[ 2467], 99.50th=[ 2601], 99.90th=[ 3037], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min= 7, max= 111, per=24.85%, avg=41.50, stdev=21.32, samples=244 iops : min= 1, max= 27, avg= 9.86, stdev= 5.35, samples=244 lat (msec) : 20=0.12%, 50=0.20%, 100=0.23%, 250=0.47%, 500=30.08% lat (msec) : 750=25.47%, 1000=17.34%, 2000=24.92%, >=2000=1.17% cpu : usr=0.09%, sys=0.69%, ctx=4874, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149649: Mon Mar 30 10:07:57 2026 read: IOPS=9, BW=39.0KiB/s (40.9kB/s)(4976KiB/124432msec) slat (usec): min=3576, max=83450, avg=15666.20, stdev=8694.63 clat (msec): min=17, max=1320, avg=482.15, stdev=154.64 lat (msec): min=32, max=1345, avg=497.83, stdev=156.70 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 300], 10.00th=[ 334], 20.00th=[ 372], | 30.00th=[ 397], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 485], | 70.00th=[ 527], 80.00th=[ 575], 90.00th=[ 659], 95.00th=[ 776], | 99.00th=[ 1070], 99.50th=[ 1167], 99.90th=[ 1284], 99.95th=[ 1318], | 99.99th=[ 1318] bw ( KiB/s): min= 7, max= 103, per=25.36%, avg=40.82, stdev=20.55, samples=238 iops : min= 1, max= 25, avg= 9.69, stdev= 5.19, samples=238 write: IOPS=10, BW=42.3KiB/s (43.3kB/s)(5264KiB/124432msec); 0 zone resets slat (usec): min=49, max=15333, avg=519.57, stdev=1265.67 clat (msec): min=287, max=3120, avg=1039.22, stdev=366.14 lat (msec): min=287, max=3121, avg=1039.74, stdev=366.18 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 575], 10.00th=[ 651], 20.00th=[ 743], | 30.00th=[ 835], 40.00th=[ 919], 50.00th=[ 1003], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1267], 90.00th=[ 1452], 95.00th=[ 1670], | 99.00th=[ 2500], 99.50th=[ 2702], 99.90th=[ 2903], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min= 7, max= 112, per=25.06%, avg=41.86, stdev=21.27, samples=246 iops : min= 1, max= 28, avg= 9.94, stdev= 5.37, samples=246 lat (msec) : 20=0.04%, 50=0.12%, 100=0.04%, 250=0.47%, 500=31.76% lat (msec) : 750=23.71%, 1000=17.38%, 2000=25.35%, >=2000=1.13% cpu : usr=0.10%, sys=0.68%, ctx=4811, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149650: Mon Mar 30 10:07:57 2026 read: IOPS=10, BW=40.0KiB/s (40.0kB/s)(4984KiB/124491msec) slat (usec): min=3939, max=56900, avg=15804.66, stdev=8310.32 clat (msec): min=9, max=1358, avg=485.41, stdev=157.20 lat (msec): min=15, max=1389, avg=501.22, stdev=159.13 clat percentiles (msec): | 1.00th=[ 128], 5.00th=[ 305], 10.00th=[ 338], 20.00th=[ 380], | 30.00th=[ 401], 40.00th=[ 430], 50.00th=[ 460], 60.00th=[ 489], | 70.00th=[ 523], 80.00th=[ 575], 90.00th=[ 659], 95.00th=[ 751], | 99.00th=[ 1070], 99.50th=[ 1150], 99.90th=[ 1334], 99.95th=[ 1351], | 99.99th=[ 1351] bw ( KiB/s): min= 7, max= 110, per=25.32%, avg=40.76, stdev=20.93, samples=238 iops : min= 1, max= 27, avg= 9.67, stdev= 5.30, samples=238 write: IOPS=10, BW=42.2KiB/s (43.2kB/s)(5256KiB/124491msec); 0 zone resets slat (usec): min=56, max=23484, avg=517.97, stdev=1594.52 clat (msec): min=88, max=3136, avg=1037.82, stdev=380.97 lat (msec): min=88, max=3136, avg=1038.35, stdev=380.94 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 550], 10.00th=[ 651], 20.00th=[ 751], | 30.00th=[ 827], 40.00th=[ 927], 50.00th=[ 1003], 60.00th=[ 1083], | 70.00th=[ 1167], 80.00th=[ 1250], 90.00th=[ 1485], 95.00th=[ 1720], | 99.00th=[ 2433], 99.50th=[ 2702], 99.90th=[ 3071], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min= 7, max= 111, per=24.85%, avg=41.51, stdev=21.01, samples=245 iops : min= 1, max= 27, avg= 9.86, stdev= 5.27, samples=245 lat (msec) : 10=0.04%, 20=0.04%, 50=0.12%, 100=0.39%, 250=0.86% lat (msec) : 500=30.86%, 750=24.14%, 1000=17.07%, 2000=25.35%, >=2000=1.13% cpu : usr=0.10%, sys=0.70%, ctx=4849, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=149651: Mon Mar 30 10:07:57 2026 read: IOPS=10, BW=40.8KiB/s (41.8kB/s)(5060KiB/124103msec) slat (usec): min=3392, max=85278, avg=15964.00, stdev=8876.05 clat (msec): min=20, max=1340, avg=494.74, stdev=167.63 lat (msec): min=29, max=1360, avg=510.72, stdev=169.53 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 300], 10.00th=[ 334], 20.00th=[ 380], | 30.00th=[ 405], 40.00th=[ 435], 50.00th=[ 468], 60.00th=[ 502], | 70.00th=[ 535], 80.00th=[ 592], 90.00th=[ 676], 95.00th=[ 810], | 99.00th=[ 1150], 99.50th=[ 1200], 99.90th=[ 1301], 99.95th=[ 1334], | 99.99th=[ 1334] bw ( KiB/s): min= 7, max= 112, per=25.61%, avg=41.22, stdev=19.39, samples=240 iops : min= 1, max= 28, avg= 9.79, stdev= 4.93, samples=240 write: IOPS=10, BW=41.7KiB/s (42.7kB/s)(5180KiB/124103msec); 0 zone resets slat (usec): min=57, max=21300, avg=496.41, stdev=1320.42 clat (msec): min=323, max=2804, avg=1031.69, stdev=348.35 lat (msec): min=324, max=2805, avg=1032.19, stdev=348.38 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 592], 10.00th=[ 651], 20.00th=[ 751], | 30.00th=[ 844], 40.00th=[ 911], 50.00th=[ 986], 60.00th=[ 1083], | 70.00th=[ 1150], 80.00th=[ 1234], 90.00th=[ 1435], 95.00th=[ 1636], | 99.00th=[ 2433], 99.50th=[ 2601], 99.90th=[ 2802], 99.95th=[ 2802], | 99.99th=[ 2802] bw ( KiB/s): min= 7, max= 96, per=24.72%, avg=41.27, stdev=19.33, samples=244 iops : min= 1, max= 24, avg= 9.80, stdev= 4.88, samples=244 lat (msec) : 50=0.23%, 100=0.23%, 250=0.27%, 500=29.80%, 750=25.78% lat (msec) : 1000=18.24%, 2000=24.53%, >=2000=0.90% cpu : usr=0.12%, sys=0.72%, ctx=5068, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=161KiB/s (165kB/s), 39.0KiB/s-40.8KiB/s (40.9kB/s-41.8kB/s), io=19.6MiB (20.5MB), run=123960-124491msec WRITE: bw=168KiB/s (172kB/s), 41.7KiB/s-42.3KiB/s (42.7kB/s-43.3kB/s), io=20.4MiB (21.4MB), run=123960-124491msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=149672: Mon Mar 30 10:08:05 2026 read: IOPS=1, BW=77.5MiB/s (81.3MB/s)(40.0MiB/516msec) slat (nsec): min=26145k, max=26145k, avg=26145274.00, stdev= 0.00 clat (nsec): min=445881k, max=445881k, avg=445881093.00, stdev= 0.00 lat (nsec): min=472031k, max=472031k, avg=472030644.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 447], 10.00th=[ 447], 20.00th=[ 447], | 30.00th=[ 447], 40.00th=[ 447], 50.00th=[ 447], 60.00th=[ 447], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] lat (msec) : 500=100.00% cpu : usr=0.19%, sys=2.33%, ctx=39, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=77.5MiB/s (81.3MB/s), 77.5MiB/s-77.5MiB/s (81.3MB/s-81.3MB/s), io=40.0MiB (41.9MB), run=516-516msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout