== sanity test 398c: run fio to test AIO ================= 11:51:49 (1787500309) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.663094 s, 63.3 MB/s osc.lustre-OST0000-osc-ffff936892e40800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146219: Sun Aug 23 11:55:19 2026 write: IOPS=12, BW=51.9KiB/s (53.1kB/s)(10.0MiB/197316msec); 0 zone resets slat (usec): min=29, max=27352, avg=316.26, stdev=1199.67 clat (msec): min=204, max=2893, avg=1231.68, stdev=382.49 lat (msec): min=208, max=2894, avg=1232.00, stdev=382.50 clat percentiles (msec): | 1.00th=[ 567], 5.00th=[ 735], 10.00th=[ 835], 20.00th=[ 944], | 30.00th=[ 1028], 40.00th=[ 1099], 50.00th=[ 1150], 60.00th=[ 1217], | 70.00th=[ 1334], 80.00th=[ 1485], 90.00th=[ 1770], 95.00th=[ 2005], | 99.00th=[ 2467], 99.50th=[ 2702], 99.90th=[ 2869], 99.95th=[ 2869], | 99.99th=[ 2903] bw ( KiB/s): min= 7, max= 128, per=25.83%, avg=53.47, stdev=25.48, samples=375 iops : min= 1, max= 32, avg=12.82, stdev= 6.38, samples=375 lat (msec) : 250=0.04%, 500=0.55%, 750=4.84%, 1000=20.78%, 2000=68.63% lat (msec) : >=2000=5.16% cpu : usr=0.06%, sys=0.24%, ctx=1941, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146220: Sun Aug 23 11:55:19 2026 write: IOPS=12, BW=51.0KiB/s (53.2kB/s)(10.0MiB/197060msec); 0 zone resets slat (usec): min=28, max=33132, avg=315.13, stdev=1249.68 clat (msec): min=59, max=2947, avg=1230.04, stdev=386.40 lat (msec): min=66, max=2947, avg=1230.36, stdev=386.47 clat percentiles (msec): | 1.00th=[ 558], 5.00th=[ 735], 10.00th=[ 835], 20.00th=[ 961], | 30.00th=[ 1020], 40.00th=[ 1083], 50.00th=[ 1150], 60.00th=[ 1234], | 70.00th=[ 1334], 80.00th=[ 1485], 90.00th=[ 1770], 95.00th=[ 2022], | 99.00th=[ 2467], 99.50th=[ 2635], 99.90th=[ 2903], 99.95th=[ 2903], | 99.99th=[ 2937] bw ( KiB/s): min= 7, max= 128, per=25.71%, avg=53.21, stdev=29.83, samples=377 iops : min= 1, max= 32, avg=12.76, stdev= 7.48, samples=377 lat (msec) : 100=0.04%, 250=0.59%, 500=0.04%, 750=4.92%, 1000=21.05% lat (msec) : 2000=67.85%, >=2000=5.51% cpu : usr=0.06%, sys=0.23%, ctx=1751, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146221: Sun Aug 23 11:55:19 2026 write: IOPS=13, BW=52.0KiB/s (53.3kB/s)(10.0MiB/196837msec); 0 zone resets slat (usec): min=27, max=33304, avg=346.77, stdev=1304.76 clat (msec): min=152, max=2949, avg=1229.27, stdev=398.85 lat (msec): min=152, max=2949, avg=1229.62, stdev=398.85 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 709], 10.00th=[ 835], 20.00th=[ 953], | 30.00th=[ 1020], 40.00th=[ 1083], 50.00th=[ 1150], 60.00th=[ 1234], | 70.00th=[ 1334], 80.00th=[ 1485], 90.00th=[ 1787], 95.00th=[ 2072], | 99.00th=[ 2467], 99.50th=[ 2702], 99.90th=[ 2869], 99.95th=[ 2903], | 99.99th=[ 2937] bw ( KiB/s): min= 7, max= 205, per=25.21%, avg=52.18, stdev=31.51, samples=384 iops : min= 1, max= 51, avg=12.50, stdev= 7.88, samples=384 lat (msec) : 250=0.70%, 500=0.55%, 750=5.04%, 1000=20.47%, 2000=67.42% lat (msec) : >=2000=5.82% cpu : usr=0.05%, sys=0.23%, ctx=1758, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146222: Sun Aug 23 11:55:19 2026 write: IOPS=12, BW=51.9KiB/s (53.1kB/s)(10.0MiB/197327msec); 0 zone resets slat (usec): min=28, max=24071, avg=325.25, stdev=1191.87 clat (msec): min=283, max=2860, avg=1231.71, stdev=383.80 lat (msec): min=285, max=2860, avg=1232.04, stdev=383.83 clat percentiles (msec): | 1.00th=[ 575], 5.00th=[ 735], 10.00th=[ 844], 20.00th=[ 961], | 30.00th=[ 1020], 40.00th=[ 1083], 50.00th=[ 1150], 60.00th=[ 1234], | 70.00th=[ 1318], 80.00th=[ 1469], 90.00th=[ 1770], 95.00th=[ 1989], | 99.00th=[ 2534], 99.50th=[ 2668], 99.90th=[ 2836], 99.95th=[ 2869], | 99.99th=[ 2869] bw ( KiB/s): min= 7, max= 128, per=25.57%, avg=52.92, stdev=26.34, samples=379 iops : min= 1, max= 32, avg=12.68, stdev= 6.60, samples=379 lat (msec) : 500=0.74%, 750=5.12%, 1000=21.60%, 2000=67.58%, >=2000=4.96% cpu : usr=0.05%, sys=0.26%, ctx=1747, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=208KiB/s (213kB/s), 51.9KiB/s-52.0KiB/s (53.1kB/s-53.3kB/s), io=40.0MiB (41.9MB), run=196837-197327msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146260: Sun Aug 23 11:57:32 2026 read: IOPS=10, BW=40.3KiB/s (41.2kB/s)(5048KiB/125363msec) slat (usec): min=3411, max=57600, avg=15857.10, stdev=8153.40 clat (msec): min=21, max=1105, avg=478.07, stdev=124.10 lat (msec): min=31, max=1124, avg=493.93, stdev=125.64 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 296], 10.00th=[ 330], 20.00th=[ 380], | 30.00th=[ 409], 40.00th=[ 439], 50.00th=[ 472], 60.00th=[ 502], | 70.00th=[ 531], 80.00th=[ 575], 90.00th=[ 642], 95.00th=[ 693], | 99.00th=[ 776], 99.50th=[ 852], 99.90th=[ 1070], 99.95th=[ 1099], | 99.99th=[ 1099] bw ( KiB/s): min= 7, max= 103, per=25.01%, avg=40.01, stdev=20.74, samples=247 iops : min= 1, max= 25, avg= 9.51, stdev= 5.18, samples=247 write: IOPS=10, BW=41.4KiB/s (42.4kB/s)(5192KiB/125363msec); 0 zone resets slat (usec): min=35, max=23734, avg=419.27, stdev=1346.45 clat (msec): min=360, max=2216, avg=1060.57, stdev=293.17 lat (msec): min=360, max=2216, avg=1060.99, stdev=293.18 clat percentiles (msec): | 1.00th=[ 481], 5.00th=[ 600], 10.00th=[ 709], 20.00th=[ 818], | 30.00th=[ 894], 40.00th=[ 961], 50.00th=[ 1036], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1301], 90.00th=[ 1452], 95.00th=[ 1569], | 99.00th=[ 1888], 99.50th=[ 2039], 99.90th=[ 2198], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 96, per=24.70%, avg=41.01, stdev=20.40, samples=247 iops : min= 1, max= 24, avg= 9.76, stdev= 5.14, samples=247 lat (msec) : 50=0.08%, 100=0.12%, 250=0.43%, 500=29.53%, 750=25.39% lat (msec) : 1000=16.48%, 2000=27.70%, >=2000=0.27% cpu : usr=0.11%, sys=0.62%, ctx=4152, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146261: Sun Aug 23 11:57:32 2026 read: IOPS=9, BW=39.7KiB/s (40.7kB/s)(4976KiB/125190msec) slat (usec): min=3488, max=58459, avg=15861.70, stdev=8115.77 clat (msec): min=34, max=1512, avg=481.43, stdev=139.63 lat (msec): min=61, max=1541, avg=497.29, stdev=140.55 clat percentiles (msec): | 1.00th=[ 146], 5.00th=[ 300], 10.00th=[ 334], 20.00th=[ 376], | 30.00th=[ 409], 40.00th=[ 435], 50.00th=[ 472], 60.00th=[ 506], | 70.00th=[ 542], 80.00th=[ 575], 90.00th=[ 642], 95.00th=[ 709], | 99.00th=[ 827], 99.50th=[ 1217], 99.90th=[ 1502], 99.95th=[ 1519], | 99.99th=[ 1519] bw ( KiB/s): min= 7, max= 111, per=25.12%, avg=40.19, stdev=19.79, samples=242 iops : min= 1, max= 27, avg= 9.54, stdev= 4.94, samples=242 write: IOPS=10, BW=42.0KiB/s (43.1kB/s)(5264KiB/125190msec); 0 zone resets slat (usec): min=31, max=13495, avg=361.13, stdev=1015.44 clat (msec): min=255, max=2238, avg=1048.35, stdev=298.07 lat (msec): min=255, max=2238, avg=1048.72, stdev=298.10 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 600], 10.00th=[ 709], 20.00th=[ 802], | 30.00th=[ 869], 40.00th=[ 953], 50.00th=[ 1020], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1284], 90.00th=[ 1452], 95.00th=[ 1569], | 99.00th=[ 1854], 99.50th=[ 2039], 99.90th=[ 2089], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 96, per=25.39%, avg=42.14, stdev=19.21, samples=244 iops : min= 1, max= 24, avg=10.04, stdev= 4.83, samples=244 lat (msec) : 50=0.04%, 100=0.31%, 250=0.51%, 500=28.48%, 750=25.20% lat (msec) : 1000=17.93%, 2000=27.23%, >=2000=0.31% cpu : usr=0.10%, sys=0.64%, ctx=4256, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146262: Sun Aug 23 11:57:32 2026 read: IOPS=9, BW=39.8KiB/s (40.7kB/s)(4984KiB/125314msec) slat (usec): min=37, max=57763, avg=16032.83, stdev=8372.73 clat (msec): min=29, max=921, avg=480.49, stdev=120.11 lat (msec): min=37, max=947, avg=496.54, stdev=121.37 clat percentiles (msec): | 1.00th=[ 239], 5.00th=[ 300], 10.00th=[ 334], 20.00th=[ 376], | 30.00th=[ 409], 40.00th=[ 443], 50.00th=[ 472], 60.00th=[ 514], | 70.00th=[ 550], 80.00th=[ 575], 90.00th=[ 642], 95.00th=[ 701], | 99.00th=[ 751], 99.50th=[ 768], 99.90th=[ 844], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 7, max= 119, per=24.85%, avg=39.76, stdev=21.03, samples=246 iops : min= 1, max= 29, avg= 9.44, stdev= 5.25, samples=246 write: IOPS=10, BW=41.9KiB/s (42.9kB/s)(5256KiB/125314msec); 0 zone resets slat (usec): min=31, max=18580, avg=377.92, stdev=1150.56 clat (msec): min=222, max=2237, avg=1053.65, stdev=301.89 lat (msec): min=222, max=2237, avg=1054.03, stdev=301.97 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 600], 10.00th=[ 684], 20.00th=[ 802], | 30.00th=[ 885], 40.00th=[ 953], 50.00th=[ 1036], 60.00th=[ 1116], | 70.00th=[ 1200], 80.00th=[ 1284], 90.00th=[ 1435], 95.00th=[ 1586], | 99.00th=[ 1854], 99.50th=[ 2039], 99.90th=[ 2140], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 95, per=25.03%, avg=41.55, stdev=18.66, samples=246 iops : min= 1, max= 23, avg= 9.89, stdev= 4.69, samples=246 lat (msec) : 50=0.08%, 100=0.12%, 250=0.43%, 500=27.93%, 750=27.30% lat (msec) : 1000=16.45%, 2000=27.34%, >=2000=0.35% cpu : usr=0.09%, sys=0.64%, ctx=4030, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146263: Sun Aug 23 11:57:32 2026 read: IOPS=10, BW=40.4KiB/s (41.3kB/s)(5060KiB/125312msec) slat (usec): min=3705, max=54061, avg=16103.35, stdev=8341.60 clat (msec): min=21, max=1224, avg=483.29, stdev=126.75 lat (msec): min=35, max=1249, avg=499.40, stdev=127.79 clat percentiles (msec): | 1.00th=[ 234], 5.00th=[ 305], 10.00th=[ 342], 20.00th=[ 380], | 30.00th=[ 409], 40.00th=[ 443], 50.00th=[ 472], 60.00th=[ 502], | 70.00th=[ 542], 80.00th=[ 584], 90.00th=[ 651], 95.00th=[ 701], | 99.00th=[ 776], 99.50th=[ 902], 99.90th=[ 1200], 99.95th=[ 1217], | 99.99th=[ 1217] bw ( KiB/s): min= 7, max= 96, per=25.83%, avg=41.32, stdev=18.46, samples=240 iops : min= 1, max= 24, avg= 9.83, stdev= 4.64, samples=240 write: IOPS=10, BW=41.3KiB/s (42.3kB/s)(5180KiB/125312msec); 0 zone resets slat (usec): min=29, max=10146, avg=382.78, stdev=893.40 clat (msec): min=445, max=2215, avg=1055.91, stdev=293.16 lat (msec): min=446, max=2215, avg=1056.29, stdev=293.16 clat percentiles (msec): | 1.00th=[ 514], 5.00th=[ 609], 10.00th=[ 709], 20.00th=[ 818], | 30.00th=[ 885], 40.00th=[ 953], 50.00th=[ 1028], 60.00th=[ 1099], | 70.00th=[ 1183], 80.00th=[ 1284], 90.00th=[ 1469], 95.00th=[ 1603], | 99.00th=[ 1854], 99.50th=[ 2056], 99.90th=[ 2140], 99.95th=[ 2232], | 99.99th=[ 2232] bw ( KiB/s): min= 7, max= 88, per=24.52%, avg=40.71, stdev=17.67, samples=248 iops : min= 1, max= 22, avg= 9.68, stdev= 4.45, samples=248 lat (msec) : 50=0.04%, 100=0.12%, 250=0.74%, 500=28.67%, 750=25.51% lat (msec) : 1000=18.09%, 2000=26.56%, >=2000=0.27% cpu : usr=0.09%, sys=0.68%, ctx=4405, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=160KiB/s (164kB/s), 39.7KiB/s-40.4KiB/s (40.7kB/s-41.3kB/s), io=19.6MiB (20.5MB), run=125190-125363msec WRITE: bw=167KiB/s (171kB/s), 41.3KiB/s-42.0KiB/s (42.3kB/s-43.1kB/s), io=20.4MiB (21.4MB), run=125190-125363msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146286: Sun Aug 23 11:57:41 2026 read: IOPS=2, BW=80.6MiB/s (84.6MB/s)(40.0MiB/496msec) slat (nsec): min=32389k, max=32389k, avg=32388979.00, stdev= 0.00 clat (nsec): min=445686k, max=445686k, avg=445685521.00, stdev= 0.00 lat (nsec): min=478103k, max=478103k, avg=478102620.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 447], 10.00th=[ 447], 20.00th=[ 447], | 30.00th=[ 447], 40.00th=[ 447], 50.00th=[ 447], 60.00th=[ 447], | 70.00th=[ 447], 80.00th=[ 447], 90.00th=[ 447], 95.00th=[ 447], | 99.00th=[ 447], 99.50th=[ 447], 99.90th=[ 447], 99.95th=[ 447], | 99.99th=[ 447] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.63%, ctx=42, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=80.6MiB/s (84.6MB/s), 80.6MiB/s-80.6MiB/s (84.6MB/s-84.6MB/s), io=40.0MiB (41.9MB), run=496-496msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout