== sanity test 398c: run fio to test AIO ================= 14:36:08 (1785954968) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.555 s, 75.6 MB/s osc.lustre-OST0000-osc-ffff92c2ec43d000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=145177: Wed Aug 5 14:41:44 2026 write: IOPS=7, BW=31.9KiB/s (32.6kB/s)(10.0MiB/321482msec); 0 zone resets slat (usec): min=49, max=30357, avg=332.28, stdev=1450.01 clat (msec): min=406, max=4236, avg=2007.90, stdev=675.47 lat (msec): min=406, max=4236, avg=2008.23, stdev=675.53 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 860], 10.00th=[ 1116], 20.00th=[ 1435], | 30.00th=[ 1653], 40.00th=[ 1854], 50.00th=[ 2005], 60.00th=[ 2198], | 70.00th=[ 2366], 80.00th=[ 2601], 90.00th=[ 2836], 95.00th=[ 3071], | 99.00th=[ 3708], 99.50th=[ 3943], 99.90th=[ 4245], 99.95th=[ 4245], | 99.99th=[ 4245] bw ( KiB/s): min= 7, max= 128, per=27.51%, avg=34.94, stdev=23.35, samples=571 iops : min= 1, max= 32, avg= 8.24, stdev= 5.85, samples=571 lat (msec) : 500=1.84%, 750=1.60%, 1000=4.53%, 2000=41.21%, >=2000=50.82% cpu : usr=0.02%, sys=0.15%, ctx=1669, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145178: Wed Aug 5 14:41:44 2026 write: IOPS=7, BW=31.8KiB/s (32.6kB/s)(10.0MiB/321773msec); 0 zone resets slat (usec): min=48, max=25567, avg=338.94, stdev=1380.13 clat (msec): min=413, max=4236, avg=2009.78, stdev=676.77 lat (msec): min=413, max=4237, avg=2010.12, stdev=676.78 clat percentiles (msec): | 1.00th=[ 451], 5.00th=[ 793], 10.00th=[ 1053], 20.00th=[ 1435], | 30.00th=[ 1670], 40.00th=[ 1871], 50.00th=[ 2005], 60.00th=[ 2198], | 70.00th=[ 2366], 80.00th=[ 2601], 90.00th=[ 2836], 95.00th=[ 3104], | 99.00th=[ 3540], 99.50th=[ 3809], 99.90th=[ 4044], 99.95th=[ 4077], | 99.99th=[ 4245] bw ( KiB/s): min= 7, max= 128, per=27.16%, avg=34.49, stdev=21.94, samples=578 iops : min= 1, max= 32, avg= 8.12, stdev= 5.53, samples=578 lat (msec) : 500=1.25%, 750=2.58%, 1000=5.08%, 2000=40.47%, >=2000=50.62% cpu : usr=0.03%, sys=0.15%, ctx=1876, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145179: Wed Aug 5 14:41:44 2026 write: IOPS=7, BW=31.9KiB/s (32.6kB/s)(10.0MiB/321252msec); 0 zone resets slat (usec): min=49, max=43824, avg=377.61, stdev=1567.75 clat (msec): min=169, max=4033, avg=2006.24, stdev=681.45 lat (msec): min=169, max=4033, avg=2006.62, stdev=681.47 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 785], 10.00th=[ 1062], 20.00th=[ 1418], | 30.00th=[ 1653], 40.00th=[ 1838], 50.00th=[ 2022], 60.00th=[ 2198], | 70.00th=[ 2400], 80.00th=[ 2601], 90.00th=[ 2836], 95.00th=[ 3104], | 99.00th=[ 3574], 99.50th=[ 3742], 99.90th=[ 4010], 99.95th=[ 4010], | 99.99th=[ 4044] bw ( KiB/s): min= 7, max= 127, per=26.98%, avg=34.26, stdev=23.16, samples=582 iops : min= 1, max= 31, avg= 8.07, stdev= 5.82, samples=582 lat (msec) : 250=0.16%, 500=1.25%, 750=2.42%, 1000=4.57%, 2000=39.77% lat (msec) : >=2000=51.84% cpu : usr=0.03%, sys=0.17%, ctx=1732, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=145180: Wed Aug 5 14:41:44 2026 write: IOPS=7, BW=31.9KiB/s (32.6kB/s)(10.0MiB/321158msec); 0 zone resets slat (usec): min=49, max=23058, avg=362.60, stdev=1373.47 clat (msec): min=124, max=4085, avg=2005.95, stdev=681.98 lat (msec): min=124, max=4085, avg=2006.31, stdev=681.97 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 793], 10.00th=[ 1083], 20.00th=[ 1418], | 30.00th=[ 1670], 40.00th=[ 1838], 50.00th=[ 2022], 60.00th=[ 2198], | 70.00th=[ 2400], 80.00th=[ 2601], 90.00th=[ 2869], 95.00th=[ 3071], | 99.00th=[ 3540], 99.50th=[ 3742], 99.90th=[ 4077], 99.95th=[ 4077], | 99.99th=[ 4077] bw ( KiB/s): min= 7, max= 175, per=26.60%, avg=33.78, stdev=21.90, samples=590 iops : min= 1, max= 43, avg= 7.95, stdev= 5.50, samples=590 lat (msec) : 250=0.59%, 500=1.02%, 750=2.46%, 1000=4.41%, 2000=40.66% lat (msec) : >=2000=50.86% cpu : usr=0.03%, sys=0.16%, ctx=1791, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=127KiB/s (130kB/s), 31.8KiB/s-31.9KiB/s (32.6kB/s-32.6kB/s), io=40.0MiB (41.9MB), run=321158-321773msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=145236: Wed Aug 5 14:45:05 2026 read: IOPS=6, BW=26.6KiB/s (27.2kB/s)(5048KiB/190096msec) slat (msec): min=2, max=120, avg=20.12, stdev=13.14 clat (msec): min=41, max=1772, avg=734.03, stdev=279.96 lat (msec): min=52, max=1790, avg=754.16, stdev=283.39 clat percentiles (msec): | 1.00th=[ 232], 5.00th=[ 321], 10.00th=[ 388], 20.00th=[ 477], | 30.00th=[ 575], 40.00th=[ 659], 50.00th=[ 709], 60.00th=[ 785], | 70.00th=[ 844], 80.00th=[ 969], 90.00th=[ 1083], 95.00th=[ 1234], | 99.00th=[ 1519], 99.50th=[ 1603], 99.90th=[ 1720], 99.95th=[ 1770], | 99.99th=[ 1770] bw ( KiB/s): min= 7, max= 111, per=26.75%, avg=28.09, stdev=17.58, samples=350 iops : min= 1, max= 27, avg= 6.55, stdev= 4.42, samples=350 write: IOPS=6, BW=27.3KiB/s (27.0kB/s)(5192KiB/190096msec); 0 zone resets slat (usec): min=52, max=22896, avg=344.54, stdev=1318.90 clat (msec): min=485, max=3665, avg=1605.58, stdev=549.95 lat (msec): min=485, max=3665, avg=1605.93, stdev=549.96 clat percentiles (msec): | 1.00th=[ 634], 5.00th=[ 860], 10.00th=[ 953], 20.00th=[ 1099], | 30.00th=[ 1267], 40.00th=[ 1401], 50.00th=[ 1552], 60.00th=[ 1670], | 70.00th=[ 1854], 80.00th=[ 2039], 90.00th=[ 2333], 95.00th=[ 2635], | 99.00th=[ 3205], 99.50th=[ 3373], 99.90th=[ 3641], 99.95th=[ 3675], | 99.99th=[ 3675] bw ( KiB/s): min= 7, max= 88, per=26.17%, avg=28.53, stdev=17.43, samples=353 iops : min= 1, max= 22, avg= 6.66, stdev= 4.39, samples=353 lat (msec) : 50=0.08%, 100=0.08%, 250=0.51%, 500=10.43%, 750=17.58% lat (msec) : 1000=18.98%, 2000=41.41%, >=2000=10.94% cpu : usr=0.06%, sys=0.42%, ctx=4462, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145237: Wed Aug 5 14:45:05 2026 read: IOPS=6, BW=26.2KiB/s (26.8kB/s)(4976KiB/190093msec) slat (usec): min=50, max=113135, avg=20125.36, stdev=13037.17 clat (msec): min=23, max=1857, avg=736.62, stdev=286.16 lat (msec): min=42, max=1867, avg=756.76, stdev=289.53 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 313], 10.00th=[ 372], 20.00th=[ 489], | 30.00th=[ 584], 40.00th=[ 659], 50.00th=[ 726], 60.00th=[ 793], | 70.00th=[ 852], 80.00th=[ 944], 90.00th=[ 1083], 95.00th=[ 1250], | 99.00th=[ 1569], 99.50th=[ 1687], 99.90th=[ 1787], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7, max= 103, per=26.55%, avg=27.87, stdev=16.60, samples=347 iops : min= 1, max= 25, avg= 6.51, stdev= 4.17, samples=347 write: IOPS=6, BW=27.7KiB/s (28.4kB/s)(5264KiB/190093msec); 0 zone resets slat (usec): min=50, max=25632, avg=392.44, stdev=1524.71 clat (msec): min=271, max=3719, avg=1594.40, stdev=569.91 lat (msec): min=271, max=3719, avg=1594.79, stdev=569.87 clat percentiles (msec): | 1.00th=[ 401], 5.00th=[ 835], 10.00th=[ 944], 20.00th=[ 1099], | 30.00th=[ 1267], 40.00th=[ 1401], 50.00th=[ 1536], 60.00th=[ 1670], | 70.00th=[ 1821], 80.00th=[ 2022], 90.00th=[ 2333], 95.00th=[ 2635], | 99.00th=[ 3306], 99.50th=[ 3440], 99.90th=[ 3708], 99.95th=[ 3708], | 99.99th=[ 3708] bw ( KiB/s): min= 7, max= 91, per=26.59%, avg=28.99, stdev=17.35, samples=353 iops : min= 1, max= 22, avg= 6.78, stdev= 4.41, samples=353 lat (msec) : 50=0.12%, 100=0.16%, 250=0.55%, 500=10.16%, 750=17.11% lat (msec) : 1000=19.84%, 2000=41.37%, >=2000=10.70% cpu : usr=0.06%, sys=0.41%, ctx=4511, majf=0, minf=15 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145238: Wed Aug 5 14:45:05 2026 read: IOPS=6, BW=26.1KiB/s (26.8kB/s)(4984KiB/190723msec) slat (msec): min=2, max=117, avg=19.52, stdev=12.17 clat (msec): min=17, max=1829, avg=730.67, stdev=284.61 lat (msec): min=27, max=1857, avg=750.20, stdev=287.37 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 330], 10.00th=[ 384], 20.00th=[ 477], | 30.00th=[ 558], 40.00th=[ 642], 50.00th=[ 718], 60.00th=[ 785], | 70.00th=[ 860], 80.00th=[ 953], 90.00th=[ 1070], 95.00th=[ 1250], | 99.00th=[ 1536], 99.50th=[ 1653], 99.90th=[ 1821], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 103, per=27.46%, avg=28.84, stdev=17.18, samples=337 iops : min= 1, max= 25, avg= 6.74, stdev= 4.32, samples=337 write: IOPS=6, BW=27.6KiB/s (28.2kB/s)(5256KiB/190723msec); 0 zone resets slat (usec): min=51, max=18786, avg=340.33, stdev=1153.86 clat (msec): min=135, max=3709, avg=1606.93, stdev=557.54 lat (msec): min=135, max=3709, avg=1607.28, stdev=557.54 clat percentiles (msec): | 1.00th=[ 542], 5.00th=[ 860], 10.00th=[ 961], 20.00th=[ 1133], | 30.00th=[ 1284], 40.00th=[ 1418], 50.00th=[ 1552], 60.00th=[ 1670], | 70.00th=[ 1838], 80.00th=[ 2022], 90.00th=[ 2366], 95.00th=[ 2702], | 99.00th=[ 3171], 99.50th=[ 3373], 99.90th=[ 3507], 99.95th=[ 3708], | 99.99th=[ 3708] bw ( KiB/s): min= 7, max= 95, per=26.42%, avg=28.80, stdev=17.29, samples=354 iops : min= 1, max= 23, avg= 6.73, stdev= 4.35, samples=354 lat (msec) : 20=0.08%, 50=0.08%, 100=0.08%, 250=0.55%, 500=10.90% lat (msec) : 750=16.52%, 1000=19.69%, 2000=41.48%, >=2000=10.62% cpu : usr=0.06%, sys=0.44%, ctx=4534, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=145239: Wed Aug 5 14:45:05 2026 read: IOPS=6, BW=26.6KiB/s (27.3kB/s)(5060KiB/189959msec) slat (msec): min=2, max=125, avg=19.87, stdev=12.35 clat (msec): min=26, max=1844, avg=729.34, stdev=290.73 lat (msec): min=32, max=1890, avg=749.21, stdev=294.06 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 317], 10.00th=[ 384], 20.00th=[ 468], | 30.00th=[ 567], 40.00th=[ 651], 50.00th=[ 709], 60.00th=[ 776], | 70.00th=[ 844], 80.00th=[ 944], 90.00th=[ 1083], 95.00th=[ 1250], | 99.00th=[ 1620], 99.50th=[ 1720], 99.90th=[ 1838], 99.95th=[ 1838], | 99.99th=[ 1838] bw ( KiB/s): min= 7, max= 135, per=26.84%, avg=28.18, stdev=17.61, samples=350 iops : min= 1, max= 33, avg= 6.57, stdev= 4.43, samples=350 write: IOPS=6, BW=27.3KiB/s (27.9kB/s)(5180KiB/189959msec); 0 zone resets slat (usec): min=51, max=24099, avg=404.94, stdev=1500.69 clat (msec): min=244, max=3727, avg=1611.00, stdev=550.09 lat (msec): min=245, max=3727, avg=1611.41, stdev=550.12 clat percentiles (msec): | 1.00th=[ 625], 5.00th=[ 835], 10.00th=[ 953], 20.00th=[ 1133], | 30.00th=[ 1284], 40.00th=[ 1418], 50.00th=[ 1536], 60.00th=[ 1670], | 70.00th=[ 1854], 80.00th=[ 2039], 90.00th=[ 2333], 95.00th=[ 2702], | 99.00th=[ 3004], 99.50th=[ 3373], 99.90th=[ 3574], 99.95th=[ 3742], | 99.99th=[ 3742] bw ( KiB/s): min= 7, max= 95, per=26.17%, avg=28.53, stdev=16.13, samples=352 iops : min= 1, max= 23, avg= 6.66, stdev= 4.08, samples=352 lat (msec) : 50=0.12%, 100=0.12%, 250=0.55%, 500=11.29%, 750=16.52% lat (msec) : 1000=19.22%, 2000=41.48%, >=2000=10.70% cpu : usr=0.06%, sys=0.45%, ctx=4567, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=105KiB/s (108kB/s), 26.1KiB/s-26.6KiB/s (26.8kB/s-27.3kB/s), io=19.6MiB (20.5MB), run=189959-190723msec WRITE: bw=110KiB/s (112kB/s), 27.3KiB/s-27.7KiB/s (27.9kB/s-28.4kB/s), io=20.4MiB (21.4MB), run=189959-190723msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=145273: Wed Aug 5 14:45:20 2026 read: IOPS=1, BW=47.8MiB/s (50.2MB/s)(40.0MiB/836msec) slat (nsec): min=37431k, max=37431k, avg=37431150.00, stdev= 0.00 clat (nsec): min=769408k, max=769408k, avg=769408177.00, stdev= 0.00 lat (nsec): min=806844k, max=806844k, avg=806843830.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 768], 5.00th=[ 768], 10.00th=[ 768], 20.00th=[ 768], | 30.00th=[ 768], 40.00th=[ 768], 50.00th=[ 768], 60.00th=[ 768], | 70.00th=[ 768], 80.00th=[ 768], 90.00th=[ 768], 95.00th=[ 768], | 99.00th=[ 768], 99.50th=[ 768], 99.90th=[ 768], 99.95th=[ 768], | 99.99th=[ 768] lat (msec) : 1000=100.00% cpu : usr=0.00%, sys=1.56%, ctx=6, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=47.8MiB/s (50.2MB/s), 47.8MiB/s-47.8MiB/s (50.2MB/s-50.2MB/s), io=40.0MiB (41.9MB), run=836-836msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout