== sanity test 398c: run fio to test AIO ================= 16:26:54 (1789590414) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.642341 s, 65.3 MB/s osc.lustre-OST0000-osc-ffff92546a0c7000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=174791: Wed Sep 16 16:29:56 2026 write: IOPS=14, BW=59.2KiB/s (60.6kB/s)(10.0MiB/173042msec); 0 zone resets slat (usec): min=25, max=33671, avg=325.72, stdev=1319.87 clat (msec): min=159, max=3108, avg=1080.79, stdev=372.40 lat (msec): min=159, max=3109, avg=1081.12, stdev=372.45 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 592], 10.00th=[ 667], 20.00th=[ 785], | 30.00th=[ 877], 40.00th=[ 986], 50.00th=[ 1053], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1318], 90.00th=[ 1519], 95.00th=[ 1737], | 99.00th=[ 2265], 99.50th=[ 2735], 99.90th=[ 3004], 99.95th=[ 3004], | 99.99th=[ 3104] bw ( KiB/s): min= 7, max= 231, per=25.90%, avg=60.86, stdev=32.53, samples=330 iops : min= 1, max= 57, avg=14.66, stdev= 8.18, samples=330 lat (msec) : 250=0.51%, 500=2.58%, 750=13.09%, 1000=26.48%, 2000=55.20% lat (msec) : >=2000=2.15% cpu : usr=0.06%, sys=0.26%, ctx=1914, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=174792: Wed Sep 16 16:29:56 2026 write: IOPS=14, BW=58.9KiB/s (60.3kB/s)(10.0MiB/173749msec); 0 zone resets slat (usec): min=29, max=22889, avg=275.15, stdev=1134.46 clat (msec): min=255, max=3014, avg=1084.31, stdev=365.64 lat (msec): min=258, max=3014, avg=1084.59, stdev=365.62 clat percentiles (msec): | 1.00th=[ 363], 5.00th=[ 584], 10.00th=[ 676], 20.00th=[ 776], | 30.00th=[ 869], 40.00th=[ 978], 50.00th=[ 1053], 60.00th=[ 1133], | 70.00th=[ 1234], 80.00th=[ 1334], 90.00th=[ 1552], 95.00th=[ 1703], | 99.00th=[ 2198], 99.50th=[ 2802], 99.90th=[ 2937], 99.95th=[ 2970], | 99.99th=[ 3004] bw ( KiB/s): min= 7, max= 198, per=25.44%, avg=59.79, stdev=29.99, samples=336 iops : min= 1, max= 49, avg=14.39, stdev= 7.55, samples=336 lat (msec) : 500=2.58%, 750=13.28%, 1000=27.19%, 2000=55.51%, >=2000=1.45% cpu : usr=0.05%, sys=0.24%, ctx=1758, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=174793: Wed Sep 16 16:29:56 2026 write: IOPS=14, BW=58.9KiB/s (60.4kB/s)(10.0MiB/173743msec); 0 zone resets slat (usec): min=30, max=29982, avg=290.33, stdev=1133.76 clat (msec): min=252, max=3110, avg=1084.35, stdev=367.31 lat (msec): min=256, max=3110, avg=1084.64, stdev=367.31 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 592], 10.00th=[ 684], 20.00th=[ 785], | 30.00th=[ 869], 40.00th=[ 969], 50.00th=[ 1053], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1334], 90.00th=[ 1536], 95.00th=[ 1737], | 99.00th=[ 2265], 99.50th=[ 2802], 99.90th=[ 2970], 99.95th=[ 3004], | 99.99th=[ 3104] bw ( KiB/s): min= 7, max= 215, per=25.30%, avg=59.45, stdev=30.27, samples=338 iops : min= 1, max= 53, avg=14.31, stdev= 7.61, samples=338 lat (msec) : 500=2.54%, 750=13.20%, 1000=28.12%, 2000=54.57%, >=2000=1.56% cpu : usr=0.06%, sys=0.26%, ctx=2197, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=174794: Wed Sep 16 16:29:56 2026 write: IOPS=14, BW=59.1KiB/s (60.5kB/s)(10.0MiB/173345msec); 0 zone resets slat (usec): min=30, max=18109, avg=305.58, stdev=1073.46 clat (msec): min=74, max=3129, avg=1082.04, stdev=371.15 lat (msec): min=74, max=3129, avg=1082.35, stdev=371.14 clat percentiles (msec): | 1.00th=[ 347], 5.00th=[ 584], 10.00th=[ 684], 20.00th=[ 785], | 30.00th=[ 869], 40.00th=[ 978], 50.00th=[ 1053], 60.00th=[ 1133], | 70.00th=[ 1217], 80.00th=[ 1334], 90.00th=[ 1536], 95.00th=[ 1737], | 99.00th=[ 2198], 99.50th=[ 2802], 99.90th=[ 3071], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min= 7, max= 223, per=24.99%, avg=58.73, stdev=27.44, samples=342 iops : min= 1, max= 55, avg=14.13, stdev= 6.90, samples=342 lat (msec) : 100=0.08%, 250=0.08%, 500=3.01%, 750=12.70%, 1000=26.76% lat (msec) : 2000=55.62%, >=2000=1.76% cpu : usr=0.05%, sys=0.25%, ctx=2017, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=236KiB/s (241kB/s), 58.9KiB/s-59.2KiB/s (60.3kB/s-60.6kB/s), io=40.0MiB (41.9MB), run=173042-173749msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=174829: Wed Sep 16 16:32:05 2026 read: IOPS=10, BW=43.2KiB/s (44.2kB/s)(5048KiB/116822msec) slat (usec): min=3134, max=66371, avg=15167.59, stdev=8026.69 clat (msec): min=26, max=904, avg=449.76, stdev=133.63 lat (msec): min=33, max=925, avg=464.94, stdev=135.35 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 279], 10.00th=[ 305], 20.00th=[ 342], | 30.00th=[ 376], 40.00th=[ 405], 50.00th=[ 426], 60.00th=[ 464], | 70.00th=[ 498], 80.00th=[ 542], 90.00th=[ 625], 95.00th=[ 709], | 99.00th=[ 869], 99.50th=[ 885], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 7, max= 119, per=25.38%, avg=43.40, stdev=20.18, samples=228 iops : min= 1, max= 29, avg=10.34, stdev= 5.07, samples=228 write: IOPS=11, BW=44.4KiB/s (45.5kB/s)(5192KiB/116822msec); 0 zone resets slat (usec): min=33, max=15641, avg=362.10, stdev=1124.30 clat (msec): min=380, max=1859, avg=983.88, stdev=252.77 lat (msec): min=380, max=1860, avg=984.25, stdev=252.80 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 625], 10.00th=[ 676], 20.00th=[ 760], | 30.00th=[ 835], 40.00th=[ 894], 50.00th=[ 953], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1200], 90.00th=[ 1318], 95.00th=[ 1469], | 99.00th=[ 1636], 99.50th=[ 1703], 99.90th=[ 1737], 99.95th=[ 1854], | 99.99th=[ 1854] bw ( KiB/s): min= 7, max= 103, per=24.88%, avg=44.29, stdev=19.07, samples=229 iops : min= 1, max= 25, avg=10.55, stdev= 4.75, samples=229 lat (msec) : 50=0.08%, 100=0.08%, 250=1.33%, 500=33.83%, 750=21.52% lat (msec) : 1000=21.25%, 2000=21.91% cpu : usr=0.14%, sys=0.71%, ctx=4765, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=174830: Wed Sep 16 16:32:05 2026 read: IOPS=10, BW=42.5KiB/s (43.6kB/s)(4976KiB/116950msec) slat (usec): min=3584, max=59831, avg=15160.86, stdev=7959.35 clat (msec): min=20, max=903, avg=445.53, stdev=126.82 lat (msec): min=32, max=909, avg=460.71, stdev=128.55 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 275], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 456], | 70.00th=[ 485], 80.00th=[ 531], 90.00th=[ 617], 95.00th=[ 684], | 99.00th=[ 818], 99.50th=[ 852], 99.90th=[ 894], 99.95th=[ 902], | 99.99th=[ 902] bw ( KiB/s): min= 7, max= 103, per=24.63%, avg=42.12, stdev=19.95, samples=230 iops : min= 1, max= 25, avg=10.01, stdev= 5.01, samples=230 write: IOPS=11, BW=45.0KiB/s (46.1kB/s)(5264KiB/116950msec); 0 zone resets slat (usec): min=30, max=24851, avg=405.24, stdev=1292.20 clat (msec): min=93, max=1906, avg=983.35, stdev=267.55 lat (msec): min=93, max=1906, avg=983.76, stdev=267.61 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 600], 10.00th=[ 667], 20.00th=[ 760], | 30.00th=[ 827], 40.00th=[ 894], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1469], | 99.00th=[ 1687], 99.50th=[ 1754], 99.90th=[ 1871], 99.95th=[ 1905], | 99.99th=[ 1905] bw ( KiB/s): min= 7, max= 103, per=24.94%, avg=44.39, stdev=20.59, samples=231 iops : min= 1, max= 25, avg=10.58, stdev= 5.16, samples=231 lat (msec) : 50=0.08%, 100=0.31%, 250=1.37%, 500=35.12%, 750=20.27% lat (msec) : 1000=20.86%, 2000=21.99% cpu : usr=0.12%, sys=0.73%, ctx=4795, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=174831: Wed Sep 16 16:32:05 2026 read: IOPS=10, BW=42.7KiB/s (43.7kB/s)(4984KiB/116775msec) slat (usec): min=51, max=65069, avg=15256.95, stdev=8430.33 clat (msec): min=16, max=914, avg=447.76, stdev=128.62 lat (msec): min=16, max=941, avg=463.02, stdev=130.74 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 279], 10.00th=[ 317], 20.00th=[ 351], | 30.00th=[ 380], 40.00th=[ 405], 50.00th=[ 430], 60.00th=[ 460], | 70.00th=[ 498], 80.00th=[ 542], 90.00th=[ 617], 95.00th=[ 676], | 99.00th=[ 835], 99.50th=[ 860], 99.90th=[ 911], 99.95th=[ 919], | 99.99th=[ 919] bw ( KiB/s): min= 7, max= 112, per=25.00%, avg=42.75, stdev=21.07, samples=229 iops : min= 1, max= 28, avg=10.17, stdev= 5.31, samples=229 write: IOPS=11, BW=45.0KiB/s (46.1kB/s)(5256KiB/116775msec); 0 zone resets slat (usec): min=32, max=21537, avg=395.11, stdev=1309.84 clat (msec): min=177, max=1877, avg=981.86, stdev=259.01 lat (msec): min=177, max=1877, avg=982.25, stdev=259.10 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 609], 10.00th=[ 676], 20.00th=[ 768], | 30.00th=[ 827], 40.00th=[ 885], 50.00th=[ 953], 60.00th=[ 1028], | 70.00th=[ 1099], 80.00th=[ 1183], 90.00th=[ 1318], 95.00th=[ 1452], | 99.00th=[ 1687], 99.50th=[ 1770], 99.90th=[ 1838], 99.95th=[ 1871], | 99.99th=[ 1871] bw ( KiB/s): min= 7, max= 95, per=24.92%, avg=44.35, stdev=19.38, samples=231 iops : min= 1, max= 23, avg=10.57, stdev= 4.83, samples=231 lat (msec) : 20=0.04%, 50=0.12%, 100=0.23%, 250=1.48%, 500=33.48% lat (msec) : 750=20.66%, 1000=21.52%, 2000=22.46% cpu : usr=0.11%, sys=0.69%, ctx=4842, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=174832: Wed Sep 16 16:32:05 2026 read: IOPS=10, BW=43.3KiB/s (44.4kB/s)(5060KiB/116821msec) slat (usec): min=3355, max=69194, avg=15426.11, stdev=8604.90 clat (msec): min=54, max=958, avg=448.80, stdev=126.88 lat (msec): min=66, max=987, avg=464.23, stdev=128.62 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 279], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 376], 40.00th=[ 405], 50.00th=[ 435], 60.00th=[ 460], | 70.00th=[ 489], 80.00th=[ 535], 90.00th=[ 609], 95.00th=[ 693], | 99.00th=[ 852], 99.50th=[ 885], 99.90th=[ 911], 99.95th=[ 961], | 99.99th=[ 961] bw ( KiB/s): min= 7, max= 111, per=25.35%, avg=43.34, stdev=20.72, samples=229 iops : min= 1, max= 27, avg=10.31, stdev= 5.22, samples=229 write: IOPS=11, BW=44.3KiB/s (45.4kB/s)(5180KiB/116821msec); 0 zone resets slat (usec): min=34, max=19508, avg=398.30, stdev=1234.69 clat (msec): min=345, max=1889, avg=987.02, stdev=261.29 lat (msec): min=345, max=1889, avg=987.42, stdev=261.36 clat percentiles (msec): | 1.00th=[ 477], 5.00th=[ 617], 10.00th=[ 676], 20.00th=[ 768], | 30.00th=[ 835], 40.00th=[ 902], 50.00th=[ 961], 60.00th=[ 1028], | 70.00th=[ 1116], 80.00th=[ 1200], 90.00th=[ 1334], 95.00th=[ 1452], | 99.00th=[ 1754], 99.50th=[ 1787], 99.90th=[ 1871], 99.95th=[ 1888], | 99.99th=[ 1888] bw ( KiB/s): min= 7, max= 104, per=24.91%, avg=44.35, stdev=18.42, samples=228 iops : min= 1, max= 26, avg=10.57, stdev= 4.60, samples=228 lat (msec) : 100=0.20%, 250=0.90%, 500=35.62%, 750=20.16%, 1000=21.60% lat (msec) : 2000=21.52% cpu : usr=0.11%, sys=0.66%, ctx=4657, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=172KiB/s (176kB/s), 42.5KiB/s-43.3KiB/s (43.6kB/s-44.4kB/s), io=19.6MiB (20.5MB), run=116775-116950msec WRITE: bw=179KiB/s (183kB/s), 44.3KiB/s-45.0KiB/s (45.4kB/s-46.1kB/s), io=20.4MiB (21.4MB), run=116775-116950msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=174852: Wed Sep 16 16:32:16 2026 read: IOPS=1, BW=70.5MiB/s (73.0MB/s)(40.0MiB/567msec) slat (nsec): min=32960k, max=32960k, avg=32959567.00, stdev= 0.00 clat (nsec): min=487359k, max=487359k, avg=487358925.00, stdev= 0.00 lat (nsec): min=520349k, max=520349k, avg=520349187.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 489], 10.00th=[ 489], 20.00th=[ 489], | 30.00th=[ 489], 40.00th=[ 489], 50.00th=[ 489], 60.00th=[ 489], | 70.00th=[ 489], 80.00th=[ 489], 90.00th=[ 489], 95.00th=[ 489], | 99.00th=[ 489], 99.50th=[ 489], 99.90th=[ 489], 99.95th=[ 489], | 99.99th=[ 489] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.47%, ctx=27, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=70.5MiB/s (73.0MB/s), 70.5MiB/s-70.5MiB/s (73.0MB/s-73.0MB/s), io=40.0MiB (41.9MB), run=567-567msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout