== sanity test 398c: run fio to test AIO ================= 04:05:37 (1786694737) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.61344 s, 68.4 MB/s osc.lustre-OST0000-osc-ffff923a072b7800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=146708: Fri Aug 14 04:08:32 2026 write: IOPS=15, BW=62.8KiB/s (64.3kB/s)(10.0MiB/163153msec); 0 zone resets slat (usec): min=37, max=19048, avg=260.63, stdev=929.47 clat (msec): min=233, max=2588, avg=1017.98, stdev=323.50 lat (msec): min=233, max=2588, avg=1018.24, stdev=323.51 clat percentiles (msec): | 1.00th=[ 300], 5.00th=[ 592], 10.00th=[ 693], 20.00th=[ 793], | 30.00th=[ 852], 40.00th=[ 902], 50.00th=[ 944], 60.00th=[ 1028], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1469], 95.00th=[ 1620], | 99.00th=[ 2022], 99.50th=[ 2198], 99.90th=[ 2534], 99.95th=[ 2601], | 99.99th=[ 2601] bw ( KiB/s): min= 7, max= 168, per=24.93%, avg=62.57, stdev=28.21, samples=322 iops : min= 1, max= 42, avg=15.20, stdev= 7.09, samples=322 lat (msec) : 250=0.62%, 500=2.11%, 750=11.91%, 1000=42.23%, 2000=42.11% lat (msec) : >=2000=1.02% cpu : usr=0.05%, sys=0.25%, ctx=1798, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146709: Fri Aug 14 04:08:32 2026 write: IOPS=15, BW=62.8KiB/s (64.3kB/s)(10.0MiB/162983msec); 0 zone resets slat (usec): min=44, max=28634, avg=262.21, stdev=1086.40 clat (msec): min=178, max=2493, avg=1017.96, stdev=326.10 lat (msec): min=178, max=2493, avg=1018.22, stdev=326.14 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 592], 10.00th=[ 701], 20.00th=[ 785], | 30.00th=[ 844], 40.00th=[ 902], 50.00th=[ 953], 60.00th=[ 1028], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1469], 95.00th=[ 1636], | 99.00th=[ 1972], 99.50th=[ 2165], 99.90th=[ 2433], 99.95th=[ 2467], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 184, per=25.66%, avg=64.40, stdev=30.71, samples=313 iops : min= 1, max= 46, avg=15.65, stdev= 7.71, samples=313 lat (msec) : 250=0.35%, 500=2.70%, 750=12.15%, 1000=41.88%, 2000=42.03% lat (msec) : >=2000=0.90% cpu : usr=0.04%, sys=0.23%, ctx=1942, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146710: Fri Aug 14 04:08:32 2026 write: IOPS=15, BW=62.8KiB/s (64.3kB/s)(10.0MiB/163158msec); 0 zone resets slat (usec): min=38, max=14063, avg=242.50, stdev=817.97 clat (msec): min=251, max=2512, avg=1018.18, stdev=322.75 lat (msec): min=251, max=2512, avg=1018.42, stdev=322.77 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 592], 10.00th=[ 693], 20.00th=[ 793], | 30.00th=[ 852], 40.00th=[ 902], 50.00th=[ 953], 60.00th=[ 1020], | 70.00th=[ 1099], 80.00th=[ 1267], 90.00th=[ 1469], 95.00th=[ 1603], | 99.00th=[ 1989], 99.50th=[ 2232], 99.90th=[ 2467], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 168, per=24.73%, avg=62.07, stdev=28.50, samples=324 iops : min= 1, max= 42, avg=15.07, stdev= 7.17, samples=324 lat (msec) : 500=3.24%, 750=12.19%, 1000=41.84%, 2000=41.88%, >=2000=0.86% cpu : usr=0.04%, sys=0.24%, ctx=1941, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=146711: Fri Aug 14 04:08:32 2026 write: IOPS=15, BW=62.8KiB/s (64.3kB/s)(10.0MiB/163161msec); 0 zone resets slat (usec): min=37, max=18436, avg=276.46, stdev=941.94 clat (msec): min=268, max=2534, avg=1018.20, stdev=322.98 lat (msec): min=268, max=2536, avg=1018.48, stdev=323.03 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 584], 10.00th=[ 701], 20.00th=[ 793], | 30.00th=[ 852], 40.00th=[ 902], 50.00th=[ 953], 60.00th=[ 1028], | 70.00th=[ 1116], 80.00th=[ 1234], 90.00th=[ 1469], 95.00th=[ 1636], | 99.00th=[ 1989], 99.50th=[ 2123], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2534] bw ( KiB/s): min= 7, max= 208, per=24.93%, avg=62.57, stdev=27.51, samples=322 iops : min= 1, max= 52, avg=15.19, stdev= 6.90, samples=322 lat (msec) : 500=2.93%, 750=11.84%, 1000=42.27%, 2000=42.11%, >=2000=0.86% cpu : usr=0.04%, sys=0.25%, ctx=2109, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=251KiB/s (257kB/s), 62.8KiB/s-62.8KiB/s (64.3kB/s-64.3kB/s), io=40.0MiB (41.9MB), run=162983-163161msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=146744: Fri Aug 14 04:10:35 2026 read: IOPS=11, BW=44.3KiB/s (45.3kB/s)(5048KiB/114045msec) slat (usec): min=3031, max=61812, avg=15313.16, stdev=8845.78 clat (msec): min=16, max=1088, avg=440.79, stdev=142.02 lat (msec): min=29, max=1096, avg=456.11, stdev=144.02 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 275], 10.00th=[ 300], 20.00th=[ 334], | 30.00th=[ 359], 40.00th=[ 388], 50.00th=[ 414], 60.00th=[ 439], | 70.00th=[ 481], 80.00th=[ 542], 90.00th=[ 625], 95.00th=[ 709], | 99.00th=[ 953], 99.50th=[ 978], 99.90th=[ 1020], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 112, per=25.44%, avg=44.53, stdev=22.20, samples=222 iops : min= 1, max= 28, avg=10.64, stdev= 5.57, samples=222 write: IOPS=11, BW=45.5KiB/s (46.6kB/s)(5192KiB/114045msec); 0 zone resets slat (usec): min=40, max=37693, avg=389.42, stdev=1568.01 clat (msec): min=293, max=2429, avg=959.44, stdev=342.62 lat (msec): min=293, max=2429, avg=959.83, stdev=342.71 clat percentiles (msec): | 1.00th=[ 409], 5.00th=[ 531], 10.00th=[ 625], 20.00th=[ 693], | 30.00th=[ 751], 40.00th=[ 818], 50.00th=[ 894], 60.00th=[ 961], | 70.00th=[ 1070], 80.00th=[ 1183], 90.00th=[ 1401], 95.00th=[ 1653], | 99.00th=[ 2198], 99.50th=[ 2299], 99.90th=[ 2400], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 135, per=24.61%, avg=45.04, stdev=23.05, samples=225 iops : min= 1, max= 33, avg=10.77, stdev= 5.80, samples=225 lat (msec) : 20=0.04%, 50=0.20%, 100=0.16%, 250=0.86%, 500=36.68% lat (msec) : 750=24.45%, 1000=19.02%, 2000=17.73%, >=2000=0.86% cpu : usr=0.09%, sys=0.64%, ctx=4371, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146745: Fri Aug 14 04:10:35 2026 read: IOPS=10, BW=43.6KiB/s (44.6kB/s)(4976KiB/114145msec) slat (usec): min=2830, max=62103, avg=15481.11, stdev=8873.70 clat (msec): min=26, max=1084, avg=432.52, stdev=138.72 lat (msec): min=31, max=1121, avg=448.01, stdev=140.57 clat percentiles (msec): | 1.00th=[ 205], 5.00th=[ 266], 10.00th=[ 292], 20.00th=[ 330], | 30.00th=[ 351], 40.00th=[ 376], 50.00th=[ 405], 60.00th=[ 430], | 70.00th=[ 472], 80.00th=[ 531], 90.00th=[ 634], 95.00th=[ 701], | 99.00th=[ 835], 99.50th=[ 894], 99.90th=[ 1070], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 111, per=24.57%, avg=43.00, stdev=21.73, samples=226 iops : min= 1, max= 27, avg=10.26, stdev= 5.48, samples=226 write: IOPS=11, BW=46.1KiB/s (47.2kB/s)(5264KiB/114145msec); 0 zone resets slat (usec): min=44, max=20040, avg=373.69, stdev=1364.14 clat (msec): min=99, max=2465, avg=961.80, stdev=356.89 lat (msec): min=99, max=2465, avg=962.17, stdev=357.01 clat percentiles (msec): | 1.00th=[ 334], 5.00th=[ 506], 10.00th=[ 609], 20.00th=[ 693], | 30.00th=[ 751], 40.00th=[ 827], 50.00th=[ 894], 60.00th=[ 969], | 70.00th=[ 1070], 80.00th=[ 1200], 90.00th=[ 1368], 95.00th=[ 1653], | 99.00th=[ 2232], 99.50th=[ 2366], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 7, max= 103, per=25.16%, avg=46.05, stdev=19.91, samples=224 iops : min= 1, max= 25, avg=11.02, stdev= 5.00, samples=224 lat (msec) : 50=0.04%, 100=0.35%, 250=1.33%, 500=37.66%, 750=23.09% lat (msec) : 1000=18.71%, 2000=17.62%, >=2000=1.21% cpu : usr=0.10%, sys=0.59%, ctx=4241, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146746: Fri Aug 14 04:10:35 2026 read: IOPS=10, BW=43.7KiB/s (44.8kB/s)(4984KiB/113969msec) slat (usec): min=90, max=64078, avg=15574.24, stdev=9018.45 clat (msec): min=23, max=1074, avg=442.24, stdev=144.45 lat (msec): min=25, max=1100, avg=457.82, stdev=146.39 clat percentiles (msec): | 1.00th=[ 199], 5.00th=[ 275], 10.00th=[ 300], 20.00th=[ 334], | 30.00th=[ 359], 40.00th=[ 384], 50.00th=[ 409], 60.00th=[ 435], | 70.00th=[ 477], 80.00th=[ 542], 90.00th=[ 659], 95.00th=[ 726], | 99.00th=[ 894], 99.50th=[ 953], 99.90th=[ 1062], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 111, per=24.95%, avg=43.66, stdev=21.75, samples=224 iops : min= 1, max= 27, avg=10.42, stdev= 5.46, samples=224 write: IOPS=11, BW=46.1KiB/s (47.2kB/s)(5256KiB/113969msec); 0 zone resets slat (usec): min=57, max=39246, avg=388.31, stdev=1727.35 clat (msec): min=132, max=2487, avg=952.56, stdev=347.72 lat (msec): min=133, max=2487, avg=952.95, stdev=347.74 clat percentiles (msec): | 1.00th=[ 355], 5.00th=[ 514], 10.00th=[ 600], 20.00th=[ 684], | 30.00th=[ 751], 40.00th=[ 818], 50.00th=[ 894], 60.00th=[ 969], | 70.00th=[ 1062], 80.00th=[ 1183], 90.00th=[ 1368], 95.00th=[ 1636], | 99.00th=[ 2165], 99.50th=[ 2333], 99.90th=[ 2467], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min= 7, max= 111, per=24.89%, avg=45.56, stdev=19.87, samples=225 iops : min= 1, max= 27, avg=10.89, stdev= 5.01, samples=225 lat (msec) : 50=0.20%, 100=0.20%, 250=1.09%, 500=36.33%, 750=24.30% lat (msec) : 1000=19.45%, 2000=17.54%, >=2000=0.90% cpu : usr=0.09%, sys=0.68%, ctx=4492, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=146747: Fri Aug 14 04:10:35 2026 read: IOPS=11, BW=44.4KiB/s (45.5kB/s)(5060KiB/113968msec) slat (usec): min=2190, max=66982, avg=15451.99, stdev=9348.58 clat (msec): min=19, max=1086, avg=440.33, stdev=135.40 lat (msec): min=33, max=1095, avg=455.79, stdev=137.61 clat percentiles (msec): | 1.00th=[ 228], 5.00th=[ 271], 10.00th=[ 300], 20.00th=[ 334], | 30.00th=[ 359], 40.00th=[ 384], 50.00th=[ 414], 60.00th=[ 443], | 70.00th=[ 485], 80.00th=[ 542], 90.00th=[ 642], 95.00th=[ 701], | 99.00th=[ 852], 99.50th=[ 919], 99.90th=[ 1053], 99.95th=[ 1083], | 99.99th=[ 1083] bw ( KiB/s): min= 7, max= 119, per=25.45%, avg=44.53, stdev=21.25, samples=223 iops : min= 1, max= 29, avg=10.64, stdev= 5.34, samples=223 write: IOPS=11, BW=45.5KiB/s (46.5kB/s)(5180KiB/113968msec); 0 zone resets slat (usec): min=43, max=15348, avg=297.88, stdev=797.83 clat (msec): min=332, max=2418, avg=960.44, stdev=354.12 lat (msec): min=332, max=2418, avg=960.74, stdev=354.16 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 510], 10.00th=[ 592], 20.00th=[ 684], | 30.00th=[ 751], 40.00th=[ 827], 50.00th=[ 894], 60.00th=[ 969], | 70.00th=[ 1053], 80.00th=[ 1200], 90.00th=[ 1401], 95.00th=[ 1670], | 99.00th=[ 2140], 99.50th=[ 2265], 99.90th=[ 2400], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 7, max= 119, per=24.66%, avg=45.12, stdev=20.00, samples=224 iops : min= 1, max= 29, avg=10.79, stdev= 5.06, samples=224 lat (msec) : 20=0.04%, 100=0.04%, 250=1.13%, 500=37.38%, 750=24.34% lat (msec) : 1000=18.32%, 2000=17.89%, >=2000=0.86% cpu : usr=0.10%, sys=0.63%, ctx=4352, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=176KiB/s (180kB/s), 43.6KiB/s-44.4KiB/s (44.6kB/s-45.5kB/s), io=19.6MiB (20.5MB), run=113968-114145msec WRITE: bw=183KiB/s (187kB/s), 45.5KiB/s-46.1KiB/s (46.5kB/s-47.2kB/s), io=20.4MiB (21.4MB), run=113968-114145msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=146767: Fri Aug 14 04:10:45 2026 read: IOPS=1, BW=57.1MiB/s (59.9MB/s)(40.0MiB/700msec) slat (nsec): min=32934k, max=32934k, avg=32934175.00, stdev= 0.00 clat (nsec): min=644425k, max=644425k, avg=644425101.00, stdev= 0.00 lat (nsec): min=677364k, max=677364k, avg=677364258.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 642], 5.00th=[ 642], 10.00th=[ 642], 20.00th=[ 642], | 30.00th=[ 642], 40.00th=[ 642], 50.00th=[ 642], 60.00th=[ 642], | 70.00th=[ 642], 80.00th=[ 642], 90.00th=[ 642], 95.00th=[ 642], | 99.00th=[ 642], 99.50th=[ 642], 99.90th=[ 642], 99.95th=[ 642], | 99.99th=[ 642] lat (msec) : 750=100.00% cpu : usr=0.00%, sys=1.43%, ctx=14, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=57.1MiB/s (59.9MB/s), 57.1MiB/s-57.1MiB/s (59.9MB/s-59.9MB/s), io=40.0MiB (41.9MB), run=700-700msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout