== sanity test 398c: run fio to test AIO ================= 05:04:48 (1785402288) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.410529 s, 102 MB/s osc.lustre-OST0000-osc-ffff922fc91fb000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179212: Thu Jul 30 05:05:41 2026 write: IOPS=62, BW=250KiB/s (256kB/s)(10.0MiB/40992msec); 0 zone resets slat (usec): min=49, max=38871, avg=415.92, stdev=1927.33 clat (msec): min=51, max=566, avg=253.77, stdev=79.60 lat (msec): min=51, max=566, avg=254.19, stdev=79.70 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 190], | 30.00th=[ 203], 40.00th=[ 215], 50.00th=[ 232], 60.00th=[ 255], | 70.00th=[ 275], 80.00th=[ 313], 90.00th=[ 363], 95.00th=[ 414], | 99.00th=[ 498], 99.50th=[ 523], 99.90th=[ 567], 99.95th=[ 567], | 99.99th=[ 567] bw ( KiB/s): min= 125, max= 388, per=25.00%, avg=249.26, stdev=78.86, samples=81 iops : min= 31, max= 97, avg=61.91, stdev=19.74, samples=81 lat (msec) : 100=0.04%, 250=58.40%, 500=40.59%, 750=0.98% cpu : usr=0.16%, sys=0.97%, ctx=2081, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179213: Thu Jul 30 05:05:41 2026 write: IOPS=62, BW=249KiB/s (255kB/s)(10.0MiB/41050msec); 0 zone resets slat (usec): min=50, max=28423, avg=385.40, stdev=1532.22 clat (msec): min=43, max=561, avg=253.95, stdev=79.67 lat (msec): min=43, max=561, avg=254.34, stdev=79.77 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 157], 10.00th=[ 174], 20.00th=[ 190], | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 234], 60.00th=[ 253], | 70.00th=[ 284], 80.00th=[ 313], 90.00th=[ 368], 95.00th=[ 414], | 99.00th=[ 502], 99.50th=[ 523], 99.90th=[ 558], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min= 95, max= 388, per=25.29%, avg=252.19, stdev=72.91, samples=80 iops : min= 23, max= 97, avg=62.64, stdev=18.23, samples=80 lat (msec) : 50=0.12%, 100=0.16%, 250=58.32%, 500=40.23%, 750=1.17% cpu : usr=0.12%, sys=1.02%, ctx=2758, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179214: Thu Jul 30 05:05:41 2026 write: IOPS=63, BW=254KiB/s (260kB/s)(10.0MiB/40374msec); 0 zone resets slat (usec): min=51, max=34288, avg=424.15, stdev=1748.75 clat (msec): min=14, max=876, avg=251.64, stdev=85.34 lat (msec): min=14, max=876, avg=252.06, stdev=85.38 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 157], 10.00th=[ 167], 20.00th=[ 190], | 30.00th=[ 203], 40.00th=[ 218], 50.00th=[ 234], 60.00th=[ 251], | 70.00th=[ 271], 80.00th=[ 313], 90.00th=[ 359], 95.00th=[ 422], | 99.00th=[ 518], 99.50th=[ 567], 99.90th=[ 877], 99.95th=[ 877], | 99.99th=[ 877] bw ( KiB/s): min= 47, max= 384, per=25.30%, avg=252.27, stdev=77.30, samples=80 iops : min= 11, max= 96, avg=62.64, stdev=19.37, samples=80 lat (msec) : 20=0.04%, 50=0.08%, 100=0.23%, 250=59.73%, 500=38.36% lat (msec) : 750=1.37%, 1000=0.20% cpu : usr=0.16%, sys=0.88%, ctx=2339, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179215: Thu Jul 30 05:05:41 2026 write: IOPS=62, BW=250KiB/s (256kB/s)(10.0MiB/40986msec); 0 zone resets slat (usec): min=50, max=42626, avg=319.16, stdev=1355.61 clat (msec): min=48, max=553, avg=253.84, stdev=80.16 lat (msec): min=48, max=553, avg=254.16, stdev=80.18 clat percentiles (msec): | 1.00th=[ 125], 5.00th=[ 155], 10.00th=[ 169], 20.00th=[ 190], | 30.00th=[ 207], 40.00th=[ 218], 50.00th=[ 236], 60.00th=[ 255], | 70.00th=[ 279], 80.00th=[ 317], 90.00th=[ 368], 95.00th=[ 405], | 99.00th=[ 518], 99.50th=[ 527], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 120, max= 382, per=25.01%, avg=249.31, stdev=74.97, samples=81 iops : min= 30, max= 95, avg=61.93, stdev=18.76, samples=81 lat (msec) : 50=0.04%, 250=58.71%, 500=39.57%, 750=1.68% cpu : usr=0.13%, sys=0.88%, ctx=1942, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=998KiB/s (1022kB/s), 249KiB/s-254KiB/s (255kB/s-260kB/s), io=40.0MiB (41.9MB), run=40374-41050msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179233: Thu Jul 30 05:06:47 2026 read: IOPS=27, BW=112KiB/s (115kB/s)(5048KiB/45083msec) slat (usec): min=1518, max=106066, avg=20018.09, stdev=13363.60 clat (msec): min=58, max=649, avg=264.70, stdev=81.51 lat (msec): min=69, max=681, avg=284.72, stdev=85.86 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 199], | 30.00th=[ 215], 40.00th=[ 234], 50.00th=[ 255], 60.00th=[ 275], | 70.00th=[ 296], 80.00th=[ 330], 90.00th=[ 368], 95.00th=[ 409], | 99.00th=[ 527], 99.50th=[ 542], 99.90th=[ 625], 99.95th=[ 651], | 99.99th=[ 651] bw ( KiB/s): min= 17, max= 200, per=25.01%, avg=111.29, stdev=36.21, samples=89 iops : min= 4, max= 50, avg=27.34, stdev= 9.10, samples=89 write: IOPS=28, BW=115KiB/s (118kB/s)(5192KiB/45083msec); 0 zone resets slat (usec): min=54, max=17929, avg=280.38, stdev=802.85 clat (msec): min=63, max=726, avg=273.64, stdev=84.95 lat (msec): min=64, max=726, avg=273.92, stdev=84.98 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 159], 10.00th=[ 182], 20.00th=[ 207], | 30.00th=[ 224], 40.00th=[ 243], 50.00th=[ 262], 60.00th=[ 284], | 70.00th=[ 309], 80.00th=[ 334], 90.00th=[ 384], 95.00th=[ 426], | 99.00th=[ 518], 99.50th=[ 651], 99.90th=[ 726], 99.95th=[ 726], | 99.99th=[ 726] bw ( KiB/s): min= 47, max= 232, per=25.11%, avg=116.25, stdev=36.32, samples=88 iops : min= 11, max= 58, avg=28.58, stdev= 9.09, samples=88 lat (msec) : 100=0.82%, 250=44.80%, 500=53.05%, 750=1.33% cpu : usr=0.21%, sys=1.43%, ctx=3929, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179234: Thu Jul 30 05:06:47 2026 read: IOPS=27, BW=110KiB/s (113kB/s)(4976KiB/45089msec) slat (msec): min=2, max=154, avg=20.41, stdev=13.38 clat (msec): min=49, max=675, avg=265.95, stdev=81.89 lat (msec): min=63, max=682, avg=286.37, stdev=85.45 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 157], 10.00th=[ 176], 20.00th=[ 201], | 30.00th=[ 218], 40.00th=[ 239], 50.00th=[ 257], 60.00th=[ 275], | 70.00th=[ 296], 80.00th=[ 330], 90.00th=[ 372], 95.00th=[ 409], | 99.00th=[ 542], 99.50th=[ 575], 99.90th=[ 617], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 11, max= 240, per=24.65%, avg=109.70, stdev=34.48, samples=89 iops : min= 2, max= 60, avg=26.93, stdev= 8.67, samples=89 write: IOPS=29, BW=117KiB/s (120kB/s)(5264KiB/45089msec); 0 zone resets slat (usec): min=42, max=46508, avg=357.05, stdev=1596.61 clat (msec): min=62, max=750, avg=273.27, stdev=81.90 lat (msec): min=62, max=752, avg=273.63, stdev=81.96 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 165], 10.00th=[ 186], 20.00th=[ 209], | 30.00th=[ 228], 40.00th=[ 245], 50.00th=[ 262], 60.00th=[ 279], | 70.00th=[ 305], 80.00th=[ 338], 90.00th=[ 384], 95.00th=[ 409], | 99.00th=[ 523], 99.50th=[ 609], 99.90th=[ 667], 99.95th=[ 751], | 99.99th=[ 751] bw ( KiB/s): min= 29, max= 247, per=25.17%, avg=116.53, stdev=41.20, samples=89 iops : min= 7, max= 61, avg=28.64, stdev=10.28, samples=89 lat (msec) : 50=0.04%, 100=0.74%, 250=43.83%, 500=54.02%, 750=1.33% lat (msec) : 1000=0.04% cpu : usr=0.20%, sys=1.47%, ctx=4206, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179235: Thu Jul 30 05:06:47 2026 read: IOPS=27, BW=111KiB/s (113kB/s)(4984KiB/44991msec) slat (usec): min=2919, max=95926, avg=20040.90, stdev=12797.13 clat (msec): min=51, max=633, avg=266.22, stdev=82.54 lat (msec): min=61, max=691, avg=286.27, stdev=86.36 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 159], 10.00th=[ 176], 20.00th=[ 199], | 30.00th=[ 220], 40.00th=[ 236], 50.00th=[ 253], 60.00th=[ 271], | 70.00th=[ 296], 80.00th=[ 330], 90.00th=[ 380], 95.00th=[ 414], | 99.00th=[ 514], 99.50th=[ 609], 99.90th=[ 625], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 32, max= 192, per=25.26%, avg=112.41, stdev=35.71, samples=87 iops : min= 8, max= 48, avg=27.62, stdev= 8.96, samples=87 write: IOPS=29, BW=117KiB/s (120kB/s)(5256KiB/44991msec); 0 zone resets slat (usec): min=52, max=15268, avg=333.86, stdev=994.77 clat (msec): min=69, max=678, avg=270.75, stdev=81.74 lat (msec): min=69, max=679, avg=271.09, stdev=81.75 clat percentiles (msec): | 1.00th=[ 122], 5.00th=[ 165], 10.00th=[ 180], 20.00th=[ 207], | 30.00th=[ 224], 40.00th=[ 241], 50.00th=[ 259], 60.00th=[ 279], | 70.00th=[ 300], 80.00th=[ 334], 90.00th=[ 376], 95.00th=[ 422], | 99.00th=[ 502], 99.50th=[ 584], 99.90th=[ 676], 99.95th=[ 676], | 99.99th=[ 676] bw ( KiB/s): min= 17, max= 239, per=25.18%, avg=116.59, stdev=41.62, samples=88 iops : min= 4, max= 59, avg=28.66, stdev=10.42, samples=88 lat (msec) : 100=0.66%, 250=45.62%, 500=52.46%, 750=1.25% cpu : usr=0.22%, sys=1.59%, ctx=4446, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179236: Thu Jul 30 05:06:47 2026 read: IOPS=28, BW=112KiB/s (115kB/s)(5060KiB/44988msec) slat (usec): min=78, max=122261, avg=20583.93, stdev=13603.53 clat (msec): min=15, max=634, avg=266.63, stdev=81.78 lat (msec): min=54, max=684, avg=287.22, stdev=85.62 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 203], | 30.00th=[ 220], 40.00th=[ 236], 50.00th=[ 255], 60.00th=[ 275], | 70.00th=[ 300], 80.00th=[ 330], 90.00th=[ 372], 95.00th=[ 414], | 99.00th=[ 523], 99.50th=[ 567], 99.90th=[ 617], 99.95th=[ 634], | 99.99th=[ 634] bw ( KiB/s): min= 54, max= 232, per=25.25%, avg=112.36, stdev=33.08, samples=88 iops : min= 13, max= 58, avg=27.60, stdev= 8.29, samples=88 write: IOPS=28, BW=115KiB/s (118kB/s)(5180KiB/44988msec); 0 zone resets slat (usec): min=38, max=21493, avg=307.90, stdev=1084.71 clat (msec): min=67, max=688, avg=274.10, stdev=81.12 lat (msec): min=67, max=688, avg=274.41, stdev=81.13 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 167], 10.00th=[ 184], 20.00th=[ 209], | 30.00th=[ 226], 40.00th=[ 245], 50.00th=[ 259], 60.00th=[ 284], | 70.00th=[ 309], 80.00th=[ 334], 90.00th=[ 384], 95.00th=[ 422], | 99.00th=[ 510], 99.50th=[ 535], 99.90th=[ 693], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min= 39, max= 213, per=24.76%, avg=114.66, stdev=36.19, samples=88 iops : min= 9, max= 53, avg=28.18, stdev= 9.10, samples=88 lat (msec) : 20=0.04%, 100=0.74%, 250=44.88%, 500=53.12%, 750=1.21% cpu : usr=0.22%, sys=1.53%, ctx=4268, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=445KiB/s (456kB/s), 110KiB/s-112KiB/s (113kB/s-115kB/s), io=19.6MiB (20.5MB), run=44988-45089msec WRITE: bw=463KiB/s (474kB/s), 115KiB/s-117KiB/s (118kB/s-120kB/s), io=20.4MiB (21.4MB), run=44988-45089msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179246: Thu Jul 30 05:06:58 2026 read: IOPS=2, BW=80.0MiB/s (83.9MB/s)(40.0MiB/500msec) slat (nsec): min=36502k, max=36502k, avg=36501649.00, stdev= 0.00 clat (nsec): min=443965k, max=443965k, avg=443964753.00, stdev= 0.00 lat (nsec): min=480471k, max=480471k, avg=480471022.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 443], 10.00th=[ 443], 20.00th=[ 443], | 30.00th=[ 443], 40.00th=[ 443], 50.00th=[ 443], 60.00th=[ 443], | 70.00th=[ 443], 80.00th=[ 443], 90.00th=[ 443], 95.00th=[ 443], | 99.00th=[ 443], 99.50th=[ 443], 99.90th=[ 443], 99.95th=[ 443], | 99.99th=[ 443] lat (msec) : 500=100.00% cpu : usr=0.00%, sys=2.00%, ctx=9, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=80.0MiB/s (83.9MB/s), 80.0MiB/s-80.0MiB/s (83.9MB/s-83.9MB/s), io=40.0MiB (41.9MB), run=500-500msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout