== sanity test 398c: run fio to test AIO ================= 04:36:09 (1787387769) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 0.538567 s, 77.9 MB/s osc.lustre-OST0000-osc-ffff9703735d9000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=179907: Sat Aug 22 04:36:53 2026 write: IOPS=76, BW=305KiB/s (313kB/s)(10.0MiB/33535msec); 0 zone resets slat (usec): min=52, max=16271, avg=363.65, stdev=1015.52 clat (msec): min=22, max=352, avg=208.82, stdev=55.56 lat (msec): min=22, max=352, avg=209.19, stdev=55.56 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 111], 10.00th=[ 136], 20.00th=[ 165], | 30.00th=[ 182], 40.00th=[ 197], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 255], 90.00th=[ 284], 95.00th=[ 300], | 99.00th=[ 338], 99.50th=[ 338], 99.90th=[ 351], 99.95th=[ 351], | 99.99th=[ 351] bw ( KiB/s): min= 166, max= 582, per=24.97%, avg=304.36, stdev=76.52, samples=66 iops : min= 41, max= 145, avg=75.68, stdev=19.07, samples=66 lat (msec) : 50=0.08%, 100=1.84%, 250=74.65%, 500=23.44% cpu : usr=0.20%, sys=1.17%, ctx=2567, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179908: Sat Aug 22 04:36:53 2026 write: IOPS=76, BW=305KiB/s (312kB/s)(10.0MiB/33584msec); 0 zone resets slat (usec): min=49, max=23461, avg=393.45, stdev=1090.44 clat (msec): min=64, max=360, avg=208.60, stdev=53.89 lat (msec): min=64, max=360, avg=208.99, stdev=53.87 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 114], 10.00th=[ 136], 20.00th=[ 165], | 30.00th=[ 182], 40.00th=[ 194], 50.00th=[ 209], 60.00th=[ 224], | 70.00th=[ 236], 80.00th=[ 255], 90.00th=[ 279], 95.00th=[ 296], | 99.00th=[ 330], 99.50th=[ 347], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 359] bw ( KiB/s): min= 168, max= 566, per=24.87%, avg=303.15, stdev=71.10, samples=66 iops : min= 42, max= 141, avg=75.36, stdev=17.70, samples=66 lat (msec) : 100=2.62%, 250=75.55%, 500=21.84% cpu : usr=0.18%, sys=1.21%, ctx=2626, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179909: Sat Aug 22 04:36:53 2026 write: IOPS=76, BW=305KiB/s (312kB/s)(10.0MiB/33580msec); 0 zone resets slat (usec): min=52, max=18757, avg=354.99, stdev=1046.63 clat (msec): min=31, max=355, avg=208.16, stdev=55.59 lat (msec): min=31, max=356, avg=208.51, stdev=55.61 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 110], 10.00th=[ 138], 20.00th=[ 165], | 30.00th=[ 180], 40.00th=[ 190], 50.00th=[ 207], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 257], 90.00th=[ 284], 95.00th=[ 300], | 99.00th=[ 330], 99.50th=[ 338], 99.90th=[ 347], 99.95th=[ 355], | 99.99th=[ 355] bw ( KiB/s): min= 136, max= 510, per=24.83%, avg=302.65, stdev=73.07, samples=66 iops : min= 34, max= 127, avg=75.26, stdev=18.15, samples=66 lat (msec) : 50=0.12%, 100=2.97%, 250=74.14%, 500=22.77% cpu : usr=0.13%, sys=1.05%, ctx=2241, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=179910: Sat Aug 22 04:36:53 2026 write: IOPS=76, BW=305KiB/s (312kB/s)(10.0MiB/33590msec); 0 zone resets slat (usec): min=53, max=37861, avg=414.49, stdev=1300.34 clat (msec): min=38, max=361, avg=208.25, stdev=55.99 lat (msec): min=38, max=361, avg=208.67, stdev=55.93 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 112], 10.00th=[ 134], 20.00th=[ 163], | 30.00th=[ 180], 40.00th=[ 192], 50.00th=[ 207], 60.00th=[ 224], | 70.00th=[ 239], 80.00th=[ 257], 90.00th=[ 279], 95.00th=[ 305], | 99.00th=[ 334], 99.50th=[ 342], 99.90th=[ 359], 99.95th=[ 363], | 99.99th=[ 363] bw ( KiB/s): min= 160, max= 574, per=24.79%, avg=302.21, stdev=76.67, samples=66 iops : min= 40, max= 143, avg=75.14, stdev=19.07, samples=66 lat (msec) : 50=0.31%, 100=2.23%, 250=74.49%, 500=22.97% cpu : usr=0.19%, sys=1.32%, ctx=3021, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1219KiB/s (1249kB/s), 305KiB/s-305KiB/s (312kB/s-313kB/s), io=40.0MiB (41.9MB), run=33535-33590msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.19 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=179925: Sat Aug 22 04:37:42 2026 read: IOPS=31, BW=124KiB/s (127kB/s)(5048KiB/40550msec) slat (usec): min=56, max=86929, avg=17620.47, stdev=10220.24 clat (msec): min=22, max=467, avg=240.57, stdev=57.92 lat (msec): min=35, max=501, avg=258.20, stdev=60.29 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 153], 10.00th=[ 171], 20.00th=[ 194], | 30.00th=[ 211], 40.00th=[ 222], 50.00th=[ 234], 60.00th=[ 247], | 70.00th=[ 266], 80.00th=[ 288], 90.00th=[ 317], 95.00th=[ 347], | 99.00th=[ 401], 99.50th=[ 418], 99.90th=[ 464], 99.95th=[ 468], | 99.99th=[ 468] bw ( KiB/s): min= 39, max= 190, per=25.03%, avg=123.65, stdev=31.13, samples=80 iops : min= 9, max= 47, avg=30.44, stdev= 7.84, samples=80 write: IOPS=32, BW=128KiB/s (131kB/s)(5192KiB/40550msec); 0 zone resets slat (usec): min=50, max=31604, avg=377.04, stdev=1485.21 clat (msec): min=24, max=460, avg=246.12, stdev=57.57 lat (msec): min=25, max=460, avg=246.50, stdev=57.58 clat percentiles (msec): | 1.00th=[ 129], 5.00th=[ 159], 10.00th=[ 180], 20.00th=[ 201], | 30.00th=[ 215], 40.00th=[ 230], 50.00th=[ 243], 60.00th=[ 255], | 70.00th=[ 271], 80.00th=[ 288], 90.00th=[ 321], 95.00th=[ 351], | 99.00th=[ 414], 99.50th=[ 430], 99.90th=[ 447], 99.95th=[ 460], | 99.99th=[ 460] bw ( KiB/s): min= 63, max= 247, per=24.79%, avg=127.43, stdev=37.22, samples=80 iops : min= 15, max= 61, avg=31.39, stdev= 9.30, samples=80 lat (msec) : 50=0.08%, 250=58.44%, 500=41.48% cpu : usr=0.20%, sys=1.67%, ctx=4407, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179926: Sat Aug 22 04:37:42 2026 read: IOPS=30, BW=123KiB/s (126kB/s)(4976KiB/40353msec) slat (usec): min=3063, max=81985, avg=17574.99, stdev=10144.18 clat (msec): min=112, max=545, avg=236.38, stdev=56.57 lat (msec): min=123, max=563, avg=253.97, stdev=59.06 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 150], 10.00th=[ 171], 20.00th=[ 190], | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 230], 60.00th=[ 247], | 70.00th=[ 262], 80.00th=[ 279], 90.00th=[ 309], 95.00th=[ 334], | 99.00th=[ 397], 99.50th=[ 451], 99.90th=[ 489], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min= 27, max= 231, per=24.99%, avg=123.43, stdev=34.05, samples=79 iops : min= 6, max= 57, avg=30.37, stdev= 8.57, samples=79 write: IOPS=32, BW=130KiB/s (134kB/s)(5264KiB/40353msec); 0 zone resets slat (usec): min=57, max=20166, avg=352.70, stdev=1228.79 clat (msec): min=118, max=515, avg=246.46, stdev=56.60 lat (msec): min=119, max=515, avg=246.82, stdev=56.55 clat percentiles (msec): | 1.00th=[ 131], 5.00th=[ 167], 10.00th=[ 182], 20.00th=[ 201], | 30.00th=[ 213], 40.00th=[ 228], 50.00th=[ 241], 60.00th=[ 257], | 70.00th=[ 271], 80.00th=[ 292], 90.00th=[ 317], 95.00th=[ 351], | 99.00th=[ 418], 99.50th=[ 426], 99.90th=[ 464], 99.95th=[ 514], | 99.99th=[ 514] bw ( KiB/s): min= 55, max= 230, per=25.46%, avg=130.89, stdev=33.85, samples=79 iops : min= 13, max= 57, avg=32.23, stdev= 8.48, samples=79 lat (msec) : 250=59.26%, 500=40.66%, 750=0.08% cpu : usr=0.23%, sys=1.69%, ctx=4583, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179927: Sat Aug 22 04:37:42 2026 read: IOPS=30, BW=123KiB/s (126kB/s)(4984KiB/40569msec) slat (usec): min=3103, max=79754, avg=17514.01, stdev=9905.61 clat (msec): min=27, max=453, avg=239.73, stdev=56.01 lat (msec): min=37, max=462, avg=257.25, stdev=58.29 clat percentiles (msec): | 1.00th=[ 134], 5.00th=[ 157], 10.00th=[ 176], 20.00th=[ 194], | 30.00th=[ 209], 40.00th=[ 222], 50.00th=[ 234], 60.00th=[ 247], | 70.00th=[ 262], 80.00th=[ 284], 90.00th=[ 313], 95.00th=[ 347], | 99.00th=[ 397], 99.50th=[ 418], 99.90th=[ 430], 99.95th=[ 456], | 99.99th=[ 456] bw ( KiB/s): min= 47, max= 230, per=24.87%, avg=122.85, stdev=35.62, samples=80 iops : min= 11, max= 57, avg=30.22, stdev= 8.91, samples=80 write: IOPS=32, BW=130KiB/s (133kB/s)(5256KiB/40569msec); 0 zone resets slat (usec): min=55, max=17395, avg=310.88, stdev=917.99 clat (msec): min=111, max=445, avg=246.25, stdev=54.99 lat (msec): min=111, max=445, avg=246.57, stdev=55.04 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 165], 10.00th=[ 180], 20.00th=[ 203], | 30.00th=[ 215], 40.00th=[ 230], 50.00th=[ 241], 60.00th=[ 255], | 70.00th=[ 271], 80.00th=[ 288], 90.00th=[ 317], 95.00th=[ 342], | 99.00th=[ 405], 99.50th=[ 426], 99.90th=[ 443], 99.95th=[ 447], | 99.99th=[ 447] bw ( KiB/s): min= 56, max= 199, per=25.03%, avg=128.64, stdev=34.13, samples=80 iops : min= 14, max= 49, avg=31.68, stdev= 8.60, samples=80 lat (msec) : 50=0.04%, 250=59.22%, 500=40.74% cpu : usr=0.17%, sys=1.53%, ctx=4460, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=179928: Sat Aug 22 04:37:42 2026 read: IOPS=31, BW=126KiB/s (129kB/s)(5060KiB/40071msec) slat (usec): min=86, max=96902, avg=17526.36, stdev=10168.76 clat (msec): min=14, max=481, avg=238.87, stdev=58.23 lat (msec): min=14, max=498, avg=256.40, stdev=60.47 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 153], 10.00th=[ 171], 20.00th=[ 194], | 30.00th=[ 209], 40.00th=[ 222], 50.00th=[ 234], 60.00th=[ 247], | 70.00th=[ 264], 80.00th=[ 284], 90.00th=[ 313], 95.00th=[ 334], | 99.00th=[ 409], 99.50th=[ 435], 99.90th=[ 472], 99.95th=[ 481], | 99.99th=[ 481] bw ( KiB/s): min= 56, max= 192, per=25.37%, avg=125.30, stdev=32.86, samples=79 iops : min= 14, max= 48, avg=30.85, stdev= 8.27, samples=79 write: IOPS=32, BW=129KiB/s (132kB/s)(5180KiB/40071msec); 0 zone resets slat (usec): min=57, max=19684, avg=368.78, stdev=1141.20 clat (msec): min=38, max=524, avg=243.24, stdev=58.91 lat (msec): min=39, max=524, avg=243.62, stdev=58.90 clat percentiles (msec): | 1.00th=[ 124], 5.00th=[ 157], 10.00th=[ 176], 20.00th=[ 199], | 30.00th=[ 213], 40.00th=[ 226], 50.00th=[ 239], 60.00th=[ 253], | 70.00th=[ 266], 80.00th=[ 279], 90.00th=[ 317], 95.00th=[ 359], | 99.00th=[ 435], 99.50th=[ 439], 99.90th=[ 485], 99.95th=[ 527], | 99.99th=[ 527] bw ( KiB/s): min= 55, max= 254, per=24.98%, avg=128.39, stdev=34.93, samples=79 iops : min= 13, max= 63, avg=31.62, stdev= 8.74, samples=79 lat (msec) : 20=0.08%, 50=0.12%, 100=0.12%, 250=59.45%, 500=40.20% lat (msec) : 750=0.04% cpu : usr=0.18%, sys=1.65%, ctx=4255, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=495KiB/s (507kB/s), 123KiB/s-126KiB/s (126kB/s-129kB/s), io=19.6MiB (20.5MB), run=40071-40569msec WRITE: bw=515KiB/s (527kB/s), 128KiB/s-130KiB/s (131kB/s-134kB/s), io=20.4MiB (21.4MB), run=40071-40569msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.19 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=179938: Sat Aug 22 04:37:52 2026 read: IOPS=2, BW=82.6MiB/s (86.7MB/s)(40.0MiB/484msec) slat (nsec): min=45012k, max=45012k, avg=45011667.00, stdev= 0.00 clat (nsec): min=403802k, max=403802k, avg=403802212.00, stdev= 0.00 lat (nsec): min=448818k, max=448818k, avg=448818122.00, stdev= 0.00 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 405], 10.00th=[ 405], 20.00th=[ 405], | 30.00th=[ 405], 40.00th=[ 405], 50.00th=[ 405], 60.00th=[ 405], | 70.00th=[ 405], 80.00th=[ 405], 90.00th=[ 405], 95.00th=[ 405], | 99.00th=[ 405], 99.50th=[ 405], 99.90th=[ 405], 99.95th=[ 405], | 99.99th=[ 405] lat (msec) : 500=100.00% cpu : usr=0.21%, sys=1.45%, ctx=8, majf=0, minf=7 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=82.6MiB/s (86.7MB/s), 82.6MiB/s-82.6MiB/s (86.7MB/s-86.7MB/s), io=40.0MiB (41.9MB), run=484-484msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout