== sanity test 398c: run fio to test AIO ================= 18:31:43 (1741905103) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.100939 s, 416 MB/s osc.lustre-OST0000-osc-ffff88009678a800.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=5418: Thu Mar 13 18:32:11 2025 write: IOPS=97, BW=391KiB/s (401kB/s)(10.0MiB/26169msec) slat (usec): min=12, max=443, avg=69.27, stdev=40.75 clat (msec): min=20, max=261, avg=163.34, stdev=31.36 lat (msec): min=20, max=261, avg=163.41, stdev=31.36 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 109], 10.00th=[ 125], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 159], 50.00th=[ 165], 60.00th=[ 174], | 70.00th=[ 178], 80.00th=[ 188], 90.00th=[ 201], 95.00th=[ 211], | 99.00th=[ 230], 99.50th=[ 241], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 262] bw ( KiB/s): min= 296, max= 688, per=25.02%, avg=390.85, stdev=58.84, samples=52 iops : min= 74, max= 172, avg=97.65, stdev=14.75, samples=52 lat (msec) : 50=0.55%, 100=3.16%, 250=95.94%, 500=0.35% cpu : usr=0.12%, sys=0.81%, ctx=2442, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=5419: Thu Mar 13 18:32:11 2025 write: IOPS=97, BW=391KiB/s (400kB/s)(10.0MiB/26221msec) slat (usec): min=12, max=366, avg=65.45, stdev=39.83 clat (msec): min=51, max=263, avg=163.57, stdev=29.49 lat (msec): min=51, max=263, avg=163.63, stdev=29.49 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 109], 10.00th=[ 128], 20.00th=[ 144], | 30.00th=[ 150], 40.00th=[ 159], 50.00th=[ 165], 60.00th=[ 171], | 70.00th=[ 178], 80.00th=[ 188], 90.00th=[ 201], 95.00th=[ 213], | 99.00th=[ 228], 99.50th=[ 232], 99.90th=[ 251], 99.95th=[ 262], | 99.99th=[ 264] bw ( KiB/s): min= 296, max= 552, per=24.86%, avg=388.29, stdev=50.55, samples=52 iops : min= 74, max= 138, avg=97.06, stdev=12.65, samples=52 lat (msec) : 100=2.66%, 250=97.19%, 500=0.16% cpu : usr=0.14%, sys=0.73%, ctx=2277, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=5420: Thu Mar 13 18:32:11 2025 write: IOPS=97, BW=391KiB/s (401kB/s)(10.0MiB/26181msec) slat (usec): min=12, max=470, avg=66.20, stdev=39.73 clat (msec): min=42, max=263, avg=163.37, stdev=29.71 lat (msec): min=43, max=263, avg=163.43, stdev=29.71 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 108], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 159], 50.00th=[ 165], 60.00th=[ 171], | 70.00th=[ 178], 80.00th=[ 186], 90.00th=[ 201], 95.00th=[ 209], | 99.00th=[ 228], 99.50th=[ 234], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 264] bw ( KiB/s): min= 303, max= 640, per=24.95%, avg=389.77, stdev=56.47, samples=52 iops : min= 75, max= 160, avg=97.38, stdev=14.16, samples=52 lat (msec) : 50=0.62%, 100=2.15%, 250=96.95%, 500=0.27% cpu : usr=0.15%, sys=0.74%, ctx=2252, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=5421: Thu Mar 13 18:32:11 2025 write: IOPS=97, BW=391KiB/s (400kB/s)(10.0MiB/26221msec) slat (usec): min=13, max=454, avg=66.56, stdev=38.12 clat (msec): min=49, max=251, avg=163.66, stdev=29.35 lat (msec): min=49, max=251, avg=163.73, stdev=29.35 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 111], 10.00th=[ 128], 20.00th=[ 142], | 30.00th=[ 153], 40.00th=[ 159], 50.00th=[ 165], 60.00th=[ 171], | 70.00th=[ 178], 80.00th=[ 188], 90.00th=[ 201], 95.00th=[ 209], | 99.00th=[ 226], 99.50th=[ 232], 99.90th=[ 251], 99.95th=[ 251], | 99.99th=[ 251] bw ( KiB/s): min= 296, max= 640, per=24.91%, avg=389.06, stdev=59.15, samples=52 iops : min= 74, max= 160, avg=97.25, stdev=14.81, samples=52 lat (msec) : 50=0.62%, 100=1.99%, 250=97.23%, 500=0.16% cpu : usr=0.16%, sys=0.72%, ctx=2408, majf=0, minf=29 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1562KiB/s (1600kB/s), 391KiB/s-391KiB/s (400kB/s-401kB/s), io=40.0MiB (41.9MB), run=26169-26221msec mix rw 40M to OST0 by fio with 4 jobs... rand-rw: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-rw: (groupid=0, jobs=1): err= 0: pid=5453: Thu Mar 13 18:32:27 2025 read: IOPS=82, BW=330KiB/s (338kB/s)(5048KiB/15280msec) slat (usec): min=658, max=3842, avg=1154.46, stdev=281.07 clat (usec): min=1270, max=125760, avg=63126.47, stdev=11723.41 lat (msec): min=2, max=126, avg=64.28, stdev=11.74 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 45], 10.00th=[ 51], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 61], 50.00th=[ 64], 60.00th=[ 67], | 70.00th=[ 70], 80.00th=[ 73], 90.00th=[ 78], 95.00th=[ 82], | 99.00th=[ 88], 99.50th=[ 94], 99.90th=[ 108], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 224, max= 423, per=25.13%, avg=329.77, stdev=52.62, samples=30 iops : min= 56, max= 105, avg=82.37, stdev=13.08, samples=30 write: IOPS=84, BW=340KiB/s (348kB/s)(5192KiB/15280msec) slat (usec): min=13, max=1432, avg=60.73, stdev=62.05 clat (msec): min=48, max=224, avg=125.49, stdev=31.79 lat (msec): min=48, max=224, avg=125.55, stdev=31.80 clat percentiles (msec): | 1.00th=[ 60], 5.00th=[ 79], 10.00th=[ 86], 20.00th=[ 99], | 30.00th=[ 108], 40.00th=[ 115], 50.00th=[ 124], 60.00th=[ 131], | 70.00th=[ 142], 80.00th=[ 153], 90.00th=[ 169], 95.00th=[ 182], | 99.00th=[ 207], 99.50th=[ 220], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 256, max= 408, per=24.82%, avg=339.10, stdev=32.18, samples=30 iops : min= 64, max= 102, avg=84.70, stdev= 8.05, samples=30 lat (msec) : 2=0.08%, 10=0.04%, 20=0.23%, 50=4.45%, 100=55.51% lat (msec) : 250=39.69% cpu : usr=0.22%, sys=1.68%, ctx=3169, majf=0, minf=34 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1262,1298,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5454: Thu Mar 13 18:32:27 2025 read: IOPS=81, BW=325KiB/s (333kB/s)(4976KiB/15289msec) slat (usec): min=31, max=2911, avg=1153.31, stdev=282.81 clat (usec): min=1410, max=106491, avg=63204.85, stdev=11717.57 lat (usec): min=1443, max=107436, avg=64360.12, stdev=11747.22 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 58], 40.00th=[ 61], 50.00th=[ 64], 60.00th=[ 67], | 70.00th=[ 69], 80.00th=[ 72], 90.00th=[ 78], 95.00th=[ 82], | 99.00th=[ 90], 99.50th=[ 94], 99.90th=[ 97], 99.95th=[ 107], | 99.99th=[ 107] bw ( KiB/s): min= 184, max= 464, per=24.75%, avg=324.77, stdev=59.29, samples=30 iops : min= 46, max= 116, avg=81.17, stdev=14.85, samples=30 write: IOPS=86, BW=344KiB/s (353kB/s)(5264KiB/15289msec) slat (usec): min=12, max=235, avg=56.03, stdev=27.26 clat (msec): min=15, max=232, avg=124.87, stdev=31.95 lat (msec): min=15, max=232, avg=124.93, stdev=31.95 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 75], 10.00th=[ 84], 20.00th=[ 100], | 30.00th=[ 110], 40.00th=[ 116], 50.00th=[ 125], 60.00th=[ 131], | 70.00th=[ 140], 80.00th=[ 153], 90.00th=[ 169], 95.00th=[ 182], | 99.00th=[ 201], 99.50th=[ 205], 99.90th=[ 222], 99.95th=[ 232], | 99.99th=[ 232] bw ( KiB/s): min= 280, max= 416, per=25.04%, avg=342.10, stdev=35.09, samples=30 iops : min= 70, max= 104, avg=85.50, stdev= 8.74, samples=30 lat (msec) : 2=0.12%, 10=0.12%, 20=0.27%, 50=5.00%, 100=53.75% lat (msec) : 250=40.74% cpu : usr=0.23%, sys=1.65%, ctx=3050, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1244,1316,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5455: Thu Mar 13 18:32:27 2025 read: IOPS=81, BW=326KiB/s (334kB/s)(4984KiB/15280msec) slat (usec): min=686, max=3040, avg=1156.03, stdev=261.23 clat (usec): min=1419, max=125739, avg=62660.75, stdev=11427.60 lat (msec): min=2, max=126, avg=63.82, stdev=11.44 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 58], 40.00th=[ 61], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 68], 80.00th=[ 71], 90.00th=[ 77], 95.00th=[ 81], | 99.00th=[ 88], 99.50th=[ 89], 99.90th=[ 93], 99.95th=[ 126], | 99.99th=[ 126] bw ( KiB/s): min= 208, max= 447, per=24.93%, avg=327.10, stdev=55.97, samples=30 iops : min= 52, max= 111, avg=81.70, stdev=13.92, samples=30 write: IOPS=85, BW=344KiB/s (352kB/s)(5256KiB/15280msec) slat (usec): min=12, max=267, avg=55.17, stdev=26.85 clat (msec): min=45, max=222, avg=125.27, stdev=30.60 lat (msec): min=45, max=222, avg=125.32, stdev=30.60 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 79], 10.00th=[ 88], 20.00th=[ 100], | 30.00th=[ 109], 40.00th=[ 116], 50.00th=[ 124], 60.00th=[ 131], | 70.00th=[ 140], 80.00th=[ 153], 90.00th=[ 167], 95.00th=[ 180], | 99.00th=[ 197], 99.50th=[ 203], 99.90th=[ 222], 99.95th=[ 222], | 99.99th=[ 222] bw ( KiB/s): min= 280, max= 400, per=25.04%, avg=342.03, stdev=32.31, samples=30 iops : min= 70, max= 100, avg=85.43, stdev= 8.12, samples=30 lat (msec) : 2=0.08%, 10=0.04%, 20=0.35%, 50=4.77%, 100=53.83% lat (msec) : 250=40.94% cpu : usr=0.29%, sys=1.60%, ctx=3097, majf=0, minf=33 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1246,1314,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-rw: (groupid=0, jobs=1): err= 0: pid=5456: Thu Mar 13 18:32:27 2025 read: IOPS=83, BW=332KiB/s (340kB/s)(5060KiB/15236msec) slat (usec): min=707, max=2961, avg=1152.56, stdev=276.33 clat (usec): min=1535, max=129346, avg=63186.18, stdev=11756.78 lat (msec): min=2, max=130, avg=64.34, stdev=11.77 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 46], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 57], 40.00th=[ 62], 50.00th=[ 64], 60.00th=[ 67], | 70.00th=[ 69], 80.00th=[ 72], 90.00th=[ 77], 95.00th=[ 81], | 99.00th=[ 93], 99.50th=[ 96], 99.90th=[ 129], 99.95th=[ 130], | 99.99th=[ 130] bw ( KiB/s): min= 232, max= 448, per=25.42%, avg=333.50, stdev=54.10, samples=30 iops : min= 58, max= 112, avg=83.30, stdev=13.49, samples=30 write: IOPS=84, BW=340KiB/s (348kB/s)(5180KiB/15236msec) slat (usec): min=13, max=367, avg=56.58, stdev=30.13 clat (msec): min=42, max=223, avg=125.07, stdev=31.18 lat (msec): min=42, max=223, avg=125.12, stdev=31.18 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 78], 10.00th=[ 86], 20.00th=[ 99], | 30.00th=[ 109], 40.00th=[ 116], 50.00th=[ 124], 60.00th=[ 131], | 70.00th=[ 140], 80.00th=[ 150], 90.00th=[ 169], 95.00th=[ 180], | 99.00th=[ 205], 99.50th=[ 213], 99.90th=[ 224], 99.95th=[ 224], | 99.99th=[ 224] bw ( KiB/s): min= 232, max= 440, per=24.84%, avg=339.37, stdev=41.99, samples=30 iops : min= 58, max= 110, avg=84.77, stdev=10.49, samples=30 lat (msec) : 2=0.08%, 10=0.08%, 20=0.20%, 50=4.61%, 100=55.35% lat (msec) : 250=39.69% cpu : usr=0.23%, sys=1.66%, ctx=3174, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1265,1295,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=1313KiB/s (1344kB/s), 325KiB/s-332KiB/s (333kB/s-340kB/s), io=19.6MiB (20.5MB), run=15236-15289msec WRITE: bw=1366KiB/s (1399kB/s), 340KiB/s-344KiB/s (348kB/s-353kB/s), io=20.4MiB (21.4MB), run=15236-15289msec AIO with large block size 40M rand-rw: (g=0): rw=randrw, bs=(R) 40.0MiB-40.0MiB, (W) 40.0MiB-40.0MiB, (T) 40.0MiB-40.0MiB, ioengine=libaio, iodepth=16 fio-3.7 Starting 1 process rand-rw: (groupid=0, jobs=1): err= 0: pid=5472: Thu Mar 13 18:32:29 2025 read: IOPS=9, BW=381MiB/s (399MB/s)(40.0MiB/105msec) slat (nsec): min=5372.4k, max=5372.4k, avg=5372385.00, stdev= 0.00 clat (nsec): min=95139k, max=95139k, avg=95139106.00, stdev= 0.00 lat (nsec): min=100524k, max=100524k, avg=100523580.00, stdev= 0.00 clat percentiles (usec): | 1.00th=[94897], 5.00th=[94897], 10.00th=[94897], 20.00th=[94897], | 30.00th=[94897], 40.00th=[94897], 50.00th=[94897], 60.00th=[94897], | 70.00th=[94897], 80.00th=[94897], 90.00th=[94897], 95.00th=[94897], | 99.00th=[94897], 99.50th=[94897], 99.90th=[94897], 99.95th=[94897], | 99.99th=[94897] lat (msec) : 100=100.00% cpu : usr=0.00%, sys=4.81%, ctx=6, majf=0, minf=29 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=381MiB/s (399MB/s), 381MiB/s-381MiB/s (399MB/s-399MB/s), io=40.0MiB (41.9MB), run=105-105msec debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout