== sanity test 398c: run fio to test AIO ================= 03:20:01 (1726125601) /usr/bin/fio debug=0 40+0 records in 40+0 records out 41943040 bytes (42 MB) copied, 0.12977 s, 323 MB/s osc.lustre-OST0000-osc-ffff880092cb6000.rpc_stats=clear writing 40M to OST0 by fio with 4 jobs... rand-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.7 Starting 4 processes rand-write: (groupid=0, jobs=1): err= 0: pid=8045: Thu Sep 12 03:20:28 2024 write: IOPS=220, BW=883KiB/s (904kB/s)(10.0MiB/11593msec) slat (usec): min=31, max=8645.1k, avg=3426.43, stdev=170861.93 clat (msec): min=4, max=8645, avg=21.61, stdev=170.67 lat (msec): min=4, max=8664, avg=25.04, stdev=241.54 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 11], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 21], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 24], 90.00th=[ 27], 95.00th=[ 29], | 99.00th=[ 36], 99.50th=[ 38], 99.90th=[ 47], 99.95th=[ 47], | 99.99th=[ 8658] bw ( KiB/s): min= 2008, max= 3696, per=100.00%, avg=3290.83, stdev=635.92, samples=6 iops : min= 502, max= 924, avg=822.67, stdev=158.96, samples=6 lat (msec) : 10=11.37%, 20=37.89%, 50=50.70% cpu : usr=0.00%, sys=1.48%, ctx=538, majf=0, minf=30 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=8046: Thu Sep 12 03:20:28 2024 write: IOPS=145, BW=584KiB/s (598kB/s)(10.0MiB/17541msec) slat (usec): min=16, max=14731k, avg=5809.52, stdev=291151.70 clat (msec): min=3, max=14750, avg=103.62, stdev=1124.02 lat (msec): min=3, max=14750, avg=109.43, stdev=1160.68 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 9], 20.00th=[ 13], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 23], 90.00th=[ 25], 95.00th=[ 27], | 99.00th=[ 32], 99.50th=[14697], 99.90th=[14697], 99.95th=[14697], | 99.99th=[14697] bw ( KiB/s): min= 1976, max= 3840, per=100.00%, avg=3305.33, stdev=696.02, samples=6 iops : min= 494, max= 960, avg=826.33, stdev=174.00, samples=6 lat (msec) : 4=1.05%, 10=12.93%, 20=51.48%, 50=33.95% cpu : usr=0.00%, sys=1.13%, ctx=1727, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=8047: Thu Sep 12 03:20:28 2024 write: IOPS=101, BW=407KiB/s (417kB/s)(10.0MiB/25134msec) slat (usec): min=16, max=17974k, avg=8741.52, stdev=361732.83 clat (usec): min=1804, max=21481k, avg=148179.71, stdev=1621539.50 lat (msec): min=2, max=21482, avg=156.92, stdev=1675.45 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 20], 60.00th=[ 22], | 70.00th=[ 26], 80.00th=[ 31], 90.00th=[ 40], 95.00th=[ 41], | 99.00th=[ 71], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 8, max= 3504, per=100.00%, avg=2211.33, stdev=1313.19, samples=9 iops : min= 2, max= 876, avg=552.67, stdev=328.49, samples=9 lat (msec) : 2=0.08%, 4=0.82%, 10=3.91%, 20=47.03%, 50=45.82% lat (msec) : 100=1.72% cpu : usr=0.04%, sys=3.82%, ctx=1342, majf=0, minf=32 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 rand-write: (groupid=0, jobs=1): err= 0: pid=8048: Thu Sep 12 03:20:28 2024 write: IOPS=118, BW=474KiB/s (486kB/s)(10.0MiB/21586msec) slat (usec): min=16, max=19339k, avg=7627.65, stdev=382227.12 clat (usec): min=1530, max=19368k, avg=127135.87, stdev=1476810.24 lat (usec): min=1573, max=19397k, avg=134764.95, stdev=1525126.05 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 6], 20.00th=[ 7], | 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 11], 60.00th=[ 13], | 70.00th=[ 17], 80.00th=[ 20], 90.00th=[ 31], 95.00th=[ 35], | 99.00th=[ 57], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113], | 99.99th=[17113] bw ( KiB/s): min= 1856, max= 7208, per=100.00%, avg=3712.40, stdev=2047.92, samples=5 iops : min= 464, max= 1802, avg=928.00, stdev=512.03, samples=5 lat (msec) : 2=0.51%, 4=6.72%, 10=38.98%, 20=35.51%, 50=17.07% lat (msec) : 100=0.62% cpu : usr=0.01%, sys=1.28%, ctx=1747, majf=0, minf=31 IO depths : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=99.4%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2560,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=1630KiB/s (1669kB/s), 407KiB/s-883KiB/s (417kB/s-904kB/s), io=40.0MiB (41.9MB), run=11593-25134msec osc.lustre-OST0000-osc-ffff880092cb6000.rpc_stats= snapshot_time: 1726125628.658715498 secs.nsecs start_time: 1726125602.504577592 secs.nsecs elapsed_time: 26.154137906 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 0 0 0 | 4103 68 68 2: 0 0 0 | 1021 17 85 4: 0 0 0 | 654 10 96 8: 0 0 0 | 147 2 98 16: 0 0 0 | 61 1 99 32: 0 0 0 | 7 0 99 64: 0 0 0 | 6 0 100 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 0 0 0 | 38 0 0 2: 0 0 0 | 250 4 4 3: 0 0 0 | 7 0 4 4: 0 0 0 | 13 0 5 5: 0 0 0 | 51 0 5 6: 0 0 0 | 137 2 8 7: 0 0 0 | 335 5 13 8: 0 0 0 | 844 14 27 9: 0 0 0 | 4316 71 99 10: 0 0 0 | 8 0 100 read write offset rpcs % cum % | rpcs % cum % 0: 0 0 0 | 4 0 0 1: 0 0 0 | 4 0 0 2: 0 0 0 | 8 0 0 4: 0 0 0 | 15 0 0 8: 0 0 0 | 32 0 1 16: 0 0 0 | 63 1 2 32: 0 0 0 | 123 2 4 64: 0 0 0 | 239 3 8 128: 0 0 0 | 420 7 15 256: 0 0 0 | 759 12 27 512: 0 0 0 | 1347 22 50 1024: 0 0 0 | 2111 35 85 2048: 0 0 0 | 874 14 100 sanity test_398c: @@@@@@ FAIL: 68% of I/O are 1-page Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7177:error() = /home/green/git/lustre-release/lustre/tests/sanity.sh:28787:test_398c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7522:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7585:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7393:run_test() = /home/green/git/lustre-release/lustre/tests/sanity.sh:28807:main() Dumping lctl log to /tmp/testlogs//sanity.test_398c.*.1726125629.log rsync: chown "/tmp/testlogs/.sanity.test_398c.debug_log.oleg458-server.1726125629.log.VamGS0" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity.test_398c.dmesg.oleg458-server.1726125629.log.5JJxEP" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]