== sanity test 123ad: Verify batching statahead works correctly ========================================================== 13:00:37 (1788454837) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff977f9f66a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff977f9f66a000.batch_stats=0 llite.lustre-ffff977f9f66a000.statahead_max=32 llite.lustre-ffff977f9f66a000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff977f9f66a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff977f9f66a000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 3.35 seconds: 29.88 ops/second llite.lustre-ffff977f9f66a000.statahead_max=0 101 real 0m2.195s user 0m0.004s sys 0m0.314s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff977f9f66a000.statahead_max=32 llite.lustre-ffff977f9f66a000.statahead_max=32 101 real 0m0.970s user 0m0.002s sys 0m0.181s 'ls -l' 100 files with statahead: 1 sec statahead total: 122 statahead wrong: 2 agl total: 117 list_total: 31 fname_total: 89 hit_total: 3805 miss_total: 1624 swrong=2 ewrong=2 - open/close 665 (time 1788454867.81 total 10.02 last 66.38) total: 900 open/close in 12.70 seconds: 70.86 ops/second llite.lustre-ffff977f9f66a000.statahead_max=0 1001 real 0m27.295s user 0m0.030s sys 0m2.992s 'ls -l' 1000 files without statahead: 27 sec llite.lustre-ffff977f9f66a000.statahead_max=32 llite.lustre-ffff977f9f66a000.statahead_max=32 1001 real 0m14.205s user 0m0.034s sys 0m1.400s 'ls -l' 1000 files with statahead: 14 sec statahead total: 123 statahead wrong: 2 agl total: 118 list_total: 32 fname_total: 89 hit_total: 4802 miss_total: 1627 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff977f9f66a000.statahead_stats= statahead total: 123 statahead wrong: 2 agl total: 118 list_total: 32 fname_total: 89 hit_total: 4802 miss_total: 1627 mdc.lustre-MDT0000-mdc-ffff977f9f66a000.batch_stats= snapshot_time: 1788454947.887034780 secs.nsecs start_time: 1788454842.872644311 secs.nsecs elapsed_time: 105.014390469 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 5 5 8: 5 12 17 16: 16 41 58 32: 16 41 100 mdc.lustre-MDT0001-mdc-ffff977f9f66a000.batch_stats= snapshot_time: 1788454947.887157251 secs.nsecs start_time: 1788454842.872779661 secs.nsecs elapsed_time: 105.014377590 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 1 2 7 8: 3 7 15 16: 20 51 66 32: 13 33 100 - open/close 810 (time 1788454960.77 total 10.01 last 80.95) - open/close 1565 (time 1788454970.78 total 20.02 last 75.42) - open/close 2304 (time 1788454980.78 total 30.02 last 73.90) - open/close 3013 (time 1788454990.80 total 40.04 last 70.77) - open/close 3908 (time 1788455000.81 total 50.04 last 89.45) - open/close 4760 (time 1788455010.82 total 60.05 last 85.13) - open/close 5472 (time 1788455020.82 total 70.05 last 71.19) - open/close 6114 (time 1788455030.82 total 80.05 last 64.19) - open/close 6502 (time 1788455040.84 total 90.07 last 38.72) - open/close 7088 (time 1788455050.85 total 100.08 last 58.56) - open/close 7898 (time 1788455060.85 total 110.09 last 80.95) - open/close 8707 (time 1788455070.86 total 120.10 last 80.82) - open/close 9492 (time 1788455080.87 total 130.10 last 78.48) total: 10000 open/close in 136.16 seconds: 73.44 ops/second llite.lustre-ffff977f9f66a000.statahead_batch_max=0 llite.lustre-ffff977f9f66a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff977f9f66a000.stats=clear mdc.lustre-MDT0001-mdc-ffff977f9f66a000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m12.398s user 0m0.309s sys 0m15.002s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff977f9f66a000.statahead_batch_max=32 llite.lustre-ffff977f9f66a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff977f9f66a000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff977f9f66a000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff977f9f66a000.stats=clear mdc.lustre-MDT0001-mdc-ffff977f9f66a000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m5.976s user 0m0.326s sys 0m14.635s batched statahead RPC= hit count=9975 mdc.lustre-MDT0000-mdc-ffff977f9f66a000.batch_stats= snapshot_time: 1788455307.772670645 secs.nsecs start_time: 1788455168.952387938 secs.nsecs elapsed_time: 138.820282707 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff977f9f66a000.batch_stats= snapshot_time: 1788455307.772737388 secs.nsecs start_time: 1788455168.952529791 secs.nsecs elapsed_time: 138.820207597 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff977f9f66a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff977f9f66a000.batch_stats=0 llite.lustre-ffff977f9f66a000.statahead_max=2048 llite.lustre-ffff977f9f66a000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff977f9f66a000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff977f9f66a000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.93 seconds: 34.11 ops/second llite.lustre-ffff977f9f66a000.statahead_max=0 101 real 0m2.579s user 0m0.004s sys 0m0.332s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff977f9f66a000.statahead_max=2048 llite.lustre-ffff977f9f66a000.statahead_max=2048 101 real 0m1.387s user 0m0.004s sys 0m0.170s 'ls -l' 100 files with statahead: 2 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10074 miss_total: 26 swrong=0 ewrong=0 - open/close 700 (time 1788455639.65 total 10.04 last 69.69) total: 900 open/close in 13.67 seconds: 65.83 ops/second llite.lustre-ffff977f9f66a000.statahead_max=0 1001 real 0m22.669s user 0m0.026s sys 0m2.788s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff977f9f66a000.statahead_max=2048 llite.lustre-ffff977f9f66a000.statahead_max=2048 1001 real 0m7.752s user 0m0.032s sys 0m1.381s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11073 miss_total: 27 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 25 seconds llite.lustre-ffff977f9f66a000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11073 miss_total: 27 mdc.lustre-MDT0000-mdc-ffff977f9f66a000.batch_stats= snapshot_time: 1788455716.393354548 secs.nsecs start_time: 1788455311.026610906 secs.nsecs elapsed_time: 405.366743642 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 28 28 8: 2 28 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff977f9f66a000.batch_stats= snapshot_time: 1788455716.393452538 secs.nsecs start_time: 1788455311.026775864 secs.nsecs elapsed_time: 405.366676674 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 0 0 14 4: 1 14 28 8: 2 28 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 - open/close 668 (time 1788455729.26 total 10.00 last 66.77) - open/close 1379 (time 1788455739.26 total 20.01 last 71.07) - open/close 2127 (time 1788455749.27 total 30.01 last 74.77) - open/close 2751 (time 1788455759.28 total 40.02 last 62.34) - open/close 3420 (time 1788455769.28 total 50.02 last 66.88) - open/close 4234 (time 1788455779.28 total 60.03 last 81.38) - open/close 4998 (time 1788455789.29 total 70.03 last 76.37) - open/close 5767 (time 1788455799.29 total 80.03 last 76.89) - open/close 6349 (time 1788455809.30 total 90.04 last 58.16) - open/close 6901 (time 1788455819.30 total 100.05 last 55.16) - open/close 7474 (time 1788455829.31 total 110.06 last 57.25) - open/close 8257 (time 1788455839.32 total 120.06 last 78.27) - open/close 8977 (time 1788455849.34 total 130.08 last 71.86) - open/close 9723 (time 1788455859.36 total 140.10 last 74.42) total: 10000 open/close in 144.62 seconds: 69.15 ops/second llite.lustre-ffff977f9f66a000.statahead_batch_max=0 llite.lustre-ffff977f9f66a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff977f9f66a000.stats=clear mdc.lustre-MDT0001-mdc-ffff977f9f66a000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.092s user 0m0.301s sys 0m15.325s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff977f9f66a000.statahead_batch_max=256 llite.lustre-ffff977f9f66a000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff977f9f66a000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff977f9f66a000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff977f9f66a000.stats=clear mdc.lustre-MDT0001-mdc-ffff977f9f66a000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m53.976s user 0m0.281s sys 0m14.522s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff977f9f66a000.batch_stats= snapshot_time: 1788456001.604335278 secs.nsecs start_time: 1788455935.951266069 secs.nsecs elapsed_time: 65.653069209 secs.nsecs subreqs per batch batches %% cum %% 1: 105 39 39 2: 46 17 56 4: 25 9 66 8: 21 7 74 16: 12 4 78 32: 14 5 83 64: 8 3 86 128: 1 0 87 256: 34 12 100 mdc.lustre-MDT0001-mdc-ffff977f9f66a000.batch_stats= snapshot_time: 1788456001.604409238 secs.nsecs start_time: 1788455935.951443975 secs.nsecs elapsed_time: 65.652965263 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff977f9f66a000.statahead_batch_max=64 llite.lustre-ffff977f9f66a000.statahead_max=128