== sanity test 123ad: Verify batching statahead works correctly ========================================================== 05:12:14 (1789549934) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9099480e4000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9099480e4000.batch_stats=0 llite.lustre-ffff9099480e4000.statahead_max=32 llite.lustre-ffff9099480e4000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9099480e4000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9099480e4000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.14 seconds: 46.73 ops/second llite.lustre-ffff9099480e4000.statahead_max=0 101 real 0m2.164s user 0m0.003s sys 0m0.299s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9099480e4000.statahead_max=32 llite.lustre-ffff9099480e4000.statahead_max=32 101 real 0m1.045s user 0m0.006s sys 0m0.192s 'ls -l' 100 files with statahead: 1 sec statahead total: 83 statahead wrong: 1 agl total: 79 list_total: 28 fname_total: 40 hit_total: 4124 miss_total: 236 swrong=1 ewrong=1 - open/close 785 (time 1789549960.20 total 10.00 last 78.49) total: 900 open/close in 11.56 seconds: 77.87 ops/second llite.lustre-ffff9099480e4000.statahead_max=0 1001 real 0m20.713s user 0m0.030s sys 0m2.886s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff9099480e4000.statahead_max=32 llite.lustre-ffff9099480e4000.statahead_max=32 1001 real 0m7.670s user 0m0.031s sys 0m1.445s 'ls -l' 1000 files with statahead: 8 sec statahead total: 84 statahead wrong: 1 agl total: 80 list_total: 29 fname_total: 40 hit_total: 5121 miss_total: 239 swrong=1 ewrong=1 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff9099480e4000.statahead_stats= statahead total: 84 statahead wrong: 1 agl total: 80 list_total: 29 fname_total: 40 hit_total: 5121 miss_total: 239 mdc.lustre-MDT0000-mdc-ffff9099480e4000.batch_stats= snapshot_time: 1789550017.510271741 secs.nsecs start_time: 1789549939.022668985 secs.nsecs elapsed_time: 78.487602756 secs.nsecs subreqs per batch batches %% cum %% 1: 3 7 7 2: 2 5 13 4: 1 2 15 8: 3 7 23 16: 22 57 81 32: 7 18 100 mdc.lustre-MDT0001-mdc-ffff9099480e4000.batch_stats= snapshot_time: 1789550017.510348374 secs.nsecs start_time: 1789549939.023426876 secs.nsecs elapsed_time: 78.486921498 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 5 5 8: 2 5 10 16: 14 35 46 32: 21 53 100 - open/close 872 (time 1789550029.61 total 10.01 last 87.13) - open/close 1707 (time 1789550039.61 total 20.01 last 83.47) - open/close 2559 (time 1789550049.62 total 30.02 last 85.14) - open/close 3454 (time 1789550059.62 total 40.02 last 89.50) - open/close 4344 (time 1789550069.62 total 50.02 last 88.95) - open/close 5266 (time 1789550079.63 total 60.03 last 92.14) - open/close 6140 (time 1789550089.64 total 70.04 last 87.32) - open/close 7074 (time 1789550099.64 total 80.04 last 93.37) - open/close 7943 (time 1789550109.65 total 90.05 last 86.87) - open/close 8782 (time 1789550119.66 total 100.06 last 83.82) - open/close 9666 (time 1789550129.66 total 110.06 last 88.34) total: 10000 open/close in 113.75 seconds: 87.91 ops/second llite.lustre-ffff9099480e4000.statahead_batch_max=0 llite.lustre-ffff9099480e4000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9099480e4000.stats=clear mdc.lustre-MDT0001-mdc-ffff9099480e4000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m45.735s user 0m0.342s sys 0m17.961s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9099480e4000.statahead_batch_max=32 llite.lustre-ffff9099480e4000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9099480e4000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9099480e4000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9099480e4000.stats=clear mdc.lustre-MDT0001-mdc-ffff9099480e4000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m22.513s user 0m0.309s sys 0m14.488s batched statahead RPC= hit count=9975 mdc.lustre-MDT0000-mdc-ffff9099480e4000.batch_stats= snapshot_time: 1789550278.311540639 secs.nsecs start_time: 1789550186.070149847 secs.nsecs elapsed_time: 92.241390792 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9099480e4000.batch_stats= snapshot_time: 1789550278.311678114 secs.nsecs start_time: 1789550186.074077474 secs.nsecs elapsed_time: 92.237600640 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9099480e4000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9099480e4000.batch_stats=0 llite.lustre-ffff9099480e4000.statahead_max=2048 llite.lustre-ffff9099480e4000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9099480e4000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9099480e4000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.14 seconds: 46.70 ops/second llite.lustre-ffff9099480e4000.statahead_max=0 101 real 0m1.886s user 0m0.005s sys 0m0.297s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9099480e4000.statahead_max=2048 llite.lustre-ffff9099480e4000.statahead_max=2048 101 real 0m0.648s user 0m0.007s sys 0m0.174s 'ls -l' 100 files with statahead: 0 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10074 miss_total: 26 swrong=0 ewrong=0 total: 900 open/close in 9.40 seconds: 95.78 ops/second llite.lustre-ffff9099480e4000.statahead_max=0 1001 real 0m19.337s user 0m0.036s sys 0m2.862s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff9099480e4000.statahead_max=2048 llite.lustre-ffff9099480e4000.statahead_max=2048 1001 real 0m4.815s user 0m0.027s sys 0m1.363s 'ls -l' 1000 files with statahead: 4 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11073 miss_total: 27 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 13 seconds llite.lustre-ffff9099480e4000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11073 miss_total: 27 mdc.lustre-MDT0000-mdc-ffff9099480e4000.batch_stats= snapshot_time: 1789550579.343324024 secs.nsecs start_time: 1789550280.264058406 secs.nsecs elapsed_time: 299.079265618 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 22 22 4: 1 11 33 8: 1 11 44 16: 1 11 55 32: 0 0 55 64: 2 22 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff9099480e4000.batch_stats= snapshot_time: 1789550579.343396414 secs.nsecs start_time: 1789550280.264217868 secs.nsecs elapsed_time: 299.079178546 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 4 44 55 16: 0 0 55 32: 1 11 66 64: 1 11 77 128: 0 0 77 256: 2 22 100 - open/close 964 (time 1789550591.23 total 10.00 last 96.36) - open/close 1852 (time 1789550601.24 total 20.01 last 88.73) - open/close 2782 (time 1789550611.24 total 30.02 last 92.96) - open/close 3699 (time 1789550621.24 total 40.02 last 91.69) - open/close 4604 (time 1789550631.26 total 50.03 last 90.37) - open/close 5519 (time 1789550641.27 total 60.04 last 91.40) - open/close 6504 (time 1789550651.28 total 70.05 last 98.40) - open/close 7407 (time 1789550661.28 total 80.06 last 90.26) - open/close 8377 (time 1789550671.29 total 90.06 last 96.94) - open/close 9266 (time 1789550681.31 total 100.08 last 88.74) total: 10000 open/close in 109.69 seconds: 91.17 ops/second llite.lustre-ffff9099480e4000.statahead_batch_max=0 llite.lustre-ffff9099480e4000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9099480e4000.stats=clear mdc.lustre-MDT0001-mdc-ffff9099480e4000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m45.162s user 0m0.306s sys 0m18.085s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9099480e4000.statahead_batch_max=256 llite.lustre-ffff9099480e4000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9099480e4000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9099480e4000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9099480e4000.stats=clear mdc.lustre-MDT0001-mdc-ffff9099480e4000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m38.418s user 0m0.276s sys 0m16.563s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9099480e4000.batch_stats= snapshot_time: 1789550788.104637420 secs.nsecs start_time: 1789550742.636535001 secs.nsecs elapsed_time: 45.468102419 secs.nsecs subreqs per batch batches %% cum %% 1: 124 50 50 2: 26 10 60 4: 19 7 68 8: 13 5 73 16: 8 3 77 32: 13 5 82 64: 8 3 85 128: 0 0 85 256: 35 14 100 mdc.lustre-MDT0001-mdc-ffff9099480e4000.batch_stats= snapshot_time: 1789550788.104705184 secs.nsecs start_time: 1789550742.636654087 secs.nsecs elapsed_time: 45.468051097 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9099480e4000.statahead_batch_max=64 llite.lustre-ffff9099480e4000.statahead_max=128