== sanity test 123ad: Verify batching statahead works correctly ========================================================== 08:02:14 (1787140934) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff967ea9564800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff967ea9564800.batch_stats=0 llite.lustre-ffff967ea9564800.statahead_max=32 llite.lustre-ffff967ea9564800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff967ea9564800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff967ea9564800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.05 seconds: 48.74 ops/second llite.lustre-ffff967ea9564800.statahead_max=0 101 real 0m2.294s user 0m0.005s sys 0m0.354s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff967ea9564800.statahead_max=32 llite.lustre-ffff967ea9564800.statahead_max=32 101 real 0m1.378s user 0m0.006s sys 0m0.191s 'ls -l' 100 files with statahead: 2 sec statahead total: 152 statahead wrong: 2 agl total: 146 list_total: 30 fname_total: 122 hit_total: 4011 miss_total: 1372 swrong=2 ewrong=2 total: 900 open/close in 9.77 seconds: 92.12 ops/second llite.lustre-ffff967ea9564800.statahead_max=0 1001 real 0m25.034s user 0m0.038s sys 0m3.215s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff967ea9564800.statahead_max=32 llite.lustre-ffff967ea9564800.statahead_max=32 1001 real 0m9.015s user 0m0.021s sys 0m1.484s 'ls -l' 1000 files with statahead: 9 sec statahead total: 153 statahead wrong: 2 agl total: 147 list_total: 31 fname_total: 122 hit_total: 5010 miss_total: 1373 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff967ea9564800.statahead_stats= statahead total: 153 statahead wrong: 2 agl total: 147 list_total: 31 fname_total: 122 hit_total: 5010 miss_total: 1373 mdc.lustre-MDT0000-mdc-ffff967ea9564800.batch_stats= snapshot_time: 1787141030.755041571 secs.nsecs start_time: 1787140939.305286834 secs.nsecs elapsed_time: 91.449754737 secs.nsecs subreqs per batch batches %% cum %% 1: 4 10 10 2: 1 2 13 4: 1 2 15 8: 2 5 21 16: 24 63 84 32: 6 15 100 mdc.lustre-MDT0001-mdc-ffff967ea9564800.batch_stats= snapshot_time: 1787141030.755280841 secs.nsecs start_time: 1787140939.305368675 secs.nsecs elapsed_time: 91.449912166 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 0 0 5 4: 1 2 7 8: 3 7 15 16: 8 20 35 32: 26 65 100 - open/close 819 (time 1787141043.33 total 10.01 last 81.80) - open/close 1779 (time 1787141053.36 total 20.05 last 95.67) - open/close 2735 (time 1787141063.38 total 30.06 last 95.47) - open/close 3595 (time 1787141073.38 total 40.07 last 85.93) - open/close 4361 (time 1787141083.40 total 50.08 last 76.49) - open/close 5168 (time 1787141093.40 total 60.09 last 80.66) - open/close 5996 (time 1787141103.40 total 70.09 last 82.80) - open/close 6937 (time 1787141113.41 total 80.10 last 94.03) - open/close 7920 (time 1787141123.42 total 90.10 last 98.21) - open/close 8928 (time 1787141133.43 total 100.11 last 100.73) - open/close 9932 (time 1787141143.43 total 110.12 last 100.33) total: 10000 open/close in 110.77 seconds: 90.28 ops/second llite.lustre-ffff967ea9564800.statahead_batch_max=0 llite.lustre-ffff967ea9564800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff967ea9564800.stats=clear mdc.lustre-MDT0001-mdc-ffff967ea9564800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m1.114s user 0m0.302s sys 0m15.924s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff967ea9564800.statahead_batch_max=32 llite.lustre-ffff967ea9564800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff967ea9564800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff967ea9564800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff967ea9564800.stats=clear mdc.lustre-MDT0001-mdc-ffff967ea9564800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m44.723s user 0m0.287s sys 0m15.223s batched statahead RPC= hit count=9985 mdc.lustre-MDT0000-mdc-ffff967ea9564800.batch_stats= snapshot_time: 1787141336.431484228 secs.nsecs start_time: 1787141216.979423121 secs.nsecs elapsed_time: 119.452061107 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff967ea9564800.batch_stats= snapshot_time: 1787141336.431544464 secs.nsecs start_time: 1787141216.979501722 secs.nsecs elapsed_time: 119.452042742 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff967ea9564800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff967ea9564800.batch_stats=0 llite.lustre-ffff967ea9564800.statahead_max=2048 llite.lustre-ffff967ea9564800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff967ea9564800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff967ea9564800.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 1.85 seconds: 54.18 ops/second llite.lustre-ffff967ea9564800.statahead_max=0 101 real 0m2.531s user 0m0.004s sys 0m0.359s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff967ea9564800.statahead_max=2048 llite.lustre-ffff967ea9564800.statahead_max=2048 101 real 0m0.815s user 0m0.006s sys 0m0.183s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10084 miss_total: 16 swrong=0 ewrong=0 - open/close 775 (time 1787141659.44 total 10.01 last 77.42) total: 900 open/close in 11.44 seconds: 78.67 ops/second llite.lustre-ffff967ea9564800.statahead_max=0 1001 real 0m21.480s user 0m0.033s sys 0m3.342s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff967ea9564800.statahead_max=2048 llite.lustre-ffff967ea9564800.statahead_max=2048 1001 real 0m7.759s user 0m0.031s sys 0m1.636s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff967ea9564800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 mdc.lustre-MDT0000-mdc-ffff967ea9564800.batch_stats= snapshot_time: 1787141726.280432779 secs.nsecs start_time: 1787141338.984028902 secs.nsecs elapsed_time: 387.296403877 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 10 10 8: 4 40 50 16: 2 20 70 32: 1 10 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 mdc.lustre-MDT0001-mdc-ffff967ea9564800.batch_stats= snapshot_time: 1787141726.280523485 secs.nsecs start_time: 1787141338.984184607 secs.nsecs elapsed_time: 387.296338878 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 10 10 4: 3 30 40 8: 1 10 50 16: 2 20 70 32: 1 10 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 - open/close 788 (time 1787141738.68 total 10.00 last 78.76) - open/close 1638 (time 1787141748.68 total 20.01 last 84.98) - open/close 2593 (time 1787141758.69 total 30.01 last 95.45) - open/close 3415 (time 1787141768.70 total 40.02 last 82.10) - open/close 4186 (time 1787141778.71 total 50.04 last 77.00) - open/close 4834 (time 1787141788.72 total 60.04 last 64.77) - open/close 5616 (time 1787141798.72 total 70.05 last 78.13) - open/close 6310 (time 1787141808.74 total 80.07 last 69.29) - open/close 7050 (time 1787141818.75 total 90.07 last 73.94) - open/close 7856 (time 1787141828.75 total 100.08 last 80.57) - open/close 8729 (time 1787141838.76 total 110.08 last 87.24) - open/close 9489 (time 1787141848.79 total 120.11 last 75.78) total: 10000 open/close in 126.85 seconds: 78.84 ops/second llite.lustre-ffff967ea9564800.statahead_batch_max=0 llite.lustre-ffff967ea9564800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff967ea9564800.stats=clear mdc.lustre-MDT0001-mdc-ffff967ea9564800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m18.079s user 0m0.322s sys 0m16.560s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff967ea9564800.statahead_batch_max=256 llite.lustre-ffff967ea9564800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff967ea9564800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff967ea9564800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff967ea9564800.stats=clear mdc.lustre-MDT0001-mdc-ffff967ea9564800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.118s user 0m0.288s sys 0m15.750s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff967ea9564800.batch_stats= snapshot_time: 1787142024.798008013 secs.nsecs start_time: 1787141945.788061241 secs.nsecs elapsed_time: 79.009946772 secs.nsecs subreqs per batch batches %% cum %% 1: 188 50 50 2: 44 11 62 4: 38 10 72 8: 34 9 81 16: 13 3 84 32: 18 4 89 64: 4 1 90 128: 0 0 90 256: 34 9 100 mdc.lustre-MDT0001-mdc-ffff967ea9564800.batch_stats= snapshot_time: 1787142024.798083399 secs.nsecs start_time: 1787141945.790130968 secs.nsecs elapsed_time: 79.007952431 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff967ea9564800.statahead_batch_max=64 llite.lustre-ffff967ea9564800.statahead_max=128