== sanity test 123ad: Verify batching statahead works correctly ========================================================== 09:43:43 (1787319823) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9611bdc30000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9611bdc30000.batch_stats=0 llite.lustre-ffff9611bdc30000.statahead_max=32 llite.lustre-ffff9611bdc30000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9611bdc30000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9611bdc30000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.47 seconds: 40.45 ops/second llite.lustre-ffff9611bdc30000.statahead_max=0 101 real 0m2.658s user 0m0.004s sys 0m0.336s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9611bdc30000.statahead_max=32 llite.lustre-ffff9611bdc30000.statahead_max=32 101 real 0m1.161s user 0m0.005s sys 0m0.187s 'ls -l' 100 files with statahead: 1 sec statahead total: 131 statahead wrong: 2 agl total: 124 list_total: 31 fname_total: 88 hit_total: 3965 miss_total: 1583 swrong=2 ewrong=2 - open/close 891 (time 1787319852.45 total 10.01 last 89.02) total: 900 open/close in 10.18 seconds: 88.45 ops/second llite.lustre-ffff9611bdc30000.statahead_max=0 1001 real 0m21.224s user 0m0.027s sys 0m3.140s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff9611bdc30000.statahead_max=32 llite.lustre-ffff9611bdc30000.statahead_max=32 1001 real 0m10.187s user 0m0.024s sys 0m1.542s 'ls -l' 1000 files with statahead: 10 sec statahead total: 132 statahead wrong: 2 agl total: 125 list_total: 32 fname_total: 88 hit_total: 4964 miss_total: 1584 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff9611bdc30000.statahead_stats= statahead total: 132 statahead wrong: 2 agl total: 125 list_total: 32 fname_total: 88 hit_total: 4964 miss_total: 1584 mdc.lustre-MDT0000-mdc-ffff9611bdc30000.batch_stats= snapshot_time: 1787319915.193449861 secs.nsecs start_time: 1787319828.572626805 secs.nsecs elapsed_time: 86.620823056 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 0 0 5 4: 3 8 13 8: 1 2 16 16: 28 75 91 32: 3 8 100 mdc.lustre-MDT0001-mdc-ffff9611bdc30000.batch_stats= snapshot_time: 1787319915.193520229 secs.nsecs start_time: 1787319828.572764770 secs.nsecs elapsed_time: 86.620755459 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 5 5 4: 2 5 10 8: 2 5 15 16: 7 17 33 32: 26 66 100 - open/close 852 (time 1787319928.18 total 10.00 last 85.20) - open/close 1784 (time 1787319938.19 total 20.01 last 93.11) - open/close 2583 (time 1787319948.20 total 30.02 last 79.83) - open/close 3492 (time 1787319958.20 total 40.02 last 90.90) - open/close 4404 (time 1787319968.21 total 50.03 last 91.12) - open/close 5306 (time 1787319978.21 total 60.03 last 90.19) - open/close 6101 (time 1787319988.23 total 70.05 last 79.32) - open/close 6694 (time 1787319998.24 total 80.06 last 59.25) - open/close 7481 (time 1787320008.25 total 90.07 last 78.66) - open/close 8252 (time 1787320018.26 total 100.08 last 77.00) - open/close 9069 (time 1787320028.26 total 110.08 last 81.68) - open/close 9833 (time 1787320038.27 total 120.09 last 76.33) total: 10000 open/close in 122.46 seconds: 81.66 ops/second llite.lustre-ffff9611bdc30000.statahead_batch_max=0 llite.lustre-ffff9611bdc30000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9611bdc30000.stats=clear mdc.lustre-MDT0001-mdc-ffff9611bdc30000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.832s user 0m0.305s sys 0m15.081s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9611bdc30000.statahead_batch_max=32 llite.lustre-ffff9611bdc30000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9611bdc30000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9611bdc30000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9611bdc30000.stats=clear mdc.lustre-MDT0001-mdc-ffff9611bdc30000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m57.811s user 0m0.346s sys 0m15.687s batched statahead RPC= hit count=9985 mdc.lustre-MDT0000-mdc-ffff9611bdc30000.batch_stats= snapshot_time: 1787320244.296696839 secs.nsecs start_time: 1787320112.522399653 secs.nsecs elapsed_time: 131.774297186 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9611bdc30000.batch_stats= snapshot_time: 1787320244.296768123 secs.nsecs start_time: 1787320112.522488659 secs.nsecs elapsed_time: 131.774279464 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9611bdc30000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9611bdc30000.batch_stats=0 llite.lustre-ffff9611bdc30000.statahead_max=2048 llite.lustre-ffff9611bdc30000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9611bdc30000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9611bdc30000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.80 seconds: 35.69 ops/second llite.lustre-ffff9611bdc30000.statahead_max=0 101 real 0m2.110s user 0m0.005s sys 0m0.339s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9611bdc30000.statahead_max=2048 llite.lustre-ffff9611bdc30000.statahead_max=2048 101 real 0m0.990s user 0m0.004s sys 0m0.200s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10084 miss_total: 16 swrong=0 ewrong=0 - open/close 770 (time 1787320532.02 total 10.01 last 76.89) total: 900 open/close in 11.93 seconds: 75.45 ops/second llite.lustre-ffff9611bdc30000.statahead_max=0 1001 real 0m21.619s user 0m0.032s sys 0m3.186s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff9611bdc30000.statahead_max=2048 llite.lustre-ffff9611bdc30000.statahead_max=2048 1001 real 0m7.209s user 0m0.033s sys 0m1.463s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff9611bdc30000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 mdc.lustre-MDT0000-mdc-ffff9611bdc30000.batch_stats= snapshot_time: 1787320597.731020676 secs.nsecs start_time: 1787320247.441673914 secs.nsecs elapsed_time: 350.289346762 secs.nsecs subreqs per batch batches %% cum %% 1: 3 37 37 2: 1 12 50 4: 1 12 62 8: 1 12 75 16: 0 0 75 32: 0 0 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff9611bdc30000.batch_stats= snapshot_time: 1787320597.731096836 secs.nsecs start_time: 1787320247.441873673 secs.nsecs elapsed_time: 350.289223163 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 2 22 44 16: 1 11 55 32: 0 0 55 64: 2 22 77 128: 0 0 77 256: 2 22 100 - open/close 710 (time 1787320610.81 total 10.00 last 70.97) - open/close 1443 (time 1787320620.82 total 20.01 last 73.28) - open/close 2203 (time 1787320630.82 total 30.01 last 75.99) - open/close 3061 (time 1787320640.82 total 40.01 last 85.77) - open/close 3905 (time 1787320650.83 total 50.02 last 84.35) - open/close 4774 (time 1787320660.83 total 60.02 last 86.87) - open/close 5758 (time 1787320670.83 total 70.02 last 98.38) - open/close 6670 (time 1787320680.84 total 80.03 last 91.12) - open/close 7571 (time 1787320690.85 total 90.04 last 90.02) - open/close 8413 (time 1787320700.86 total 100.05 last 84.16) - open/close 9350 (time 1787320710.86 total 110.05 last 93.70) total: 10000 open/close in 117.40 seconds: 85.18 ops/second llite.lustre-ffff9611bdc30000.statahead_batch_max=0 llite.lustre-ffff9611bdc30000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9611bdc30000.stats=clear mdc.lustre-MDT0001-mdc-ffff9611bdc30000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.786s user 0m0.336s sys 0m15.991s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9611bdc30000.statahead_batch_max=256 llite.lustre-ffff9611bdc30000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9611bdc30000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9611bdc30000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9611bdc30000.stats=clear mdc.lustre-MDT0001-mdc-ffff9611bdc30000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m55.093s user 0m0.288s sys 0m15.911s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9611bdc30000.batch_stats= snapshot_time: 1787320867.854963740 secs.nsecs start_time: 1787320796.119132823 secs.nsecs elapsed_time: 71.735830917 secs.nsecs subreqs per batch batches %% cum %% 1: 127 40 40 2: 60 19 59 4: 23 7 67 8: 36 11 78 16: 13 4 83 32: 15 4 87 64: 3 0 88 128: 0 0 88 256: 35 11 100 mdc.lustre-MDT0001-mdc-ffff9611bdc30000.batch_stats= snapshot_time: 1787320867.855032480 secs.nsecs start_time: 1787320796.119272960 secs.nsecs elapsed_time: 71.735759520 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9611bdc30000.statahead_batch_max=64 llite.lustre-ffff9611bdc30000.statahead_max=128