== sanity test 123ad: Verify batching statahead works correctly ========================================================== 22:20:34 (1781230834) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffffa0fad02ae000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa0fad02ae000.batch_stats=0 llite.lustre-ffffa0fad02ae000.statahead_max=32 llite.lustre-ffffa0fad02ae000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffffa0fad02ae000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa0fad02ae000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.31 seconds: 43.34 ops/second llite.lustre-ffffa0fad02ae000.statahead_max=0 101 real 0m2.631s user 0m0.011s sys 0m0.407s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffffa0fad02ae000.statahead_max=32 llite.lustre-ffffa0fad02ae000.statahead_max=32 101 real 0m1.400s user 0m0.006s sys 0m0.204s 'ls -l' 100 files with statahead: 1 sec statahead total: 165 statahead wrong: 2 agl total: 159 list_total: 32 fname_total: 115 hit_total: 3711 miss_total: 1621 swrong=2 ewrong=2 - open/close 749 (time 1781230862.65 total 10.00 last 74.87) total: 900 open/close in 11.75 seconds: 76.57 ops/second llite.lustre-ffffa0fad02ae000.statahead_max=0 1001 real 0m28.954s user 0m0.042s sys 0m3.662s 'ls -l' 1000 files without statahead: 29 sec llite.lustre-ffffa0fad02ae000.statahead_max=32 llite.lustre-ffffa0fad02ae000.statahead_max=32 1001 real 0m13.112s user 0m0.036s sys 0m1.723s 'ls -l' 1000 files with statahead: 13 sec statahead total: 166 statahead wrong: 2 agl total: 160 list_total: 33 fname_total: 115 hit_total: 4707 miss_total: 1625 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 24 seconds llite.lustre-ffffa0fad02ae000.statahead_stats= statahead total: 166 statahead wrong: 2 agl total: 160 list_total: 33 fname_total: 115 hit_total: 4707 miss_total: 1625 mdc.lustre-MDT0000-mdc-ffffa0fad02ae000.batch_stats= snapshot_time: 1781230947.605211819 secs.nsecs start_time: 1781230838.689862854 secs.nsecs elapsed_time: 108.915348965 secs.nsecs subreqs per batch batches % cum % 1: 3 8 8 2: 0 0 8 4: 3 8 16 8: 0 0 16 16: 27 72 89 32: 4 10 100 mdc.lustre-MDT0001-mdc-ffffa0fad02ae000.batch_stats= snapshot_time: 1781230947.605294874 secs.nsecs start_time: 1781230838.689978586 secs.nsecs elapsed_time: 108.915316288 secs.nsecs subreqs per batch batches % cum % 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 2 5 12 16: 13 32 45 32: 22 55 100 - open/close 846 (time 1781230960.26 total 10.00 last 84.58) - open/close 1657 (time 1781230970.26 total 20.00 last 81.09) - open/close 2532 (time 1781230980.27 total 30.01 last 87.47) - open/close 3242 (time 1781230990.27 total 40.01 last 70.99) - open/close 4059 (time 1781231000.27 total 50.01 last 81.69) - open/close 4807 (time 1781231010.27 total 60.01 last 74.78) - open/close 5571 (time 1781231020.28 total 70.02 last 76.35) - open/close 6443 (time 1781231030.28 total 80.02 last 87.17) - open/close 7276 (time 1781231040.29 total 90.03 last 83.27) - open/close 8026 (time 1781231050.30 total 100.04 last 74.91) - open/close 8872 (time 1781231060.32 total 110.06 last 84.40) - open/close 9697 (time 1781231070.32 total 120.06 last 82.47) total: 10000 open/close in 123.64 seconds: 80.88 ops/second llite.lustre-ffffa0fad02ae000.statahead_batch_max=0 llite.lustre-ffffa0fad02ae000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa0fad02ae000.stats=clear mdc.lustre-MDT0001-mdc-ffffa0fad02ae000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m12.956s user 0m0.414s sys 0m18.017s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffffa0fad02ae000.statahead_batch_max=32 llite.lustre-ffffa0fad02ae000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa0fad02ae000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffffa0fad02ae000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffffa0fad02ae000.stats=clear mdc.lustre-MDT0001-mdc-ffffa0fad02ae000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m9.060s user 0m0.410s sys 0m17.154s batched statahead RPC= hit count=9982 mdc.lustre-MDT0000-mdc-ffffa0fad02ae000.batch_stats= snapshot_time: 1781231301.092378607 secs.nsecs start_time: 1781231157.635723347 secs.nsecs elapsed_time: 143.456655260 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffffa0fad02ae000.batch_stats= snapshot_time: 1781231301.092508731 secs.nsecs start_time: 1781231157.644560967 secs.nsecs elapsed_time: 143.447947764 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffffa0fad02ae000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa0fad02ae000.batch_stats=0 llite.lustre-ffffa0fad02ae000.statahead_max=2048 llite.lustre-ffffa0fad02ae000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffffa0fad02ae000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffffa0fad02ae000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.65 seconds: 37.68 ops/second llite.lustre-ffffa0fad02ae000.statahead_max=0 101 real 0m2.508s user 0m0.008s sys 0m0.411s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffffa0fad02ae000.statahead_max=2048 llite.lustre-ffffa0fad02ae000.statahead_max=2048 101 real 0m1.195s user 0m0.006s sys 0m0.202s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10081 miss_total: 19 swrong=0 ewrong=0 - open/close 757 (time 1781231625.04 total 10.00 last 75.67) total: 900 open/close in 12.42 seconds: 72.49 ops/second llite.lustre-ffffa0fad02ae000.statahead_max=0 1001 real 0m24.264s user 0m0.039s sys 0m3.623s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffffa0fad02ae000.statahead_max=2048 llite.lustre-ffffa0fad02ae000.statahead_max=2048 1001 real 0m8.420s user 0m0.042s sys 0m1.847s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffffa0fad02ae000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 mdc.lustre-MDT0000-mdc-ffffa0fad02ae000.batch_stats= snapshot_time: 1781231695.466168510 secs.nsecs start_time: 1781231304.164352314 secs.nsecs elapsed_time: 391.301816196 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 2 28 28 8: 2 28 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffffa0fad02ae000.batch_stats= snapshot_time: 1781231695.466251164 secs.nsecs start_time: 1781231304.164543481 secs.nsecs elapsed_time: 391.301707683 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 2: 0 0 0 4: 3 42 42 8: 1 14 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 - open/close 717 (time 1781231708.18 total 10.01 last 71.60) - open/close 1529 (time 1781231718.20 total 20.03 last 81.08) - open/close 2229 (time 1781231728.20 total 30.03 last 69.99) - open/close 2935 (time 1781231738.20 total 40.03 last 70.57) - open/close 3574 (time 1781231748.21 total 50.04 last 63.88) - open/close 4261 (time 1781231758.21 total 60.04 last 68.68) - open/close 4899 (time 1781231768.21 total 70.04 last 63.77) - open/close 5537 (time 1781231778.22 total 80.05 last 63.79) - open/close 6104 (time 1781231788.23 total 90.06 last 56.63) - open/close 6780 (time 1781231798.24 total 100.07 last 67.53) - open/close 7503 (time 1781231808.25 total 110.08 last 72.24) - open/close 8217 (time 1781231818.25 total 120.08 last 71.39) - open/close 8995 (time 1781231828.26 total 130.09 last 77.74) - open/close 9600 (time 1781231838.28 total 140.11 last 60.36) total: 10000 open/close in 146.55 seconds: 68.24 ops/second llite.lustre-ffffa0fad02ae000.statahead_batch_max=0 llite.lustre-ffffa0fad02ae000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa0fad02ae000.stats=clear mdc.lustre-MDT0001-mdc-ffffa0fad02ae000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m12.758s user 0m0.383s sys 0m17.330s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffffa0fad02ae000.statahead_batch_max=256 llite.lustre-ffffa0fad02ae000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffffa0fad02ae000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffffa0fad02ae000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffffa0fad02ae000.stats=clear mdc.lustre-MDT0001-mdc-ffffa0fad02ae000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m13.014s user 0m0.315s sys 0m17.668s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffffa0fad02ae000.batch_stats= snapshot_time: 1781232021.855277725 secs.nsecs start_time: 1781231932.382155683 secs.nsecs elapsed_time: 89.473122042 secs.nsecs subreqs per batch batches % cum % 1: 199 51 51 2: 51 13 64 4: 45 11 76 8: 24 6 82 16: 10 2 85 32: 20 5 90 64: 3 0 91 128: 0 0 91 256: 34 8 100 mdc.lustre-MDT0001-mdc-ffffa0fad02ae000.batch_stats= snapshot_time: 1781232021.855339915 secs.nsecs start_time: 1781231932.382361576 secs.nsecs elapsed_time: 89.472978339 secs.nsecs subreqs per batch batches % cum % 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffffa0fad02ae000.statahead_batch_max=64 llite.lustre-ffffa0fad02ae000.statahead_max=128