== sanity test 123ad: Verify batching statahead works correctly ========================================================== 17:37:04 (1787607424) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9ad1d1f0f800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ad1d1f0f800.batch_stats=0 llite.lustre-ffff9ad1d1f0f800.statahead_max=32 llite.lustre-ffff9ad1d1f0f800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9ad1d1f0f800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ad1d1f0f800.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 4.68 seconds: 21.39 ops/second llite.lustre-ffff9ad1d1f0f800.statahead_max=0 101 real 0m4.628s user 0m0.003s sys 0m0.338s 'ls -l' 100 files without statahead: 5 sec llite.lustre-ffff9ad1d1f0f800.statahead_max=32 llite.lustre-ffff9ad1d1f0f800.statahead_max=32 101 real 0m1.723s user 0m0.006s sys 0m0.203s 'ls -l' 100 files with statahead: 1 sec statahead total: 145 statahead wrong: 2 agl total: 139 list_total: 32 fname_total: 112 hit_total: 3646 miss_total: 1727 swrong=2 ewrong=2 - open/close 650 (time 1787607462.22 total 10.01 last 64.92) total: 900 open/close in 13.64 seconds: 66.00 ops/second llite.lustre-ffff9ad1d1f0f800.statahead_max=0 1001 real 0m22.920s user 0m0.037s sys 0m3.165s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff9ad1d1f0f800.statahead_max=32 llite.lustre-ffff9ad1d1f0f800.statahead_max=32 1001 real 0m11.649s user 0m0.036s sys 0m1.452s 'ls -l' 1000 files with statahead: 12 sec statahead total: 146 statahead wrong: 2 agl total: 140 list_total: 33 fname_total: 112 hit_total: 4644 miss_total: 1729 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff9ad1d1f0f800.statahead_stats= statahead total: 146 statahead wrong: 2 agl total: 140 list_total: 33 fname_total: 112 hit_total: 4644 miss_total: 1729 mdc.lustre-MDT0000-mdc-ffff9ad1d1f0f800.batch_stats= snapshot_time: 1787607536.716763223 secs.nsecs start_time: 1787607431.286230753 secs.nsecs elapsed_time: 105.430532470 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 2 5 7 4: 2 5 12 8: 2 5 17 16: 19 47 65 32: 14 35 100 mdc.lustre-MDT0001-mdc-ffff9ad1d1f0f800.batch_stats= snapshot_time: 1787607536.717072989 secs.nsecs start_time: 1787607431.286728075 secs.nsecs elapsed_time: 105.430344914 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 7 7 8: 3 7 15 16: 19 48 64 32: 14 35 100 - open/close 578 (time 1787607549.57 total 10.02 last 57.71) - open/close 1111 (time 1787607559.57 total 20.02 last 53.29) - open/close 1849 (time 1787607569.57 total 30.02 last 73.79) - open/close 2595 (time 1787607579.57 total 40.02 last 74.58) - open/close 3312 (time 1787607589.58 total 50.03 last 71.65) - open/close 4061 (time 1787607599.59 total 60.03 last 74.87) - open/close 4817 (time 1787607609.59 total 70.03 last 75.60) - open/close 5509 (time 1787607619.60 total 80.05 last 69.09) - open/close 6255 (time 1787607629.61 total 90.06 last 74.53) - open/close 6931 (time 1787607639.61 total 100.06 last 67.58) - open/close 7542 (time 1787607649.62 total 110.07 last 61.05) - open/close 8200 (time 1787607659.63 total 120.08 last 65.75) - open/close 8838 (time 1787607669.64 total 130.09 last 63.75) - open/close 9530 (time 1787607679.65 total 140.09 last 69.15) total: 10000 open/close in 147.29 seconds: 67.89 ops/second llite.lustre-ffff9ad1d1f0f800.statahead_batch_max=0 llite.lustre-ffff9ad1d1f0f800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ad1d1f0f800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ad1d1f0f800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m6.610s user 0m0.365s sys 0m15.595s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9ad1d1f0f800.statahead_batch_max=32 llite.lustre-ffff9ad1d1f0f800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ad1d1f0f800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9ad1d1f0f800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9ad1d1f0f800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ad1d1f0f800.stats=clear ls -l files in d123ad.sanity: 10001 real 2m1.095s user 0m0.341s sys 0m14.351s batched statahead RPC= hit count=9978 mdc.lustre-MDT0000-mdc-ffff9ad1d1f0f800.batch_stats= snapshot_time: 1787607894.330983643 secs.nsecs start_time: 1787607761.698030123 secs.nsecs elapsed_time: 132.632953520 secs.nsecs subreqs per batch batches %% cum %% 1: 1 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9ad1d1f0f800.batch_stats= snapshot_time: 1787607894.331052086 secs.nsecs start_time: 1787607761.698174821 secs.nsecs elapsed_time: 132.632877265 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9ad1d1f0f800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ad1d1f0f800.batch_stats=0 llite.lustre-ffff9ad1d1f0f800.statahead_max=2048 llite.lustre-ffff9ad1d1f0f800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9ad1d1f0f800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ad1d1f0f800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.66 seconds: 37.61 ops/second llite.lustre-ffff9ad1d1f0f800.statahead_max=0 101 real 0m2.542s user 0m0.006s sys 0m0.310s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9ad1d1f0f800.statahead_max=2048 llite.lustre-ffff9ad1d1f0f800.statahead_max=2048 101 real 0m0.992s user 0m0.009s sys 0m0.171s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10077 miss_total: 23 swrong=0 ewrong=0 - open/close 892 (time 1787608234.18 total 10.00 last 89.17) total: 900 open/close in 10.13 seconds: 88.88 ops/second llite.lustre-ffff9ad1d1f0f800.statahead_max=0 1001 real 0m25.243s user 0m0.047s sys 0m3.100s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff9ad1d1f0f800.statahead_max=2048 llite.lustre-ffff9ad1d1f0f800.statahead_max=2048 1001 real 0m6.654s user 0m0.029s sys 0m1.455s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff9ad1d1f0f800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 mdc.lustre-MDT0000-mdc-ffff9ad1d1f0f800.batch_stats= snapshot_time: 1787608299.687633935 secs.nsecs start_time: 1787607897.308091530 secs.nsecs elapsed_time: 402.379542405 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 25 25 8: 3 37 62 16: 0 0 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff9ad1d1f0f800.batch_stats= snapshot_time: 1787608299.688927173 secs.nsecs start_time: 1787607897.308240896 secs.nsecs elapsed_time: 402.380686277 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 25 25 8: 2 25 50 16: 1 12 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 - open/close 813 (time 1787608312.40 total 10.01 last 81.23) - open/close 1600 (time 1787608322.41 total 20.01 last 78.67) - open/close 2336 (time 1787608332.41 total 30.02 last 73.55) - open/close 3098 (time 1787608342.42 total 40.02 last 76.18) - open/close 3830 (time 1787608352.42 total 50.02 last 73.19) - open/close 4540 (time 1787608362.42 total 60.03 last 70.97) - open/close 5361 (time 1787608372.43 total 70.04 last 82.04) - open/close 5995 (time 1787608382.45 total 80.05 last 63.30) - open/close 6658 (time 1787608392.47 total 90.08 last 66.12) - open/close 7249 (time 1787608402.47 total 100.08 last 59.10) - open/close 7956 (time 1787608412.48 total 110.08 last 70.68) - open/close 8654 (time 1787608422.48 total 120.09 last 69.76) - open/close 9388 (time 1787608432.49 total 130.09 last 73.36) - open/close 9963 (time 1787608442.49 total 140.09 last 57.50) total: 10000 open/close in 140.55 seconds: 71.15 ops/second llite.lustre-ffff9ad1d1f0f800.statahead_batch_max=0 llite.lustre-ffff9ad1d1f0f800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ad1d1f0f800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ad1d1f0f800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m8.917s user 0m0.328s sys 0m14.948s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9ad1d1f0f800.statahead_batch_max=256 llite.lustre-ffff9ad1d1f0f800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ad1d1f0f800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9ad1d1f0f800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9ad1d1f0f800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ad1d1f0f800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m54.496s user 0m0.310s sys 0m14.401s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9ad1d1f0f800.batch_stats= snapshot_time: 1787608588.100151306 secs.nsecs start_time: 1787608521.960528973 secs.nsecs elapsed_time: 66.139622333 secs.nsecs subreqs per batch batches %% cum %% 1: 92 39 39 2: 34 14 54 4: 28 12 66 8: 11 4 71 16: 8 3 74 32: 17 7 81 64: 6 2 84 128: 1 0 84 256: 35 15 100 mdc.lustre-MDT0001-mdc-ffff9ad1d1f0f800.batch_stats= snapshot_time: 1787608588.100243896 secs.nsecs start_time: 1787608521.960657114 secs.nsecs elapsed_time: 66.139586782 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9ad1d1f0f800.statahead_batch_max=64 llite.lustre-ffff9ad1d1f0f800.statahead_max=128