== sanity test 123ad: Verify batching statahead works correctly ========================================================== 16:45:35 (1787777135) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff946a02f0e000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff946a02f0e000.batch_stats=0 llite.lustre-ffff946a02f0e000.statahead_max=32 llite.lustre-ffff946a02f0e000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff946a02f0e000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff946a02f0e000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.32 seconds: 43.11 ops/second llite.lustre-ffff946a02f0e000.statahead_max=0 101 real 0m1.922s user 0m0.006s sys 0m0.314s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff946a02f0e000.statahead_max=32 llite.lustre-ffff946a02f0e000.statahead_max=32 101 real 0m1.062s user 0m0.007s sys 0m0.166s 'ls -l' 100 files with statahead: 1 sec statahead total: 103 statahead wrong: 2 agl total: 97 list_total: 30 fname_total: 69 hit_total: 3871 miss_total: 1500 swrong=2 ewrong=2 - open/close 826 (time 1787777163.05 total 10.02 last 82.43) total: 900 open/close in 11.18 seconds: 80.51 ops/second llite.lustre-ffff946a02f0e000.statahead_max=0 1001 real 0m18.881s user 0m0.033s sys 0m2.847s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff946a02f0e000.statahead_max=32 llite.lustre-ffff946a02f0e000.statahead_max=32 1001 real 0m9.469s user 0m0.032s sys 0m1.317s 'ls -l' 1000 files with statahead: 9 sec statahead total: 104 statahead wrong: 2 agl total: 98 list_total: 31 fname_total: 69 hit_total: 4869 miss_total: 1502 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff946a02f0e000.statahead_stats= statahead total: 104 statahead wrong: 2 agl total: 98 list_total: 31 fname_total: 69 hit_total: 4869 miss_total: 1502 mdc.lustre-MDT0000-mdc-ffff946a02f0e000.batch_stats= snapshot_time: 1787777224.520444406 secs.nsecs start_time: 1787777140.545337900 secs.nsecs elapsed_time: 83.975106506 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 1 2 8 4: 2 5 13 8: 0 0 13 16: 28 77 91 32: 3 8 100 mdc.lustre-MDT0001-mdc-ffff946a02f0e000.batch_stats= snapshot_time: 1787777224.520498276 secs.nsecs start_time: 1787777140.545480401 secs.nsecs elapsed_time: 83.975017875 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 1 2 7 4: 0 0 7 8: 2 5 12 16: 10 25 37 32: 25 62 100 - open/close 843 (time 1787777237.71 total 10.00 last 84.30) - open/close 1683 (time 1787777247.71 total 20.00 last 83.99) - open/close 2531 (time 1787777257.73 total 30.02 last 84.66) - open/close 3397 (time 1787777267.73 total 40.02 last 86.55) - open/close 4266 (time 1787777277.75 total 50.04 last 86.75) - open/close 4948 (time 1787777287.77 total 60.06 last 68.05) - open/close 5809 (time 1787777297.78 total 70.07 last 86.03) - open/close 6576 (time 1787777307.78 total 80.08 last 76.66) - open/close 7505 (time 1787777317.79 total 90.09 last 92.80) - open/close 8489 (time 1787777327.81 total 100.10 last 98.28) - open/close 9292 (time 1787777337.81 total 110.10 last 80.29) total: 10000 open/close in 117.67 seconds: 84.99 ops/second llite.lustre-ffff946a02f0e000.statahead_batch_max=0 llite.lustre-ffff946a02f0e000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff946a02f0e000.stats=clear mdc.lustre-MDT0001-mdc-ffff946a02f0e000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m6.761s user 0m0.283s sys 0m13.802s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff946a02f0e000.statahead_batch_max=32 llite.lustre-ffff946a02f0e000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff946a02f0e000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff946a02f0e000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff946a02f0e000.stats=clear mdc.lustre-MDT0001-mdc-ffff946a02f0e000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m41.290s user 0m0.289s sys 0m13.244s batched statahead RPC= hit count=9981 mdc.lustre-MDT0000-mdc-ffff946a02f0e000.batch_stats= snapshot_time: 1787777529.232654854 secs.nsecs start_time: 1787777419.370809760 secs.nsecs elapsed_time: 109.861845094 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff946a02f0e000.batch_stats= snapshot_time: 1787777529.232805492 secs.nsecs start_time: 1787777419.371018088 secs.nsecs elapsed_time: 109.861787404 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff946a02f0e000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff946a02f0e000.batch_stats=0 llite.lustre-ffff946a02f0e000.statahead_max=2048 llite.lustre-ffff946a02f0e000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff946a02f0e000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff946a02f0e000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.11 seconds: 47.45 ops/second llite.lustre-ffff946a02f0e000.statahead_max=0 101 real 0m2.467s user 0m0.006s sys 0m0.327s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff946a02f0e000.statahead_max=2048 llite.lustre-ffff946a02f0e000.statahead_max=2048 101 real 0m1.014s user 0m0.008s sys 0m0.171s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10080 miss_total: 20 swrong=0 ewrong=0 - open/close 508 (time 1787777826.20 total 10.00 last 50.78) total: 900 open/close in 16.38 seconds: 54.96 ops/second llite.lustre-ffff946a02f0e000.statahead_max=0 1001 real 0m24.857s user 0m0.034s sys 0m3.078s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff946a02f0e000.statahead_max=2048 llite.lustre-ffff946a02f0e000.statahead_max=2048 1001 real 0m8.267s user 0m0.036s sys 0m1.365s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff946a02f0e000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 mdc.lustre-MDT0000-mdc-ffff946a02f0e000.batch_stats= snapshot_time: 1787777901.340084720 secs.nsecs start_time: 1787777532.061278966 secs.nsecs elapsed_time: 369.278805754 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 28 28 8: 2 28 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff946a02f0e000.batch_stats= snapshot_time: 1787777901.340161293 secs.nsecs start_time: 1787777532.063562548 secs.nsecs elapsed_time: 369.276598745 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 42 42 8: 1 14 57 16: 0 0 57 32: 0 0 57 64: 1 14 71 128: 0 0 71 256: 2 28 100 - open/close 794 (time 1787777914.00 total 10.00 last 79.40) - open/close 1574 (time 1787777924.00 total 20.00 last 78.00) - open/close 2281 (time 1787777934.00 total 30.01 last 70.65) - open/close 3021 (time 1787777944.02 total 40.02 last 73.89) - open/close 3771 (time 1787777954.03 total 50.03 last 74.93) - open/close 4439 (time 1787777964.04 total 60.04 last 66.74) - open/close 5252 (time 1787777974.04 total 70.05 last 81.27) - open/close 5964 (time 1787777984.05 total 80.05 last 71.16) - open/close 6855 (time 1787777994.05 total 90.05 last 89.08) - open/close 7710 (time 1787778004.06 total 100.07 last 85.37) - open/close 8506 (time 1787778014.08 total 110.08 last 79.51) - open/close 9241 (time 1787778024.08 total 120.09 last 73.44) - open/close 9838 (time 1787778034.10 total 130.11 last 59.59) total: 10000 open/close in 133.20 seconds: 75.08 ops/second llite.lustre-ffff946a02f0e000.statahead_batch_max=0 llite.lustre-ffff946a02f0e000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff946a02f0e000.stats=clear mdc.lustre-MDT0001-mdc-ffff946a02f0e000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m5.792s user 0m0.295s sys 0m13.572s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff946a02f0e000.statahead_batch_max=256 llite.lustre-ffff946a02f0e000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff946a02f0e000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff946a02f0e000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff946a02f0e000.stats=clear mdc.lustre-MDT0001-mdc-ffff946a02f0e000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.179s user 0m0.296s sys 0m14.051s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff946a02f0e000.batch_stats= snapshot_time: 1787778180.944230706 secs.nsecs start_time: 1787778113.518564382 secs.nsecs elapsed_time: 67.425666324 secs.nsecs subreqs per batch batches %% cum %% 1: 137 47 47 2: 43 14 62 4: 20 6 68 8: 20 6 75 16: 9 3 78 32: 20 6 85 64: 6 2 87 128: 1 0 88 256: 34 11 100 mdc.lustre-MDT0001-mdc-ffff946a02f0e000.batch_stats= snapshot_time: 1787778180.944300499 secs.nsecs start_time: 1787778113.518712991 secs.nsecs elapsed_time: 67.425587508 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff946a02f0e000.statahead_batch_max=64 llite.lustre-ffff946a02f0e000.statahead_max=128