== sanity test 123ad: Verify batching statahead works correctly ========================================================== 07:40:38 (1786189238) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff96e7748c5000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff96e7748c5000.batch_stats=0 llite.lustre-ffff96e7748c5000.statahead_max=32 llite.lustre-ffff96e7748c5000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff96e7748c5000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff96e7748c5000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 3.38 seconds: 29.61 ops/second llite.lustre-ffff96e7748c5000.statahead_max=0 101 real 0m2.602s user 0m0.005s sys 0m0.333s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff96e7748c5000.statahead_max=32 llite.lustre-ffff96e7748c5000.statahead_max=32 101 real 0m1.132s user 0m0.005s sys 0m0.182s 'ls -l' 100 files with statahead: 1 sec statahead total: 154 statahead wrong: 2 agl total: 147 list_total: 31 fname_total: 92 hit_total: 4405 miss_total: 1291 swrong=2 ewrong=2 - open/close 752 (time 1786189269.82 total 10.01 last 75.11) total: 900 open/close in 12.50 seconds: 71.98 ops/second llite.lustre-ffff96e7748c5000.statahead_max=0 1001 real 0m22.966s user 0m0.030s sys 0m3.157s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff96e7748c5000.statahead_max=32 llite.lustre-ffff96e7748c5000.statahead_max=32 1001 real 0m15.774s user 0m0.030s sys 0m1.787s 'ls -l' 1000 files with statahead: 16 sec statahead total: 155 statahead wrong: 2 agl total: 148 list_total: 32 fname_total: 92 hit_total: 5404 miss_total: 1292 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 35 seconds llite.lustre-ffff96e7748c5000.statahead_stats= statahead total: 155 statahead wrong: 2 agl total: 148 list_total: 32 fname_total: 92 hit_total: 5404 miss_total: 1292 mdc.lustre-MDT0000-mdc-ffff96e7748c5000.batch_stats= snapshot_time: 1786189359.674265383 secs.nsecs start_time: 1786189244.319823490 secs.nsecs elapsed_time: 115.354441893 secs.nsecs subreqs per batch batches %% cum %% 1: 3 8 8 2: 0 0 8 4: 2 5 13 8: 3 8 21 16: 24 64 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff96e7748c5000.batch_stats= snapshot_time: 1786189359.674373817 secs.nsecs start_time: 1786189244.319909878 secs.nsecs elapsed_time: 115.354463939 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 3 7 9 8: 3 7 17 16: 12 29 46 32: 22 53 100 - open/close 573 (time 1786189372.76 total 10.04 last 57.10) - open/close 1290 (time 1786189382.77 total 20.04 last 71.69) - open/close 2103 (time 1786189392.77 total 30.04 last 81.27) - open/close 2831 (time 1786189402.77 total 40.04 last 72.80) - open/close 3573 (time 1786189412.78 total 50.05 last 74.10) - open/close 4168 (time 1786189422.79 total 60.06 last 59.49) - open/close 4882 (time 1786189432.79 total 70.07 last 71.33) - open/close 5684 (time 1786189442.80 total 80.07 last 80.20) - open/close 6332 (time 1786189452.80 total 90.07 last 64.78) - open/close 7008 (time 1786189462.80 total 100.07 last 67.58) - open/close 7776 (time 1786189472.80 total 110.08 last 76.77) - open/close 8598 (time 1786189482.81 total 120.08 last 82.16) - open/close 9303 (time 1786189492.82 total 130.09 last 70.42) total: 10000 open/close in 138.92 seconds: 71.98 ops/second llite.lustre-ffff96e7748c5000.statahead_batch_max=0 llite.lustre-ffff96e7748c5000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff96e7748c5000.stats=clear mdc.lustre-MDT0001-mdc-ffff96e7748c5000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m35.284s user 0m0.384s sys 0m18.170s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff96e7748c5000.statahead_batch_max=32 llite.lustre-ffff96e7748c5000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff96e7748c5000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff96e7748c5000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff96e7748c5000.stats=clear mdc.lustre-MDT0001-mdc-ffff96e7748c5000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m18.296s user 0m0.309s sys 0m16.495s batched statahead RPC= hit count=9991 mdc.lustre-MDT0000-mdc-ffff96e7748c5000.batch_stats= snapshot_time: 1786189816.524302086 secs.nsecs start_time: 1786189664.735425086 secs.nsecs elapsed_time: 151.788877000 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff96e7748c5000.batch_stats= snapshot_time: 1786189816.524371548 secs.nsecs start_time: 1786189664.735562786 secs.nsecs elapsed_time: 151.788808762 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff96e7748c5000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff96e7748c5000.batch_stats=0 llite.lustre-ffff96e7748c5000.statahead_max=2048 llite.lustre-ffff96e7748c5000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff96e7748c5000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff96e7748c5000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.72 seconds: 36.72 ops/second llite.lustre-ffff96e7748c5000.statahead_max=0 101 real 0m2.396s user 0m0.004s sys 0m0.335s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff96e7748c5000.statahead_max=2048 llite.lustre-ffff96e7748c5000.statahead_max=2048 101 real 0m1.139s user 0m0.000s sys 0m0.163s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10090 miss_total: 10 swrong=0 ewrong=0 - open/close 709 (time 1786190185.57 total 10.00 last 70.89) total: 900 open/close in 13.45 seconds: 66.91 ops/second llite.lustre-ffff96e7748c5000.statahead_max=0 1001 real 0m23.351s user 0m0.035s sys 0m3.383s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff96e7748c5000.statahead_max=2048 llite.lustre-ffff96e7748c5000.statahead_max=2048 1001 real 0m10.721s user 0m0.032s sys 0m1.742s 'ls -l' 1000 files with statahead: 11 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11089 miss_total: 11 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 33 seconds llite.lustre-ffff96e7748c5000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11089 miss_total: 11 mdc.lustre-MDT0000-mdc-ffff96e7748c5000.batch_stats= snapshot_time: 1786190272.297618933 secs.nsecs start_time: 1786189818.560258254 secs.nsecs elapsed_time: 453.737360679 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 4 50 50 16: 1 12 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff96e7748c5000.batch_stats= snapshot_time: 1786190272.297699867 secs.nsecs start_time: 1786189818.560384304 secs.nsecs elapsed_time: 453.737315563 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 4 50 50 8: 1 12 62 16: 0 0 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 - open/close 858 (time 1786190285.68 total 10.01 last 85.75) - open/close 1516 (time 1786190295.68 total 20.01 last 65.74) - open/close 2213 (time 1786190305.69 total 30.02 last 69.68) - open/close 2893 (time 1786190315.69 total 40.03 last 67.95) - open/close 3723 (time 1786190325.70 total 50.03 last 82.98) - open/close 4458 (time 1786190335.71 total 60.04 last 73.42) - open/close 5204 (time 1786190345.72 total 70.05 last 74.53) - open/close 5946 (time 1786190355.72 total 80.05 last 74.18) - open/close 6713 (time 1786190365.74 total 90.07 last 76.58) - open/close 7458 (time 1786190375.89 total 100.22 last 73.40) - open/close 8187 (time 1786190385.90 total 110.23 last 72.82) - open/close 8880 (time 1786190395.90 total 120.23 last 69.27) - open/close 9638 (time 1786190405.92 total 130.25 last 75.66) total: 10000 open/close in 135.20 seconds: 73.97 ops/second llite.lustre-ffff96e7748c5000.statahead_batch_max=0 llite.lustre-ffff96e7748c5000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff96e7748c5000.stats=clear mdc.lustre-MDT0001-mdc-ffff96e7748c5000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m8.376s user 0m0.305s sys 0m16.779s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff96e7748c5000.statahead_batch_max=256 llite.lustre-ffff96e7748c5000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff96e7748c5000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff96e7748c5000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff96e7748c5000.stats=clear mdc.lustre-MDT0001-mdc-ffff96e7748c5000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m16.768s user 0m0.284s sys 0m15.884s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff96e7748c5000.batch_stats= snapshot_time: 1786190637.360742488 secs.nsecs start_time: 1786190548.659426659 secs.nsecs elapsed_time: 88.701315829 secs.nsecs subreqs per batch batches %% cum %% 1: 168 52 52 2: 36 11 63 4: 34 10 74 8: 12 3 77 16: 12 3 81 32: 17 5 86 64: 6 1 88 128: 2 0 89 256: 34 10 100 mdc.lustre-MDT0001-mdc-ffff96e7748c5000.batch_stats= snapshot_time: 1786190637.360812024 secs.nsecs start_time: 1786190548.659504961 secs.nsecs elapsed_time: 88.701307063 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff96e7748c5000.statahead_batch_max=64 llite.lustre-ffff96e7748c5000.statahead_max=128