== sanity test 123ad: Verify batching statahead works correctly ========================================================== 10:27:09 (1782224829) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9f4c9874b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f4c9874b800.batch_stats=0 llite.lustre-ffff9f4c9874b800.statahead_max=32 llite.lustre-ffff9f4c9874b800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9f4c9874b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f4c9874b800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.93 seconds: 34.09 ops/second llite.lustre-ffff9f4c9874b800.statahead_max=0 101 real 0m3.160s user 0m0.006s sys 0m0.336s 'ls -l' 100 files without statahead: 4 sec llite.lustre-ffff9f4c9874b800.statahead_max=32 llite.lustre-ffff9f4c9874b800.statahead_max=32 101 real 0m1.422s user 0m0.005s sys 0m0.197s 'ls -l' 100 files with statahead: 2 sec statahead total: 148 statahead wrong: 2 agl total: 142 list_total: 32 fname_total: 101 hit_total: 3958 miss_total: 1671 swrong=2 ewrong=2 - open/close 701 (time 1782224861.94 total 10.00 last 70.08) total: 900 open/close in 12.97 seconds: 69.36 ops/second llite.lustre-ffff9f4c9874b800.statahead_max=0 1001 real 0m27.400s user 0m0.036s sys 0m3.398s 'ls -l' 1000 files without statahead: 28 sec llite.lustre-ffff9f4c9874b800.statahead_max=32 llite.lustre-ffff9f4c9874b800.statahead_max=32 1001 real 0m10.864s user 0m0.034s sys 0m1.754s 'ls -l' 1000 files with statahead: 11 sec statahead total: 149 statahead wrong: 2 agl total: 143 list_total: 33 fname_total: 101 hit_total: 4956 miss_total: 1673 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff9f4c9874b800.statahead_stats= statahead total: 149 statahead wrong: 2 agl total: 143 list_total: 33 fname_total: 101 hit_total: 4956 miss_total: 1673 mdc.lustre-MDT0000-mdc-ffff9f4c9874b800.batch_stats= snapshot_time: 1782224940.278115041 secs.nsecs start_time: 1782224836.147983255 secs.nsecs elapsed_time: 104.130131786 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 4 11 13 8: 1 2 16 16: 26 72 88 32: 4 11 100 mdc.lustre-MDT0001-mdc-ffff9f4c9874b800.batch_stats= snapshot_time: 1782224940.278303542 secs.nsecs start_time: 1782224836.148081769 secs.nsecs elapsed_time: 104.130221773 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 1 2 5 8: 2 5 10 16: 11 28 38 32: 24 61 100 - open/close 729 (time 1782224953.37 total 10.02 last 72.78) - open/close 1436 (time 1782224963.37 total 20.02 last 70.68) - open/close 2074 (time 1782224973.39 total 30.04 last 63.69) - open/close 2764 (time 1782224983.39 total 40.04 last 69.00) - open/close 3468 (time 1782224993.40 total 50.05 last 70.31) - open/close 4210 (time 1782225003.40 total 60.05 last 74.18) - open/close 4852 (time 1782225013.41 total 70.06 last 64.13) - open/close 5546 (time 1782225023.42 total 80.07 last 69.39) - open/close 6255 (time 1782225033.42 total 90.07 last 70.89) - open/close 6952 (time 1782225043.43 total 100.08 last 69.60) - open/close 7666 (time 1782225053.44 total 110.09 last 71.33) - open/close 8424 (time 1782225063.48 total 120.13 last 75.52) - open/close 9112 (time 1782225073.49 total 130.14 last 68.74) - open/close 9851 (time 1782225083.49 total 140.14 last 73.89) total: 10000 open/close in 142.19 seconds: 70.33 ops/second llite.lustre-ffff9f4c9874b800.statahead_batch_max=0 llite.lustre-ffff9f4c9874b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f4c9874b800.stats=clear mdc.lustre-MDT0001-mdc-ffff9f4c9874b800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m12.184s user 0m0.374s sys 0m16.865s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9f4c9874b800.statahead_batch_max=32 llite.lustre-ffff9f4c9874b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f4c9874b800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9f4c9874b800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9f4c9874b800.stats=clear mdc.lustre-MDT0001-mdc-ffff9f4c9874b800.stats=clear ls -l files in d123ad.sanity: 10001 real 2m12.807s user 0m0.394s sys 0m17.152s batched statahead RPC= hit count=9983 mdc.lustre-MDT0000-mdc-ffff9f4c9874b800.batch_stats= snapshot_time: 1782225317.341458261 secs.nsecs start_time: 1782225169.322875018 secs.nsecs elapsed_time: 148.018583243 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9f4c9874b800.batch_stats= snapshot_time: 1782225317.341559295 secs.nsecs start_time: 1782225169.323005718 secs.nsecs elapsed_time: 148.018553577 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9f4c9874b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f4c9874b800.batch_stats=0 llite.lustre-ffff9f4c9874b800.statahead_max=2048 llite.lustre-ffff9f4c9874b800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9f4c9874b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f4c9874b800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.98 seconds: 33.51 ops/second llite.lustre-ffff9f4c9874b800.statahead_max=0 101 real 0m2.649s user 0m0.005s sys 0m0.382s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9f4c9874b800.statahead_max=2048 llite.lustre-ffff9f4c9874b800.statahead_max=2048 101 real 0m0.767s user 0m0.007s sys 0m0.190s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10082 miss_total: 18 swrong=0 ewrong=0 - open/close 713 (time 1782225656.94 total 10.01 last 71.25) total: 900 open/close in 13.23 seconds: 68.00 ops/second llite.lustre-ffff9f4c9874b800.statahead_max=0 1001 real 0m26.233s user 0m0.041s sys 0m3.088s 'ls -l' 1000 files without statahead: 27 sec llite.lustre-ffff9f4c9874b800.statahead_max=2048 llite.lustre-ffff9f4c9874b800.statahead_max=2048 1001 real 0m7.355s user 0m0.028s sys 0m1.574s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff9f4c9874b800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 mdc.lustre-MDT0000-mdc-ffff9f4c9874b800.batch_stats= snapshot_time: 1782225728.289901250 secs.nsecs start_time: 1782225320.162309008 secs.nsecs elapsed_time: 408.127592242 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 12 12 8: 3 37 50 16: 1 12 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff9f4c9874b800.batch_stats= snapshot_time: 1782225728.289989771 secs.nsecs start_time: 1782225320.162541231 secs.nsecs elapsed_time: 408.127448540 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 37 37 8: 2 25 62 16: 0 0 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 - open/close 659 (time 1782225740.65 total 10.01 last 65.86) - open/close 1340 (time 1782225750.68 total 20.03 last 67.93) - open/close 2089 (time 1782225760.68 total 30.03 last 74.90) - open/close 2856 (time 1782225770.69 total 40.04 last 76.65) - open/close 3567 (time 1782225780.69 total 50.04 last 71.07) - open/close 4326 (time 1782225790.70 total 60.05 last 75.85) - open/close 5060 (time 1782225800.71 total 70.06 last 73.34) - open/close 5842 (time 1782225810.71 total 80.07 last 78.14) - open/close 6569 (time 1782225820.72 total 90.07 last 72.68) - open/close 7265 (time 1782225830.72 total 100.07 last 69.58) - open/close 7963 (time 1782225840.74 total 110.09 last 69.67) - open/close 8660 (time 1782225850.75 total 120.10 last 69.63) - open/close 9328 (time 1782225860.75 total 130.11 last 66.75) - open/close 9949 (time 1782225870.76 total 140.11 last 62.08) total: 10000 open/close in 140.85 seconds: 71.00 ops/second llite.lustre-ffff9f4c9874b800.statahead_batch_max=0 llite.lustre-ffff9f4c9874b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f4c9874b800.stats=clear mdc.lustre-MDT0001-mdc-ffff9f4c9874b800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.316s user 0m0.326s sys 0m18.328s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9f4c9874b800.statahead_batch_max=256 llite.lustre-ffff9f4c9874b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f4c9874b800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9f4c9874b800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9f4c9874b800.stats=clear mdc.lustre-MDT0001-mdc-ffff9f4c9874b800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m53.071s user 0m0.280s sys 0m17.984s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9f4c9874b800.batch_stats= snapshot_time: 1782226003.884366335 secs.nsecs start_time: 1782225938.203384365 secs.nsecs elapsed_time: 65.680981970 secs.nsecs subreqs per batch batches %% cum %% 1: 280 57 57 2: 71 14 72 4: 45 9 81 8: 21 4 85 16: 12 2 88 32: 19 3 92 64: 2 0 92 128: 2 0 93 256: 33 6 100 mdc.lustre-MDT0001-mdc-ffff9f4c9874b800.batch_stats= snapshot_time: 1782226003.884540147 secs.nsecs start_time: 1782225938.203651917 secs.nsecs elapsed_time: 65.680888230 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9f4c9874b800.statahead_batch_max=64 llite.lustre-ffff9f4c9874b800.statahead_max=128