== sanity test 123ad: Verify batching statahead works correctly ========================================================== 23:02:02 (1788750122) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8a464a0f7000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a464a0f7000.batch_stats=0 llite.lustre-ffff8a464a0f7000.statahead_max=32 llite.lustre-ffff8a464a0f7000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8a464a0f7000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a464a0f7000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.72 seconds: 36.71 ops/second llite.lustre-ffff8a464a0f7000.statahead_max=0 101 real 0m2.496s user 0m0.005s sys 0m0.320s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8a464a0f7000.statahead_max=32 llite.lustre-ffff8a464a0f7000.statahead_max=32 101 real 0m0.955s user 0m0.008s sys 0m0.165s 'ls -l' 100 files with statahead: 1 sec statahead total: 129 statahead wrong: 2 agl total: 124 list_total: 30 fname_total: 85 hit_total: 4078 miss_total: 1640 swrong=2 ewrong=2 - open/close 728 (time 1788750150.61 total 10.00 last 72.79) total: 900 open/close in 12.33 seconds: 73.01 ops/second llite.lustre-ffff8a464a0f7000.statahead_max=0 1001 real 0m20.802s user 0m0.035s sys 0m3.084s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff8a464a0f7000.statahead_max=32 llite.lustre-ffff8a464a0f7000.statahead_max=32 1001 real 0m8.241s user 0m0.031s sys 0m1.320s 'ls -l' 1000 files with statahead: 9 sec statahead total: 130 statahead wrong: 2 agl total: 125 list_total: 31 fname_total: 85 hit_total: 5075 miss_total: 1643 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff8a464a0f7000.statahead_stats= statahead total: 130 statahead wrong: 2 agl total: 125 list_total: 31 fname_total: 85 hit_total: 5075 miss_total: 1643 mdc.lustre-MDT0000-mdc-ffff8a464a0f7000.batch_stats= snapshot_time: 1788750216.371552723 secs.nsecs start_time: 1788750127.673794357 secs.nsecs elapsed_time: 88.697758366 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 3 7 15 16: 19 48 64 32: 14 35 100 mdc.lustre-MDT0001-mdc-ffff8a464a0f7000.batch_stats= snapshot_time: 1788750216.371628656 secs.nsecs start_time: 1788750127.673870216 secs.nsecs elapsed_time: 88.697758440 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 5 5 8: 3 7 12 16: 22 56 69 32: 12 30 100 - open/close 645 (time 1788750228.98 total 10.01 last 64.46) - open/close 1310 (time 1788750238.99 total 20.02 last 66.42) - open/close 1927 (time 1788750249.00 total 30.02 last 61.67) - open/close 2643 (time 1788750259.01 total 40.03 last 71.55) - open/close 3276 (time 1788750269.01 total 50.03 last 63.29) - open/close 3937 (time 1788750279.01 total 60.04 last 66.07) - open/close 4631 (time 1788750289.03 total 70.05 last 69.30) - open/close 5410 (time 1788750299.03 total 80.06 last 77.87) - open/close 6135 (time 1788750309.04 total 90.06 last 72.46) - open/close 6929 (time 1788750319.04 total 100.07 last 79.34) - open/close 7711 (time 1788750329.05 total 110.07 last 78.17) - open/close 8491 (time 1788750339.06 total 120.08 last 77.92) - open/close 9255 (time 1788750349.06 total 130.09 last 76.38) total: 10000 open/close in 139.94 seconds: 71.46 ops/second llite.lustre-ffff8a464a0f7000.statahead_batch_max=0 llite.lustre-ffff8a464a0f7000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a464a0f7000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a464a0f7000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.160s user 0m0.317s sys 0m14.773s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a464a0f7000.statahead_batch_max=32 llite.lustre-ffff8a464a0f7000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a464a0f7000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a464a0f7000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a464a0f7000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a464a0f7000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m45.558s user 0m0.323s sys 0m13.896s batched statahead RPC= hit count=9976 mdc.lustre-MDT0000-mdc-ffff8a464a0f7000.batch_stats= snapshot_time: 1788750541.834916266 secs.nsecs start_time: 1788750426.123960147 secs.nsecs elapsed_time: 115.710956119 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8a464a0f7000.batch_stats= snapshot_time: 1788750541.834994474 secs.nsecs start_time: 1788750426.124037894 secs.nsecs elapsed_time: 115.710956580 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8a464a0f7000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a464a0f7000.batch_stats=0 llite.lustre-ffff8a464a0f7000.statahead_max=2048 llite.lustre-ffff8a464a0f7000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8a464a0f7000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8a464a0f7000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.55 seconds: 39.17 ops/second llite.lustre-ffff8a464a0f7000.statahead_max=0 101 real 0m3.725s user 0m0.006s sys 0m0.367s 'ls -l' 100 files without statahead: 4 sec llite.lustre-ffff8a464a0f7000.statahead_max=2048 llite.lustre-ffff8a464a0f7000.statahead_max=2048 101 real 0m1.006s user 0m0.003s sys 0m0.211s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10075 miss_total: 25 swrong=0 ewrong=0 - open/close 719 (time 1788750866.19 total 10.01 last 71.83) total: 900 open/close in 12.74 seconds: 70.64 ops/second llite.lustre-ffff8a464a0f7000.statahead_max=0 1001 real 0m25.986s user 0m0.040s sys 0m3.088s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff8a464a0f7000.statahead_max=2048 llite.lustre-ffff8a464a0f7000.statahead_max=2048 1001 real 0m6.990s user 0m0.028s sys 0m1.200s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11074 miss_total: 26 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff8a464a0f7000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11074 miss_total: 26 mdc.lustre-MDT0000-mdc-ffff8a464a0f7000.batch_stats= snapshot_time: 1788750935.646075675 secs.nsecs start_time: 1788750544.763688516 secs.nsecs elapsed_time: 390.882387159 secs.nsecs subreqs per batch batches %% cum %% 1: 2 28 28 2: 0 0 28 4: 3 42 71 8: 0 0 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff8a464a0f7000.batch_stats= snapshot_time: 1788750935.646140474 secs.nsecs start_time: 1788750544.763888610 secs.nsecs elapsed_time: 390.882251864 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 10 10 8: 2 20 30 16: 1 10 40 32: 3 30 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 757 (time 1788750948.49 total 10.01 last 75.64) - open/close 1401 (time 1788750958.51 total 20.03 last 64.29) - open/close 2037 (time 1788750968.52 total 30.03 last 63.57) - open/close 2856 (time 1788750978.52 total 40.03 last 81.86) - open/close 3586 (time 1788750988.54 total 50.05 last 72.86) - open/close 4299 (time 1788750998.55 total 60.06 last 71.23) - open/close 5036 (time 1788751008.55 total 70.06 last 73.70) - open/close 5797 (time 1788751018.56 total 80.07 last 76.06) - open/close 6516 (time 1788751028.58 total 90.09 last 71.74) - open/close 7204 (time 1788751038.59 total 100.10 last 68.71) - open/close 7966 (time 1788751048.59 total 110.11 last 76.19) - open/close 8777 (time 1788751058.59 total 120.11 last 81.10) - open/close 9510 (time 1788751068.62 total 130.13 last 73.12) total: 10000 open/close in 137.64 seconds: 72.65 ops/second llite.lustre-ffff8a464a0f7000.statahead_batch_max=0 llite.lustre-ffff8a464a0f7000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a464a0f7000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a464a0f7000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.136s user 0m0.298s sys 0m14.845s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8a464a0f7000.statahead_batch_max=256 llite.lustre-ffff8a464a0f7000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8a464a0f7000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8a464a0f7000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8a464a0f7000.stats=clear mdc.lustre-MDT0001-mdc-ffff8a464a0f7000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m49.211s user 0m0.290s sys 0m14.251s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8a464a0f7000.batch_stats= snapshot_time: 1788751203.314502327 secs.nsecs start_time: 1788751143.959928464 secs.nsecs elapsed_time: 59.354573863 secs.nsecs subreqs per batch batches %% cum %% 1: 54 31 31 2: 21 12 43 4: 15 8 51 8: 12 6 58 16: 10 5 64 32: 24 13 78 64: 3 1 79 128: 0 0 79 256: 35 20 100 mdc.lustre-MDT0001-mdc-ffff8a464a0f7000.batch_stats= snapshot_time: 1788751203.314572489 secs.nsecs start_time: 1788751143.960164366 secs.nsecs elapsed_time: 59.354408123 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8a464a0f7000.statahead_batch_max=64 llite.lustre-ffff8a464a0f7000.statahead_max=128