== sanity test 123ad: Verify batching statahead works correctly ========================================================== 22:38:35 (1787711915) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9db31853a800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9db31853a800.batch_stats=0 llite.lustre-ffff9db31853a800.statahead_max=32 llite.lustre-ffff9db31853a800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9db31853a800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9db31853a800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 3.13 seconds: 31.92 ops/second llite.lustre-ffff9db31853a800.statahead_max=0 101 real 0m2.760s user 0m0.008s sys 0m0.316s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9db31853a800.statahead_max=32 llite.lustre-ffff9db31853a800.statahead_max=32 101 real 0m1.058s user 0m0.004s sys 0m0.179s 'ls -l' 100 files with statahead: 1 sec statahead total: 125 statahead wrong: 2 agl total: 119 list_total: 30 fname_total: 90 hit_total: 4033 miss_total: 1433 swrong=2 ewrong=2 - open/close 714 (time 1787711945.41 total 10.01 last 71.35) total: 900 open/close in 12.35 seconds: 72.89 ops/second llite.lustre-ffff9db31853a800.statahead_max=0 1001 real 0m25.425s user 0m0.037s sys 0m3.059s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff9db31853a800.statahead_max=32 llite.lustre-ffff9db31853a800.statahead_max=32 1001 real 0m9.645s user 0m0.029s sys 0m1.400s 'ls -l' 1000 files with statahead: 10 sec statahead total: 126 statahead wrong: 2 agl total: 120 list_total: 31 fname_total: 90 hit_total: 5031 miss_total: 1435 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 31 seconds llite.lustre-ffff9db31853a800.statahead_stats= statahead total: 126 statahead wrong: 2 agl total: 120 list_total: 31 fname_total: 90 hit_total: 5031 miss_total: 1435 mdc.lustre-MDT0000-mdc-ffff9db31853a800.batch_stats= snapshot_time: 1787712027.501102129 secs.nsecs start_time: 1787711920.843852265 secs.nsecs elapsed_time: 106.657249864 secs.nsecs subreqs per batch batches %% cum %% 1: 3 8 8 2: 2 5 13 4: 0 0 13 8: 3 8 21 16: 23 62 83 32: 6 16 100 mdc.lustre-MDT0001-mdc-ffff9db31853a800.batch_stats= snapshot_time: 1787712027.501421200 secs.nsecs start_time: 1787711920.843931985 secs.nsecs elapsed_time: 106.657489215 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 3 7 12 16: 12 30 43 32: 22 56 100 - open/close 672 (time 1787712040.37 total 10.01 last 67.12) - open/close 1420 (time 1787712050.38 total 20.02 last 74.75) - open/close 2004 (time 1787712060.38 total 30.02 last 58.39) - open/close 2583 (time 1787712070.39 total 40.04 last 57.81) - open/close 3177 (time 1787712080.39 total 50.04 last 59.40) - open/close 3912 (time 1787712090.39 total 60.04 last 73.50) - open/close 4497 (time 1787712100.41 total 70.06 last 58.39) - open/close 5206 (time 1787712110.42 total 80.06 last 70.88) - open/close 5916 (time 1787712120.43 total 90.07 last 70.91) - open/close 6597 (time 1787712130.43 total 100.08 last 68.07) - open/close 7326 (time 1787712140.44 total 110.08 last 72.87) - open/close 8011 (time 1787712150.44 total 120.09 last 68.47) - open/close 8721 (time 1787712160.45 total 130.10 last 70.93) - open/close 9425 (time 1787712170.45 total 140.10 last 70.40) total: 10000 open/close in 148.05 seconds: 67.55 ops/second llite.lustre-ffff9db31853a800.statahead_batch_max=0 llite.lustre-ffff9db31853a800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9db31853a800.stats=clear mdc.lustre-MDT0001-mdc-ffff9db31853a800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.162s user 0m0.328s sys 0m15.756s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9db31853a800.statahead_batch_max=32 llite.lustre-ffff9db31853a800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9db31853a800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9db31853a800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9db31853a800.stats=clear mdc.lustre-MDT0001-mdc-ffff9db31853a800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m44.879s user 0m0.289s sys 0m13.963s batched statahead RPC= hit count=9978 mdc.lustre-MDT0000-mdc-ffff9db31853a800.batch_stats= snapshot_time: 1787712370.458819137 secs.nsecs start_time: 1787712255.733758766 secs.nsecs elapsed_time: 114.725060371 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9db31853a800.batch_stats= snapshot_time: 1787712370.458957184 secs.nsecs start_time: 1787712255.733897216 secs.nsecs elapsed_time: 114.725059968 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9db31853a800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9db31853a800.batch_stats=0 llite.lustre-ffff9db31853a800.statahead_max=2048 llite.lustre-ffff9db31853a800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9db31853a800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9db31853a800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.37 seconds: 29.67 ops/second llite.lustre-ffff9db31853a800.statahead_max=0 101 real 0m2.595s user 0m0.004s sys 0m0.326s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9db31853a800.statahead_max=2048 llite.lustre-ffff9db31853a800.statahead_max=2048 101 real 0m0.845s user 0m0.002s sys 0m0.159s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10077 miss_total: 23 swrong=0 ewrong=0 - open/close 767 (time 1787712701.25 total 10.00 last 76.67) total: 900 open/close in 12.02 seconds: 74.90 ops/second llite.lustre-ffff9db31853a800.statahead_max=0 1001 real 0m23.750s user 0m0.049s sys 0m3.074s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff9db31853a800.statahead_max=2048 llite.lustre-ffff9db31853a800.statahead_max=2048 1001 real 0m6.303s user 0m0.032s sys 0m1.600s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff9db31853a800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11076 miss_total: 24 mdc.lustre-MDT0000-mdc-ffff9db31853a800.batch_stats= snapshot_time: 1787712767.259613887 secs.nsecs start_time: 1787712372.347857533 secs.nsecs elapsed_time: 394.911756354 secs.nsecs subreqs per batch batches %% cum %% 1: 1 10 10 2: 0 0 10 4: 1 10 20 8: 3 30 50 16: 2 20 70 32: 0 0 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 mdc.lustre-MDT0001-mdc-ffff9db31853a800.batch_stats= snapshot_time: 1787712767.259689417 secs.nsecs start_time: 1787712372.348580301 secs.nsecs elapsed_time: 394.911109116 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 30 30 8: 2 20 50 16: 1 10 60 32: 2 20 80 64: 0 0 80 128: 0 0 80 256: 2 20 100 - open/close 651 (time 1787712779.83 total 10.01 last 65.03) - open/close 1358 (time 1787712789.84 total 20.02 last 70.64) - open/close 2088 (time 1787712799.84 total 30.02 last 72.98) - open/close 2768 (time 1787712809.85 total 40.03 last 67.95) - open/close 3470 (time 1787712819.85 total 50.03 last 70.17) - open/close 4202 (time 1787712829.85 total 60.04 last 73.16) - open/close 4983 (time 1787712839.86 total 70.04 last 78.06) - open/close 5660 (time 1787712849.86 total 80.05 last 67.67) - open/close 6319 (time 1787712859.86 total 90.05 last 65.89) - open/close 7000 (time 1787712869.88 total 100.06 last 68.00) - open/close 7689 (time 1787712879.89 total 110.08 last 68.81) - open/close 8327 (time 1787712889.91 total 120.09 last 63.70) - open/close 8964 (time 1787712899.93 total 130.11 last 63.56) - open/close 9681 (time 1787712909.93 total 140.11 last 71.69) total: 10000 open/close in 144.61 seconds: 69.15 ops/second llite.lustre-ffff9db31853a800.statahead_batch_max=0 llite.lustre-ffff9db31853a800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9db31853a800.stats=clear mdc.lustre-MDT0001-mdc-ffff9db31853a800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m17.772s user 0m0.306s sys 0m15.186s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9db31853a800.statahead_batch_max=256 llite.lustre-ffff9db31853a800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9db31853a800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9db31853a800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9db31853a800.stats=clear mdc.lustre-MDT0001-mdc-ffff9db31853a800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m52.605s user 0m0.286s sys 0m14.169s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9db31853a800.batch_stats= snapshot_time: 1787713063.693987984 secs.nsecs start_time: 1787713000.225863592 secs.nsecs elapsed_time: 63.468124392 secs.nsecs subreqs per batch batches %% cum %% 1: 94 38 38 2: 41 16 54 4: 26 10 65 8: 17 6 72 16: 13 5 77 32: 14 5 82 64: 6 2 85 128: 2 0 86 256: 34 13 100 mdc.lustre-MDT0001-mdc-ffff9db31853a800.batch_stats= snapshot_time: 1787713063.694059891 secs.nsecs start_time: 1787713000.226020010 secs.nsecs elapsed_time: 63.468039881 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9db31853a800.statahead_batch_max=64 llite.lustre-ffff9db31853a800.statahead_max=128