== sanity test 123ad: Verify batching statahead works correctly ========================================================== 14:37:00 (1783967820) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8c11847cf800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c11847cf800.batch_stats=0 llite.lustre-ffff8c11847cf800.statahead_max=32 llite.lustre-ffff8c11847cf800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8c11847cf800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c11847cf800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.44 seconds: 41.03 ops/second llite.lustre-ffff8c11847cf800.statahead_max=0 101 real 0m2.540s user 0m0.005s sys 0m0.340s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8c11847cf800.statahead_max=32 llite.lustre-ffff8c11847cf800.statahead_max=32 101 real 0m1.127s user 0m0.003s sys 0m0.192s 'ls -l' 100 files with statahead: 1 sec statahead total: 103 statahead wrong: 2 agl total: 98 list_total: 31 fname_total: 70 hit_total: 4605 miss_total: 1284 swrong=2 ewrong=2 - open/close 837 (time 1783967846.45 total 10.01 last 83.59) total: 900 open/close in 10.77 seconds: 83.55 ops/second llite.lustre-ffff8c11847cf800.statahead_max=0 1001 real 0m19.027s user 0m0.022s sys 0m3.104s 'ls -l' 1000 files without statahead: 19 sec llite.lustre-ffff8c11847cf800.statahead_max=32 llite.lustre-ffff8c11847cf800.statahead_max=32 1001 real 0m7.601s user 0m0.034s sys 0m1.610s 'ls -l' 1000 files with statahead: 8 sec statahead total: 104 statahead wrong: 2 agl total: 99 list_total: 32 fname_total: 70 hit_total: 5603 miss_total: 1286 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 16 seconds llite.lustre-ffff8c11847cf800.statahead_stats= statahead total: 104 statahead wrong: 2 agl total: 99 list_total: 32 fname_total: 70 hit_total: 5603 miss_total: 1286 mdc.lustre-MDT0000-mdc-ffff8c11847cf800.batch_stats= snapshot_time: 1783967899.615803005 secs.nsecs start_time: 1783967824.255767304 secs.nsecs elapsed_time: 75.360035701 secs.nsecs subreqs per batch batches %% cum %% 1: 3 7 7 2: 2 5 13 4: 2 5 18 8: 3 7 26 16: 23 60 86 32: 5 13 100 mdc.lustre-MDT0001-mdc-ffff8c11847cf800.batch_stats= snapshot_time: 1783967899.615867021 secs.nsecs start_time: 1783967824.255857354 secs.nsecs elapsed_time: 75.360009667 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 3 7 12 16: 12 30 43 32: 22 56 100 - open/close 871 (time 1783967911.72 total 10.01 last 87.05) - open/close 1729 (time 1783967921.73 total 20.01 last 85.77) - open/close 2611 (time 1783967931.76 total 30.05 last 87.87) - open/close 3507 (time 1783967941.77 total 40.06 last 89.52) - open/close 4332 (time 1783967951.78 total 50.06 last 82.42) - open/close 5171 (time 1783967961.79 total 60.07 last 83.84) - open/close 6057 (time 1783967971.80 total 70.08 last 88.56) - open/close 6888 (time 1783967981.80 total 80.08 last 83.06) - open/close 7763 (time 1783967991.81 total 90.09 last 87.46) - open/close 8642 (time 1783968001.82 total 100.10 last 87.74) - open/close 9507 (time 1783968011.82 total 110.11 last 86.50) total: 10000 open/close in 116.19 seconds: 86.07 ops/second llite.lustre-ffff8c11847cf800.statahead_batch_max=0 llite.lustre-ffff8c11847cf800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c11847cf800.stats=clear mdc.lustre-MDT0001-mdc-ffff8c11847cf800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m55.842s user 0m0.300s sys 0m18.351s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8c11847cf800.statahead_batch_max=32 llite.lustre-ffff8c11847cf800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c11847cf800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8c11847cf800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8c11847cf800.stats=clear mdc.lustre-MDT0001-mdc-ffff8c11847cf800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m40.322s user 0m0.275s sys 0m15.916s batched statahead RPC= hit count=9987 mdc.lustre-MDT0000-mdc-ffff8c11847cf800.batch_stats= snapshot_time: 1783968199.106785894 secs.nsecs start_time: 1783968084.179445204 secs.nsecs elapsed_time: 114.927340690 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8c11847cf800.batch_stats= snapshot_time: 1783968199.106852109 secs.nsecs start_time: 1783968084.179569182 secs.nsecs elapsed_time: 114.927282927 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8c11847cf800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c11847cf800.batch_stats=0 llite.lustre-ffff8c11847cf800.statahead_max=2048 llite.lustre-ffff8c11847cf800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8c11847cf800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c11847cf800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.92 seconds: 34.21 ops/second llite.lustre-ffff8c11847cf800.statahead_max=0 101 real 0m1.878s user 0m0.003s sys 0m0.328s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8c11847cf800.statahead_max=2048 llite.lustre-ffff8c11847cf800.statahead_max=2048 101 real 0m0.868s user 0m0.003s sys 0m0.201s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10086 miss_total: 14 swrong=0 ewrong=0 total: 900 open/close in 9.71 seconds: 92.67 ops/second llite.lustre-ffff8c11847cf800.statahead_max=0 1001 real 0m19.736s user 0m0.033s sys 0m3.004s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff8c11847cf800.statahead_max=2048 llite.lustre-ffff8c11847cf800.statahead_max=2048 1001 real 0m5.214s user 0m0.020s sys 0m1.709s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff8c11847cf800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 mdc.lustre-MDT0000-mdc-ffff8c11847cf800.batch_stats= snapshot_time: 1783968537.963906420 secs.nsecs start_time: 1783968201.303404789 secs.nsecs elapsed_time: 336.660501631 secs.nsecs subreqs per batch batches %% cum %% 1: 2 25 25 2: 1 12 37 4: 2 25 62 8: 0 0 62 16: 0 0 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff8c11847cf800.batch_stats= snapshot_time: 1783968537.963980126 secs.nsecs start_time: 1783968201.303547470 secs.nsecs elapsed_time: 336.660432656 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 4 44 55 16: 0 0 55 32: 1 11 66 64: 0 0 66 128: 1 11 77 256: 2 22 100 - open/close 872 (time 1783968550.20 total 10.01 last 87.12) - open/close 1768 (time 1783968560.20 total 20.01 last 89.58) - open/close 2627 (time 1783968570.20 total 30.01 last 85.88) - open/close 3486 (time 1783968580.22 total 40.03 last 85.78) - open/close 4177 (time 1783968590.23 total 50.04 last 69.03) - open/close 4985 (time 1783968600.23 total 60.04 last 80.76) - open/close 5650 (time 1783968610.24 total 70.05 last 66.43) - open/close 6321 (time 1783968620.25 total 80.06 last 67.08) - open/close 7155 (time 1783968630.25 total 90.06 last 83.36) - open/close 7866 (time 1783968640.26 total 100.07 last 71.04) - open/close 8714 (time 1783968650.26 total 110.07 last 84.79) - open/close 9645 (time 1783968660.27 total 120.08 last 93.03) total: 10000 open/close in 124.46 seconds: 80.35 ops/second llite.lustre-ffff8c11847cf800.statahead_batch_max=0 llite.lustre-ffff8c11847cf800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c11847cf800.stats=clear mdc.lustre-MDT0001-mdc-ffff8c11847cf800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m0.433s user 0m0.286s sys 0m18.429s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8c11847cf800.statahead_batch_max=256 llite.lustre-ffff8c11847cf800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c11847cf800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8c11847cf800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8c11847cf800.stats=clear mdc.lustre-MDT0001-mdc-ffff8c11847cf800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m52.173s user 0m0.269s sys 0m18.153s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8c11847cf800.batch_stats= snapshot_time: 1783968801.737179147 secs.nsecs start_time: 1783968734.996696296 secs.nsecs elapsed_time: 66.740482851 secs.nsecs subreqs per batch batches %% cum %% 1: 275 56 56 2: 75 15 72 4: 41 8 80 8: 23 4 85 16: 12 2 88 32: 17 3 91 64: 6 1 92 128: 1 0 93 256: 33 6 100 mdc.lustre-MDT0001-mdc-ffff8c11847cf800.batch_stats= snapshot_time: 1783968801.737449520 secs.nsecs start_time: 1783968734.996922189 secs.nsecs elapsed_time: 66.740527331 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8c11847cf800.statahead_batch_max=64 llite.lustre-ffff8c11847cf800.statahead_max=128