== sanity test 123ad: Verify batching statahead works correctly ========================================================== 05:53:14 (1787392394) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff943745a5d000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff943745a5d000.batch_stats=0 llite.lustre-ffff943745a5d000.statahead_max=32 llite.lustre-ffff943745a5d000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff943745a5d000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff943745a5d000.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.42 seconds: 41.40 ops/second llite.lustre-ffff943745a5d000.statahead_max=0 101 real 0m2.119s user 0m0.003s sys 0m0.332s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff943745a5d000.statahead_max=32 llite.lustre-ffff943745a5d000.statahead_max=32 101 real 0m1.004s user 0m0.006s sys 0m0.179s 'ls -l' 100 files with statahead: 1 sec statahead total: 136 statahead wrong: 2 agl total: 130 list_total: 31 fname_total: 105 hit_total: 4131 miss_total: 1411 swrong=2 ewrong=2 - open/close 652 (time 1787392423.01 total 10.01 last 65.16) total: 900 open/close in 13.51 seconds: 66.61 ops/second llite.lustre-ffff943745a5d000.statahead_max=0 1001 real 0m20.435s user 0m0.031s sys 0m3.358s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff943745a5d000.statahead_max=32 llite.lustre-ffff943745a5d000.statahead_max=32 1001 real 0m9.325s user 0m0.028s sys 0m1.504s 'ls -l' 1000 files with statahead: 10 sec statahead total: 137 statahead wrong: 2 agl total: 131 list_total: 32 fname_total: 105 hit_total: 5128 miss_total: 1414 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff943745a5d000.statahead_stats= statahead total: 137 statahead wrong: 2 agl total: 131 list_total: 32 fname_total: 105 hit_total: 5128 miss_total: 1414 mdc.lustre-MDT0000-mdc-ffff943745a5d000.batch_stats= snapshot_time: 1787392492.878809683 secs.nsecs start_time: 1787392400.373119728 secs.nsecs elapsed_time: 92.505689955 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 3 7 10 8: 3 7 17 16: 17 42 60 32: 16 40 100 mdc.lustre-MDT0001-mdc-ffff943745a5d000.batch_stats= snapshot_time: 1787392492.878979374 secs.nsecs start_time: 1787392400.373257058 secs.nsecs elapsed_time: 92.505722316 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 5 12 12 8: 1 2 15 16: 20 51 66 32: 13 33 100 - open/close 713 (time 1787392505.63 total 10.01 last 71.26) - open/close 1460 (time 1787392515.63 total 20.01 last 74.70) - open/close 2225 (time 1787392525.63 total 30.01 last 76.46) - open/close 3047 (time 1787392535.64 total 40.02 last 82.16) - open/close 3889 (time 1787392545.64 total 50.02 last 84.17) - open/close 4642 (time 1787392555.64 total 60.02 last 75.27) - open/close 5443 (time 1787392565.65 total 70.03 last 80.05) - open/close 6234 (time 1787392575.66 total 80.04 last 79.03) - open/close 6980 (time 1787392585.66 total 90.04 last 74.58) - open/close 7804 (time 1787392595.66 total 100.04 last 82.38) - open/close 8661 (time 1787392605.67 total 110.05 last 85.68) - open/close 9393 (time 1787392615.67 total 120.05 last 73.15) total: 10000 open/close in 128.56 seconds: 77.78 ops/second llite.lustre-ffff943745a5d000.statahead_batch_max=0 llite.lustre-ffff943745a5d000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff943745a5d000.stats=clear mdc.lustre-MDT0001-mdc-ffff943745a5d000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m10.226s user 0m0.305s sys 0m15.840s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff943745a5d000.statahead_batch_max=32 llite.lustre-ffff943745a5d000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff943745a5d000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff943745a5d000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff943745a5d000.stats=clear mdc.lustre-MDT0001-mdc-ffff943745a5d000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m57.965s user 0m0.315s sys 0m15.334s batched statahead RPC= hit count=9976 mdc.lustre-MDT0000-mdc-ffff943745a5d000.batch_stats= snapshot_time: 1787392838.248088920 secs.nsecs start_time: 1787392704.814755712 secs.nsecs elapsed_time: 133.433333208 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff943745a5d000.batch_stats= snapshot_time: 1787392838.248156416 secs.nsecs start_time: 1787392704.814855718 secs.nsecs elapsed_time: 133.433300698 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff943745a5d000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff943745a5d000.batch_stats=0 llite.lustre-ffff943745a5d000.statahead_max=2048 llite.lustre-ffff943745a5d000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff943745a5d000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff943745a5d000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.28 seconds: 43.83 ops/second llite.lustre-ffff943745a5d000.statahead_max=0 101 real 0m2.440s user 0m0.003s sys 0m0.320s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff943745a5d000.statahead_max=2048 llite.lustre-ffff943745a5d000.statahead_max=2048 101 real 0m0.829s user 0m0.007s sys 0m0.180s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10074 miss_total: 26 swrong=0 ewrong=0 - open/close 758 (time 1787393150.79 total 10.00 last 75.78) total: 900 open/close in 11.83 seconds: 76.09 ops/second llite.lustre-ffff943745a5d000.statahead_max=0 1001 real 0m26.543s user 0m0.035s sys 0m3.451s 'ls -l' 1000 files without statahead: 27 sec llite.lustre-ffff943745a5d000.statahead_max=2048 llite.lustre-ffff943745a5d000.statahead_max=2048 1001 real 0m6.559s user 0m0.018s sys 0m1.501s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11073 miss_total: 27 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff943745a5d000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11073 miss_total: 27 mdc.lustre-MDT0000-mdc-ffff943745a5d000.batch_stats= snapshot_time: 1787393217.612183115 secs.nsecs start_time: 1787392840.616735052 secs.nsecs elapsed_time: 376.995448063 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 14 14 4: 2 28 42 8: 2 28 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff943745a5d000.batch_stats= snapshot_time: 1787393217.612243466 secs.nsecs start_time: 1787392840.616903765 secs.nsecs elapsed_time: 376.995339701 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 10 10 8: 2 20 30 16: 3 30 60 32: 1 10 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 848 (time 1787393229.81 total 10.01 last 84.74) - open/close 1557 (time 1787393239.81 total 20.01 last 70.89) - open/close 2314 (time 1787393249.81 total 30.01 last 75.66) - open/close 3059 (time 1787393259.82 total 40.02 last 74.43) - open/close 3742 (time 1787393269.83 total 50.03 last 68.29) - open/close 4439 (time 1787393279.83 total 60.03 last 69.65) - open/close 5092 (time 1787393289.85 total 70.05 last 65.17) - open/close 5822 (time 1787393299.87 total 80.07 last 72.91) - open/close 6571 (time 1787393309.88 total 90.08 last 74.82) - open/close 7323 (time 1787393319.90 total 100.10 last 75.04) - open/close 7913 (time 1787393329.91 total 110.10 last 58.97) - open/close 8665 (time 1787393339.91 total 120.11 last 75.19) - open/close 9463 (time 1787393349.92 total 130.11 last 79.73) total: 10000 open/close in 137.78 seconds: 72.58 ops/second llite.lustre-ffff943745a5d000.statahead_batch_max=0 llite.lustre-ffff943745a5d000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff943745a5d000.stats=clear mdc.lustre-MDT0001-mdc-ffff943745a5d000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.406s user 0m0.299s sys 0m15.783s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff943745a5d000.statahead_batch_max=256 llite.lustre-ffff943745a5d000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff943745a5d000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff943745a5d000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff943745a5d000.stats=clear mdc.lustre-MDT0001-mdc-ffff943745a5d000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.221s user 0m0.329s sys 0m16.471s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff943745a5d000.batch_stats= snapshot_time: 1787393501.868989135 secs.nsecs start_time: 1787393427.323599208 secs.nsecs elapsed_time: 74.545389927 secs.nsecs subreqs per batch batches %% cum %% 1: 162 54 54 2: 31 10 64 4: 27 9 73 8: 12 4 77 16: 8 2 80 32: 19 6 86 64: 5 1 88 128: 0 0 88 256: 35 11 100 mdc.lustre-MDT0001-mdc-ffff943745a5d000.batch_stats= snapshot_time: 1787393501.869052465 secs.nsecs start_time: 1787393427.323892488 secs.nsecs elapsed_time: 74.545159977 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff943745a5d000.statahead_batch_max=64 llite.lustre-ffff943745a5d000.statahead_max=128