== sanity test 123ad: Verify batching statahead works correctly ========================================================== 07:51:23 (1787745083) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats=0 llite.lustre-ffff93f6a0950800.statahead_max=32 llite.lustre-ffff93f6a0950800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.86 seconds: 34.99 ops/second llite.lustre-ffff93f6a0950800.statahead_max=0 101 real 0m2.715s user 0m0.005s sys 0m0.305s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff93f6a0950800.statahead_max=32 llite.lustre-ffff93f6a0950800.statahead_max=32 101 real 0m1.322s user 0m0.004s sys 0m0.151s 'ls -l' 100 files with statahead: 2 sec statahead total: 163 statahead wrong: 2 agl total: 156 list_total: 31 fname_total: 112 hit_total: 4360 miss_total: 781 swrong=2 ewrong=2 - open/close 680 (time 1787745113.28 total 10.00 last 68.00) total: 900 open/close in 14.03 seconds: 64.14 ops/second llite.lustre-ffff93f6a0950800.statahead_max=0 1001 real 0m31.551s user 0m0.037s sys 0m2.862s 'ls -l' 1000 files without statahead: 32 sec llite.lustre-ffff93f6a0950800.statahead_max=32 llite.lustre-ffff93f6a0950800.statahead_max=32 1001 real 0m9.922s user 0m0.020s sys 0m1.393s 'ls -l' 1000 files with statahead: 10 sec statahead total: 164 statahead wrong: 2 agl total: 157 list_total: 32 fname_total: 112 hit_total: 5354 miss_total: 787 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff93f6a0950800.statahead_stats= statahead total: 164 statahead wrong: 2 agl total: 157 list_total: 32 fname_total: 112 hit_total: 5354 miss_total: 787 mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787745195.372439914 secs.nsecs start_time: 1787745088.626704931 secs.nsecs elapsed_time: 106.745734983 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 3 7 10 8: 4 10 20 16: 20 50 70 32: 12 30 100 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787745195.372506433 secs.nsecs start_time: 1787745088.626844544 secs.nsecs elapsed_time: 106.745661889 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 7 7 8: 2 5 12 16: 21 53 66 32: 13 33 100 - open/close 759 (time 1787745207.87 total 10.00 last 75.90) - open/close 1484 (time 1787745217.88 total 20.01 last 72.45) - open/close 2199 (time 1787745227.89 total 30.02 last 71.40) - open/close 2823 (time 1787745237.92 total 40.05 last 62.20) - open/close 3425 (time 1787745247.93 total 50.06 last 60.15) - open/close 3875 (time 1787745257.94 total 60.07 last 44.98) - open/close 4359 (time 1787745267.96 total 70.08 last 48.31) - open/close 4815 (time 1787745277.96 total 80.09 last 45.59) - open/close 5371 (time 1787745288.00 total 90.13 last 55.38) - open/close 6009 (time 1787745298.00 total 100.13 last 63.76) - open/close 6603 (time 1787745308.01 total 110.13 last 59.38) - open/close 7098 (time 1787745318.04 total 120.17 last 49.35) - open/close 7518 (time 1787745328.04 total 130.17 last 41.99) - open/close 8017 (time 1787745338.04 total 140.17 last 49.89) - open/close 8620 (time 1787745348.05 total 150.18 last 60.26) - open/close 9187 (time 1787745358.06 total 160.19 last 56.65) - open/close 9849 (time 1787745368.06 total 170.19 last 66.18) total: 10000 open/close in 172.28 seconds: 58.05 ops/second llite.lustre-ffff93f6a0950800.statahead_batch_max=0 llite.lustre-ffff93f6a0950800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f6a0950800.stats=clear mdc.lustre-MDT0001-mdc-ffff93f6a0950800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m22.127s user 0m0.319s sys 0m14.297s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff93f6a0950800.statahead_batch_max=32 llite.lustre-ffff93f6a0950800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff93f6a0950800.stats=clear mdc.lustre-MDT0001-mdc-ffff93f6a0950800.stats=clear ls -l files in d123ad.sanity: 10001 real 2m9.080s user 0m0.284s sys 0m13.983s batched statahead RPC= hit count=9971 mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787745604.274694705 secs.nsecs start_time: 1787745463.247936796 secs.nsecs elapsed_time: 141.026757909 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 2 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787745604.276170873 secs.nsecs start_time: 1787745463.248038767 secs.nsecs elapsed_time: 141.028132106 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats=0 llite.lustre-ffff93f6a0950800.statahead_max=2048 llite.lustre-ffff93f6a0950800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 4.04 seconds: 24.74 ops/second llite.lustre-ffff93f6a0950800.statahead_max=0 101 real 0m2.728s user 0m0.005s sys 0m0.320s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff93f6a0950800.statahead_max=2048 llite.lustre-ffff93f6a0950800.statahead_max=2048 101 real 0m0.878s user 0m0.005s sys 0m0.158s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10070 miss_total: 30 swrong=0 ewrong=0 - open/close 788 (time 1787745999.89 total 10.01 last 78.74) total: 900 open/close in 11.80 seconds: 76.27 ops/second llite.lustre-ffff93f6a0950800.statahead_max=0 1001 real 0m25.168s user 0m0.032s sys 0m2.909s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff93f6a0950800.statahead_max=2048 llite.lustre-ffff93f6a0950800.statahead_max=2048 1001 real 0m7.135s user 0m0.042s sys 0m1.427s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11069 miss_total: 31 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 24 seconds llite.lustre-ffff93f6a0950800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11069 miss_total: 31 mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787746072.336635249 secs.nsecs start_time: 1787745607.404672935 secs.nsecs elapsed_time: 464.931962314 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 0 0 14 4: 1 14 28 8: 2 28 57 16: 1 14 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787746072.336708717 secs.nsecs start_time: 1787745607.404842305 secs.nsecs elapsed_time: 464.931866412 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 2 22 44 16: 1 11 55 32: 0 0 55 64: 1 11 66 128: 1 11 77 256: 2 22 100 - open/close 723 (time 1787746085.04 total 10.00 last 72.30) - open/close 1418 (time 1787746095.05 total 20.01 last 69.43) - open/close 2128 (time 1787746105.05 total 30.01 last 70.99) - open/close 2737 (time 1787746115.06 total 40.01 last 60.89) - open/close 3373 (time 1787746125.07 total 50.02 last 63.53) - open/close 4108 (time 1787746135.07 total 60.03 last 73.49) - open/close 4834 (time 1787746145.07 total 70.03 last 72.60) - open/close 5537 (time 1787746155.07 total 80.03 last 70.29) - open/close 6160 (time 1787746165.08 total 90.04 last 62.23) - open/close 6870 (time 1787746175.09 total 100.04 last 70.99) - open/close 7494 (time 1787746185.09 total 110.05 last 62.37) - open/close 8088 (time 1787746195.10 total 120.05 last 59.36) - open/close 8631 (time 1787746205.11 total 130.06 last 54.25) - open/close 9143 (time 1787746215.11 total 140.07 last 51.18) - open/close 9793 (time 1787746225.11 total 150.07 last 65.00) total: 10000 open/close in 152.92 seconds: 65.39 ops/second llite.lustre-ffff93f6a0950800.statahead_batch_max=0 llite.lustre-ffff93f6a0950800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f6a0950800.stats=clear mdc.lustre-MDT0001-mdc-ffff93f6a0950800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m12.049s user 0m0.334s sys 0m14.554s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff93f6a0950800.statahead_batch_max=256 llite.lustre-ffff93f6a0950800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff93f6a0950800.stats=clear mdc.lustre-MDT0001-mdc-ffff93f6a0950800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.939s user 0m0.303s sys 0m14.403s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787746379.082422980 secs.nsecs start_time: 1787746310.687025182 secs.nsecs elapsed_time: 68.395397798 secs.nsecs subreqs per batch batches %% cum %% 1: 45 33 33 2: 11 8 42 4: 8 6 48 8: 7 5 53 16: 4 3 56 32: 14 10 66 64: 5 3 70 128: 4 3 73 256: 35 26 100 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787746379.082488935 secs.nsecs start_time: 1787746310.687115959 secs.nsecs elapsed_time: 68.395372976 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff93f6a0950800.statahead_batch_max=64 llite.lustre-ffff93f6a0950800.statahead_max=128