== sanity test 123ad: Verify batching statahead works correctly ========================================================== 06:44:44 (1789987484) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9ecda5cb5800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ecda5cb5800.batch_stats=0 llite.lustre-ffff9ecda5cb5800.statahead_max=32 llite.lustre-ffff9ecda5cb5800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9ecda5cb5800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ecda5cb5800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.93 seconds: 34.13 ops/second llite.lustre-ffff9ecda5cb5800.statahead_max=0 101 real 0m3.324s user 0m0.009s sys 0m0.333s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9ecda5cb5800.statahead_max=32 llite.lustre-ffff9ecda5cb5800.statahead_max=32 101 real 0m1.109s user 0m0.004s sys 0m0.175s 'ls -l' 100 files with statahead: 1 sec statahead total: 77 statahead wrong: 1 agl total: 72 list_total: 26 fname_total: 51 hit_total: 3766 miss_total: 212 swrong=1 ewrong=1 - open/close 638 (time 1789987514.52 total 10.01 last 63.74) total: 900 open/close in 14.30 seconds: 62.92 ops/second llite.lustre-ffff9ecda5cb5800.statahead_max=0 1001 real 0m28.149s user 0m0.039s sys 0m3.106s 'ls -l' 1000 files without statahead: 29 sec llite.lustre-ffff9ecda5cb5800.statahead_max=32 llite.lustre-ffff9ecda5cb5800.statahead_max=32 1001 real 0m15.641s user 0m0.028s sys 0m1.636s 'ls -l' 1000 files with statahead: 16 sec statahead total: 78 statahead wrong: 1 agl total: 73 list_total: 27 fname_total: 51 hit_total: 4764 miss_total: 214 swrong=1 ewrong=1 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 24 seconds llite.lustre-ffff9ecda5cb5800.statahead_stats= statahead total: 78 statahead wrong: 1 agl total: 73 list_total: 27 fname_total: 51 hit_total: 4764 miss_total: 214 mdc.lustre-MDT0000-mdc-ffff9ecda5cb5800.batch_stats= snapshot_time: 1789987603.268009355 secs.nsecs start_time: 1789987489.567527052 secs.nsecs elapsed_time: 113.700482303 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 5 5 8: 4 10 15 16: 21 53 69 32: 12 30 100 mdc.lustre-MDT0001-mdc-ffff9ecda5cb5800.batch_stats= snapshot_time: 1789987603.268088243 secs.nsecs start_time: 1789987489.567652365 secs.nsecs elapsed_time: 113.700435878 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 3 7 10 8: 2 5 15 16: 18 46 61 32: 15 38 100 - open/close 654 (time 1789987616.27 total 10.02 last 65.29) - open/close 1339 (time 1789987626.27 total 20.02 last 68.48) - open/close 1977 (time 1789987636.28 total 30.02 last 63.77) - open/close 2573 (time 1789987646.29 total 40.03 last 59.54) - open/close 3151 (time 1789987656.30 total 50.05 last 57.73) - open/close 3754 (time 1789987666.31 total 60.06 last 60.21) - open/close 4386 (time 1789987676.32 total 70.07 last 63.14) - open/close 5082 (time 1789987686.33 total 80.07 last 69.58) - open/close 5739 (time 1789987696.33 total 90.08 last 65.67) - open/close 6392 (time 1789987706.36 total 100.11 last 65.09) - open/close 6933 (time 1789987716.36 total 110.11 last 54.09) - open/close 7480 (time 1789987726.37 total 120.11 last 54.69) - open/close 7965 (time 1789987736.37 total 130.12 last 48.46) - open/close 8538 (time 1789987746.40 total 140.14 last 57.17) - open/close 9215 (time 1789987756.41 total 150.15 last 67.63) - open/close 9813 (time 1789987766.41 total 160.16 last 59.76) total: 10000 open/close in 162.60 seconds: 61.50 ops/second llite.lustre-ffff9ecda5cb5800.statahead_batch_max=0 llite.lustre-ffff9ecda5cb5800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ecda5cb5800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ecda5cb5800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.991s user 0m0.372s sys 0m15.662s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9ecda5cb5800.statahead_batch_max=32 llite.lustre-ffff9ecda5cb5800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ecda5cb5800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9ecda5cb5800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9ecda5cb5800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ecda5cb5800.stats=clear ls -l files in d123ad.sanity: 10001 real 2m7.669s user 0m0.407s sys 0m16.002s batched statahead RPC= hit count=9987 mdc.lustre-MDT0000-mdc-ffff9ecda5cb5800.batch_stats= snapshot_time: 1789987988.367140464 secs.nsecs start_time: 1789987848.103916646 secs.nsecs elapsed_time: 140.263223818 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9ecda5cb5800.batch_stats= snapshot_time: 1789987988.367230143 secs.nsecs start_time: 1789987848.104112579 secs.nsecs elapsed_time: 140.263117564 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9ecda5cb5800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ecda5cb5800.batch_stats=0 llite.lustre-ffff9ecda5cb5800.statahead_max=2048 llite.lustre-ffff9ecda5cb5800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9ecda5cb5800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9ecda5cb5800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.06 seconds: 32.73 ops/second llite.lustre-ffff9ecda5cb5800.statahead_max=0 101 real 0m2.909s user 0m0.007s sys 0m0.430s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9ecda5cb5800.statahead_max=2048 llite.lustre-ffff9ecda5cb5800.statahead_max=2048 101 real 0m1.200s user 0m0.005s sys 0m0.219s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10086 miss_total: 14 swrong=0 ewrong=0 - open/close 725 (time 1789988363.98 total 10.01 last 72.40) total: 900 open/close in 13.08 seconds: 68.79 ops/second llite.lustre-ffff9ecda5cb5800.statahead_max=0 1001 real 0m28.822s user 0m0.051s sys 0m3.262s 'ls -l' 1000 files without statahead: 29 sec llite.lustre-ffff9ecda5cb5800.statahead_max=2048 llite.lustre-ffff9ecda5cb5800.statahead_max=2048 1001 real 0m7.066s user 0m0.036s sys 0m1.598s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 23 seconds llite.lustre-ffff9ecda5cb5800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 mdc.lustre-MDT0000-mdc-ffff9ecda5cb5800.batch_stats= snapshot_time: 1789988438.671268744 secs.nsecs start_time: 1789987991.283290456 secs.nsecs elapsed_time: 447.387978288 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 14 14 4: 2 28 42 8: 1 14 57 16: 1 14 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff9ecda5cb5800.batch_stats= snapshot_time: 1789988438.671614665 secs.nsecs start_time: 1789987991.292649554 secs.nsecs elapsed_time: 447.378965111 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 10 10 8: 4 40 50 16: 0 0 50 32: 2 20 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 678 (time 1789988451.30 total 10.00 last 67.77) - open/close 1326 (time 1789988461.31 total 20.02 last 64.71) - open/close 1992 (time 1789988471.32 total 30.02 last 66.57) - open/close 2668 (time 1789988481.33 total 40.03 last 67.55) - open/close 3348 (time 1789988491.34 total 50.05 last 67.89) - open/close 3948 (time 1789988501.35 total 60.05 last 59.97) - open/close 4579 (time 1789988511.35 total 70.06 last 63.08) - open/close 5228 (time 1789988521.36 total 80.06 last 64.84) - open/close 5898 (time 1789988531.36 total 90.07 last 66.98) - open/close 6512 (time 1789988541.37 total 100.07 last 61.37) - open/close 7152 (time 1789988551.38 total 110.08 last 63.92) - open/close 7849 (time 1789988561.39 total 120.10 last 69.61) - open/close 8480 (time 1789988571.40 total 130.11 last 63.03) - open/close 9074 (time 1789988581.42 total 140.13 last 59.30) - open/close 9671 (time 1789988591.42 total 150.13 last 59.69) total: 10000 open/close in 154.76 seconds: 64.62 ops/second llite.lustre-ffff9ecda5cb5800.statahead_batch_max=0 llite.lustre-ffff9ecda5cb5800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ecda5cb5800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ecda5cb5800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m10.192s user 0m0.414s sys 0m18.374s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9ecda5cb5800.statahead_batch_max=256 llite.lustre-ffff9ecda5cb5800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9ecda5cb5800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9ecda5cb5800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9ecda5cb5800.stats=clear mdc.lustre-MDT0001-mdc-ffff9ecda5cb5800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m51.257s user 0m0.353s sys 0m17.456s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9ecda5cb5800.batch_stats= snapshot_time: 1789988737.556135569 secs.nsecs start_time: 1789988674.992293647 secs.nsecs elapsed_time: 62.563841922 secs.nsecs subreqs per batch batches %% cum %% 1: 159 51 51 2: 45 14 66 4: 30 9 75 8: 12 3 79 16: 7 2 81 32: 12 3 85 64: 8 2 88 128: 2 0 88 256: 34 11 100 mdc.lustre-MDT0001-mdc-ffff9ecda5cb5800.batch_stats= snapshot_time: 1789988737.556214386 secs.nsecs start_time: 1789988674.992769470 secs.nsecs elapsed_time: 62.563444916 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9ecda5cb5800.statahead_batch_max=64 llite.lustre-ffff9ecda5cb5800.statahead_max=128