== sanity test 123ad: Verify batching statahead works correctly ========================================================== 00:19:25 (1788841165) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9166f437b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9166f437b800.batch_stats=0 llite.lustre-ffff9166f437b800.statahead_max=32 llite.lustre-ffff9166f437b800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9166f437b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9166f437b800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.00 seconds: 50.03 ops/second llite.lustre-ffff9166f437b800.statahead_max=0 101 real 0m1.478s user 0m0.005s sys 0m0.294s 'ls -l' 100 files without statahead: 1 sec llite.lustre-ffff9166f437b800.statahead_max=32 llite.lustre-ffff9166f437b800.statahead_max=32 101 real 0m1.023s user 0m0.004s sys 0m0.157s 'ls -l' 100 files with statahead: 1 sec statahead total: 117 statahead wrong: 1 agl total: 111 list_total: 32 fname_total: 81 hit_total: 3912 miss_total: 256 swrong=1 ewrong=1 total: 900 open/close in 8.79 seconds: 102.36 ops/second llite.lustre-ffff9166f437b800.statahead_max=0 1001 real 0m17.602s user 0m0.034s sys 0m2.800s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff9166f437b800.statahead_max=32 llite.lustre-ffff9166f437b800.statahead_max=32 1001 real 0m8.374s user 0m0.017s sys 0m1.382s 'ls -l' 1000 files with statahead: 8 sec statahead total: 118 statahead wrong: 1 agl total: 112 list_total: 33 fname_total: 81 hit_total: 4905 miss_total: 263 swrong=1 ewrong=1 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff9166f437b800.statahead_stats= statahead total: 118 statahead wrong: 1 agl total: 112 list_total: 33 fname_total: 81 hit_total: 4905 miss_total: 263 mdc.lustre-MDT0000-mdc-ffff9166f437b800.batch_stats= snapshot_time: 1788841245.088720283 secs.nsecs start_time: 1788841169.877119802 secs.nsecs elapsed_time: 75.211600481 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 3 7 12 8: 2 5 17 16: 19 47 65 32: 14 35 100 mdc.lustre-MDT0001-mdc-ffff9166f437b800.batch_stats= snapshot_time: 1788841245.088906912 secs.nsecs start_time: 1788841169.877244794 secs.nsecs elapsed_time: 75.211662118 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 2 2 8: 5 12 15 16: 21 53 69 32: 12 30 100 - open/close 927 (time 1788841257.86 total 10.00 last 92.68) - open/close 1871 (time 1788841267.88 total 20.02 last 94.23) - open/close 2844 (time 1788841277.89 total 30.02 last 97.27) - open/close 3690 (time 1788841287.90 total 40.03 last 84.51) - open/close 4754 (time 1788841297.90 total 50.04 last 106.35) - open/close 5848 (time 1788841307.91 total 60.04 last 109.36) - open/close 6920 (time 1788841317.91 total 70.05 last 107.15) - open/close 7961 (time 1788841327.92 total 80.06 last 104.02) - open/close 8961 (time 1788841337.92 total 90.06 last 99.96) - open/close 9961 (time 1788841347.92 total 100.06 last 99.99) total: 10000 open/close in 100.43 seconds: 99.57 ops/second llite.lustre-ffff9166f437b800.statahead_batch_max=0 llite.lustre-ffff9166f437b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9166f437b800.stats=clear mdc.lustre-MDT0001-mdc-ffff9166f437b800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.949s user 0m0.337s sys 0m15.080s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9166f437b800.statahead_batch_max=32 llite.lustre-ffff9166f437b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9166f437b800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9166f437b800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9166f437b800.stats=clear mdc.lustre-MDT0001-mdc-ffff9166f437b800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m44.115s user 0m0.309s sys 0m14.118s batched statahead RPC= hit count=9981 mdc.lustre-MDT0000-mdc-ffff9166f437b800.batch_stats= snapshot_time: 1788841534.260033153 secs.nsecs start_time: 1788841419.913745609 secs.nsecs elapsed_time: 114.346287544 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 0 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9166f437b800.batch_stats= snapshot_time: 1788841534.260192352 secs.nsecs start_time: 1788841419.913953914 secs.nsecs elapsed_time: 114.346238438 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9166f437b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9166f437b800.batch_stats=0 llite.lustre-ffff9166f437b800.statahead_max=2048 llite.lustre-ffff9166f437b800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9166f437b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9166f437b800.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.57 seconds: 38.96 ops/second llite.lustre-ffff9166f437b800.statahead_max=0 101 real 0m2.310s user 0m0.004s sys 0m0.324s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9166f437b800.statahead_max=2048 llite.lustre-ffff9166f437b800.statahead_max=2048 101 real 0m0.964s user 0m0.005s sys 0m0.162s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10080 miss_total: 20 swrong=0 ewrong=0 - open/close 756 (time 1788841834.03 total 10.01 last 75.50) total: 900 open/close in 12.13 seconds: 74.22 ops/second llite.lustre-ffff9166f437b800.statahead_max=0 1001 real 0m21.129s user 0m0.038s sys 0m2.990s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff9166f437b800.statahead_max=2048 llite.lustre-ffff9166f437b800.statahead_max=2048 1001 real 0m7.136s user 0m0.031s sys 0m1.524s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff9166f437b800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 mdc.lustre-MDT0000-mdc-ffff9166f437b800.batch_stats= snapshot_time: 1788841897.844389323 secs.nsecs start_time: 1788841537.457313347 secs.nsecs elapsed_time: 360.387075976 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 4 44 44 16: 2 22 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff9166f437b800.batch_stats= snapshot_time: 1788841897.844513196 secs.nsecs start_time: 1788841537.457490026 secs.nsecs elapsed_time: 360.387023170 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 22 22 4: 2 22 44 8: 1 11 55 16: 1 11 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 - open/close 743 (time 1788841911.54 total 10.02 last 74.18) - open/close 1468 (time 1788841921.55 total 20.03 last 72.36) - open/close 2341 (time 1788841931.56 total 30.04 last 87.26) - open/close 3182 (time 1788841941.56 total 40.04 last 84.10) - open/close 4007 (time 1788841951.56 total 50.04 last 82.47) - open/close 4672 (time 1788841961.57 total 60.05 last 66.47) - open/close 5441 (time 1788841971.57 total 70.05 last 76.88) - open/close 6181 (time 1788841981.57 total 80.06 last 73.97) - open/close 7027 (time 1788841991.59 total 90.07 last 84.48) - open/close 7836 (time 1788842001.59 total 100.07 last 80.89) - open/close 8610 (time 1788842011.60 total 110.08 last 77.35) - open/close 9506 (time 1788842021.60 total 120.08 last 89.57) total: 10000 open/close in 125.49 seconds: 79.69 ops/second llite.lustre-ffff9166f437b800.statahead_batch_max=0 llite.lustre-ffff9166f437b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9166f437b800.stats=clear mdc.lustre-MDT0001-mdc-ffff9166f437b800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m0.968s user 0m0.334s sys 0m14.827s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9166f437b800.statahead_batch_max=256 llite.lustre-ffff9166f437b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9166f437b800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9166f437b800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9166f437b800.stats=clear mdc.lustre-MDT0001-mdc-ffff9166f437b800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m54.192s user 0m0.356s sys 0m14.634s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9166f437b800.batch_stats= snapshot_time: 1788842160.705465259 secs.nsecs start_time: 1788842096.570099290 secs.nsecs elapsed_time: 64.135365969 secs.nsecs subreqs per batch batches %% cum %% 1: 83 35 35 2: 49 20 55 4: 24 10 66 8: 13 5 71 16: 8 3 75 32: 13 5 80 64: 10 4 84 128: 2 0 85 256: 34 14 100 mdc.lustre-MDT0001-mdc-ffff9166f437b800.batch_stats= snapshot_time: 1788842160.705533682 secs.nsecs start_time: 1788842096.570248550 secs.nsecs elapsed_time: 64.135285132 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9166f437b800.statahead_batch_max=64 llite.lustre-ffff9166f437b800.statahead_max=128