== sanity test 123ad: Verify batching statahead works correctly ========================================================== 06:25:56 (1789122356) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff949ec801b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff949ec801b800.batch_stats=0 llite.lustre-ffff949ec801b800.statahead_max=32 llite.lustre-ffff949ec801b800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff949ec801b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff949ec801b800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.34 seconds: 42.82 ops/second llite.lustre-ffff949ec801b800.statahead_max=0 101 real 0m2.243s user 0m0.003s sys 0m0.287s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff949ec801b800.statahead_max=32 llite.lustre-ffff949ec801b800.statahead_max=32 101 real 0m0.886s user 0m0.006s sys 0m0.179s 'ls -l' 100 files with statahead: 1 sec statahead total: 97 statahead wrong: 2 agl total: 92 list_total: 30 fname_total: 58 hit_total: 4237 miss_total: 768 swrong=2 ewrong=2 - open/close 878 (time 1789122385.56 total 10.01 last 87.70) total: 900 open/close in 10.37 seconds: 86.82 ops/second llite.lustre-ffff949ec801b800.statahead_max=0 1001 real 0m22.395s user 0m0.027s sys 0m2.813s 'ls -l' 1000 files without statahead: 22 sec llite.lustre-ffff949ec801b800.statahead_max=32 llite.lustre-ffff949ec801b800.statahead_max=32 1001 real 0m12.089s user 0m0.031s sys 0m1.404s 'ls -l' 1000 files with statahead: 12 sec statahead total: 98 statahead wrong: 2 agl total: 93 list_total: 31 fname_total: 58 hit_total: 5230 miss_total: 775 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff949ec801b800.statahead_stats= statahead total: 98 statahead wrong: 2 agl total: 93 list_total: 31 fname_total: 58 hit_total: 5230 miss_total: 775 mdc.lustre-MDT0000-mdc-ffff949ec801b800.batch_stats= snapshot_time: 1789122454.666529558 secs.nsecs start_time: 1789122361.848077409 secs.nsecs elapsed_time: 92.818452149 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 2 5 10 4: 2 5 15 8: 2 5 21 16: 24 63 84 32: 6 15 100 mdc.lustre-MDT0001-mdc-ffff949ec801b800.batch_stats= snapshot_time: 1789122454.666589822 secs.nsecs start_time: 1789122361.851566711 secs.nsecs elapsed_time: 92.815023111 secs.nsecs subreqs per batch batches %% cum %% 1: 2 4 4 2: 0 0 4 4: 1 2 7 8: 3 7 14 16: 14 34 48 32: 21 51 100 - open/close 906 (time 1789122466.99 total 10.00 last 90.56) - open/close 1836 (time 1789122477.00 total 20.01 last 92.92) - open/close 2864 (time 1789122487.01 total 30.02 last 102.73) - open/close 3857 (time 1789122497.03 total 40.04 last 99.08) - open/close 4738 (time 1789122507.04 total 50.05 last 88.00) - open/close 5682 (time 1789122517.04 total 60.05 last 94.39) - open/close 6589 (time 1789122527.05 total 70.06 last 90.63) - open/close 7499 (time 1789122537.05 total 80.07 last 90.96) - open/close 8393 (time 1789122547.06 total 90.07 last 89.31) - open/close 9268 (time 1789122557.06 total 100.08 last 87.50) total: 10000 open/close in 108.17 seconds: 92.44 ops/second llite.lustre-ffff949ec801b800.statahead_batch_max=0 llite.lustre-ffff949ec801b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff949ec801b800.stats=clear mdc.lustre-MDT0001-mdc-ffff949ec801b800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m4.535s user 0m0.322s sys 0m14.486s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff949ec801b800.statahead_batch_max=32 llite.lustre-ffff949ec801b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff949ec801b800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff949ec801b800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff949ec801b800.stats=clear mdc.lustre-MDT0001-mdc-ffff949ec801b800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m39.879s user 0m0.274s sys 0m13.595s batched statahead RPC= hit count=9982 mdc.lustre-MDT0000-mdc-ffff949ec801b800.batch_stats= snapshot_time: 1789122748.369963462 secs.nsecs start_time: 1789122637.780688268 secs.nsecs elapsed_time: 110.589275194 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 0 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff949ec801b800.batch_stats= snapshot_time: 1789122748.370033263 secs.nsecs start_time: 1789122637.780807158 secs.nsecs elapsed_time: 110.589226105 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff949ec801b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff949ec801b800.batch_stats=0 llite.lustre-ffff949ec801b800.statahead_max=2048 llite.lustre-ffff949ec801b800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff949ec801b800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff949ec801b800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.03 seconds: 49.32 ops/second llite.lustre-ffff949ec801b800.statahead_max=0 101 real 0m1.903s user 0m0.008s sys 0m0.292s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff949ec801b800.statahead_max=2048 llite.lustre-ffff949ec801b800.statahead_max=2048 101 real 0m0.791s user 0m0.003s sys 0m0.159s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10081 miss_total: 19 swrong=0 ewrong=0 total: 900 open/close in 9.10 seconds: 98.91 ops/second llite.lustre-ffff949ec801b800.statahead_max=0 1001 real 0m23.444s user 0m0.040s sys 0m2.938s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff949ec801b800.statahead_max=2048 llite.lustre-ffff949ec801b800.statahead_max=2048 1001 real 0m6.183s user 0m0.026s sys 0m1.495s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff949ec801b800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 mdc.lustre-MDT0000-mdc-ffff949ec801b800.batch_stats= snapshot_time: 1789123080.855264450 secs.nsecs start_time: 1789122750.546733930 secs.nsecs elapsed_time: 330.308530520 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 14 14 4: 3 42 57 8: 1 14 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff949ec801b800.batch_stats= snapshot_time: 1789123080.855323902 secs.nsecs start_time: 1789122750.546870936 secs.nsecs elapsed_time: 330.308452966 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 3 30 30 16: 2 20 50 32: 2 20 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 870 (time 1789123093.60 total 10.00 last 86.99) - open/close 1635 (time 1789123103.60 total 20.00 last 76.48) - open/close 2527 (time 1789123113.61 total 30.01 last 89.17) - open/close 3432 (time 1789123123.62 total 40.02 last 90.41) - open/close 4386 (time 1789123133.63 total 50.03 last 95.34) - open/close 5304 (time 1789123143.63 total 60.03 last 91.72) - open/close 6136 (time 1789123153.64 total 70.04 last 83.15) - open/close 7093 (time 1789123163.64 total 80.04 last 95.66) - open/close 7918 (time 1789123173.66 total 90.06 last 82.39) - open/close 8704 (time 1789123183.67 total 100.07 last 78.51) - open/close 9543 (time 1789123193.67 total 110.07 last 83.88) total: 10000 open/close in 116.18 seconds: 86.07 ops/second llite.lustre-ffff949ec801b800.statahead_batch_max=0 llite.lustre-ffff949ec801b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff949ec801b800.stats=clear mdc.lustre-MDT0001-mdc-ffff949ec801b800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.133s user 0m0.288s sys 0m14.412s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff949ec801b800.statahead_batch_max=256 llite.lustre-ffff949ec801b800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff949ec801b800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff949ec801b800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff949ec801b800.stats=clear mdc.lustre-MDT0001-mdc-ffff949ec801b800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m47.522s user 0m0.280s sys 0m14.299s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff949ec801b800.batch_stats= snapshot_time: 1789123328.334804115 secs.nsecs start_time: 1789123271.694142939 secs.nsecs elapsed_time: 56.640661176 secs.nsecs subreqs per batch batches %% cum %% 1: 84 43 43 2: 18 9 52 4: 15 7 60 8: 10 5 65 16: 9 4 70 32: 12 6 76 64: 11 5 81 128: 0 0 81 256: 35 18 100 mdc.lustre-MDT0001-mdc-ffff949ec801b800.batch_stats= snapshot_time: 1789123328.335019327 secs.nsecs start_time: 1789123271.694227264 secs.nsecs elapsed_time: 56.640792063 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff949ec801b800.statahead_batch_max=64 llite.lustre-ffff949ec801b800.statahead_max=128