== sanity test 123ad: Verify batching statahead works correctly ========================================================== 14:47:17 (1789584437) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9d72514bc800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d72514bc800.batch_stats=0 llite.lustre-ffff9d72514bc800.statahead_max=32 llite.lustre-ffff9d72514bc800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9d72514bc800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d72514bc800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.18 seconds: 45.97 ops/second llite.lustre-ffff9d72514bc800.statahead_max=0 101 real 0m2.723s user 0m0.005s sys 0m0.316s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9d72514bc800.statahead_max=32 llite.lustre-ffff9d72514bc800.statahead_max=32 101 real 0m1.097s user 0m0.003s sys 0m0.191s 'ls -l' 100 files with statahead: 1 sec statahead total: 126 statahead wrong: 2 agl total: 120 list_total: 31 fname_total: 84 hit_total: 3948 miss_total: 1230 swrong=2 ewrong=2 total: 900 open/close in 9.97 seconds: 90.28 ops/second llite.lustre-ffff9d72514bc800.statahead_max=0 1001 real 0m19.790s user 0m0.033s sys 0m2.877s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff9d72514bc800.statahead_max=32 llite.lustre-ffff9d72514bc800.statahead_max=32 1001 real 0m8.073s user 0m0.026s sys 0m1.360s 'ls -l' 1000 files with statahead: 8 sec statahead total: 127 statahead wrong: 2 agl total: 121 list_total: 32 fname_total: 84 hit_total: 4941 miss_total: 1237 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff9d72514bc800.statahead_stats= statahead total: 127 statahead wrong: 2 agl total: 121 list_total: 32 fname_total: 84 hit_total: 4941 miss_total: 1237 mdc.lustre-MDT0000-mdc-ffff9d72514bc800.batch_stats= snapshot_time: 1789584525.923507950 secs.nsecs start_time: 1789584442.670441393 secs.nsecs elapsed_time: 83.253066557 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 7 7 8: 3 7 15 16: 20 51 66 32: 13 33 100 mdc.lustre-MDT0001-mdc-ffff9d72514bc800.batch_stats= snapshot_time: 1789584525.929118389 secs.nsecs start_time: 1789584442.670555950 secs.nsecs elapsed_time: 83.258562439 secs.nsecs subreqs per batch batches %% cum %% 1: 3 7 7 2: 0 0 7 4: 2 4 12 8: 2 4 17 16: 21 51 68 32: 13 31 100 - open/close 989 (time 1789584539.04 total 10.00 last 98.87) - open/close 1868 (time 1789584549.04 total 20.01 last 87.86) - open/close 2843 (time 1789584559.05 total 30.01 last 97.45) - open/close 3828 (time 1789584569.05 total 40.01 last 98.47) - open/close 4878 (time 1789584579.05 total 50.02 last 104.95) - open/close 5920 (time 1789584589.06 total 60.02 last 104.17) - open/close 6997 (time 1789584599.07 total 70.03 last 107.59) - open/close 7990 (time 1789584609.07 total 80.04 last 99.24) - open/close 8779 (time 1789584619.09 total 90.06 last 78.74) - open/close 9753 (time 1789584629.10 total 100.07 last 97.29) total: 10000 open/close in 102.24 seconds: 97.81 ops/second llite.lustre-ffff9d72514bc800.statahead_batch_max=0 llite.lustre-ffff9d72514bc800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d72514bc800.stats=clear mdc.lustre-MDT0001-mdc-ffff9d72514bc800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.310s user 0m0.333s sys 0m14.840s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9d72514bc800.statahead_batch_max=32 llite.lustre-ffff9d72514bc800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d72514bc800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9d72514bc800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9d72514bc800.stats=clear mdc.lustre-MDT0001-mdc-ffff9d72514bc800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m41.739s user 0m0.291s sys 0m13.081s batched statahead RPC= hit count=9984 mdc.lustre-MDT0000-mdc-ffff9d72514bc800.batch_stats= snapshot_time: 1789584816.797462555 secs.nsecs start_time: 1789584707.350632539 secs.nsecs elapsed_time: 109.446830016 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9d72514bc800.batch_stats= snapshot_time: 1789584816.797533099 secs.nsecs start_time: 1789584707.350984569 secs.nsecs elapsed_time: 109.446548530 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9d72514bc800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d72514bc800.batch_stats=0 llite.lustre-ffff9d72514bc800.statahead_max=2048 llite.lustre-ffff9d72514bc800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9d72514bc800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d72514bc800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.06 seconds: 48.47 ops/second llite.lustre-ffff9d72514bc800.statahead_max=0 101 real 0m1.484s user 0m0.007s sys 0m0.271s 'ls -l' 100 files without statahead: 1 sec llite.lustre-ffff9d72514bc800.statahead_max=2048 llite.lustre-ffff9d72514bc800.statahead_max=2048 101 real 0m0.928s user 0m0.006s sys 0m0.159s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10083 miss_total: 17 swrong=0 ewrong=0 - open/close 645 (time 1789585102.03 total 10.01 last 64.44) total: 900 open/close in 13.26 seconds: 67.88 ops/second llite.lustre-ffff9d72514bc800.statahead_max=0 1001 real 0m20.515s user 0m0.037s sys 0m2.735s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff9d72514bc800.statahead_max=2048 llite.lustre-ffff9d72514bc800.statahead_max=2048 1001 real 0m7.555s user 0m0.034s sys 0m1.295s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 16 seconds llite.lustre-ffff9d72514bc800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11082 miss_total: 18 mdc.lustre-MDT0000-mdc-ffff9d72514bc800.batch_stats= snapshot_time: 1789585162.465645849 secs.nsecs start_time: 1789584819.816494247 secs.nsecs elapsed_time: 342.649151602 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 40 40 8: 1 20 60 16: 0 0 60 32: 0 0 60 64: 0 0 60 128: 0 0 60 256: 2 40 100 mdc.lustre-MDT0001-mdc-ffff9d72514bc800.batch_stats= snapshot_time: 1789585162.465746023 secs.nsecs start_time: 1789584819.816635398 secs.nsecs elapsed_time: 342.649110625 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 2 22 33 16: 1 11 44 32: 1 11 55 64: 2 22 77 128: 0 0 77 256: 2 22 100 - open/close 732 (time 1789585174.97 total 10.00 last 73.18) - open/close 1677 (time 1789585184.98 total 20.01 last 94.40) - open/close 2785 (time 1789585194.98 total 30.02 last 110.78) - open/close 3741 (time 1789585204.99 total 40.02 last 95.52) - open/close 4761 (time 1789585214.99 total 50.02 last 101.99) - open/close 5578 (time 1789585224.99 total 60.03 last 81.69) - open/close 6378 (time 1789585235.00 total 70.03 last 79.93) - open/close 7360 (time 1789585245.00 total 80.04 last 98.18) - open/close 8268 (time 1789585255.00 total 90.04 last 90.78) - open/close 9047 (time 1789585265.02 total 100.06 last 77.73) - open/close 9917 (time 1789585275.03 total 110.07 last 86.93) total: 10000 open/close in 110.92 seconds: 90.15 ops/second llite.lustre-ffff9d72514bc800.statahead_batch_max=0 llite.lustre-ffff9d72514bc800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d72514bc800.stats=clear mdc.lustre-MDT0001-mdc-ffff9d72514bc800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m59.925s user 0m0.312s sys 0m14.003s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9d72514bc800.statahead_batch_max=256 llite.lustre-ffff9d72514bc800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d72514bc800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9d72514bc800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9d72514bc800.stats=clear mdc.lustre-MDT0001-mdc-ffff9d72514bc800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.940s user 0m0.285s sys 0m13.662s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9d72514bc800.batch_stats= snapshot_time: 1789585414.752561840 secs.nsecs start_time: 1789585344.012518493 secs.nsecs elapsed_time: 70.740043347 secs.nsecs subreqs per batch batches %% cum %% 1: 167 50 50 2: 43 12 63 4: 31 9 72 8: 20 6 78 16: 11 3 81 32: 19 5 87 64: 6 1 89 128: 2 0 90 256: 33 9 100 mdc.lustre-MDT0001-mdc-ffff9d72514bc800.batch_stats= snapshot_time: 1789585414.752723398 secs.nsecs start_time: 1789585344.012612841 secs.nsecs elapsed_time: 70.740110557 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9d72514bc800.statahead_batch_max=64 llite.lustre-ffff9d72514bc800.statahead_max=128