== sanity test 123ad: Verify batching statahead works correctly ========================================================== 06:06:47 (1787911607) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9dfa85267800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9dfa85267800.batch_stats=0 llite.lustre-ffff9dfa85267800.statahead_max=32 llite.lustre-ffff9dfa85267800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9dfa85267800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9dfa85267800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.45 seconds: 28.96 ops/second llite.lustre-ffff9dfa85267800.statahead_max=0 101 real 0m2.934s user 0m0.007s sys 0m0.307s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9dfa85267800.statahead_max=32 llite.lustre-ffff9dfa85267800.statahead_max=32 101 real 0m1.038s user 0m0.005s sys 0m0.168s 'ls -l' 100 files with statahead: 1 sec statahead total: 124 statahead wrong: 2 agl total: 119 list_total: 30 fname_total: 94 hit_total: 4101 miss_total: 1573 swrong=2 ewrong=2 - open/close 749 (time 1787911636.90 total 10.00 last 74.87) total: 900 open/close in 12.22 seconds: 73.68 ops/second llite.lustre-ffff9dfa85267800.statahead_max=0 1001 real 0m23.572s user 0m0.043s sys 0m2.879s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff9dfa85267800.statahead_max=32 llite.lustre-ffff9dfa85267800.statahead_max=32 1001 real 0m9.648s user 0m0.031s sys 0m1.531s 'ls -l' 1000 files with statahead: 10 sec statahead total: 125 statahead wrong: 2 agl total: 120 list_total: 31 fname_total: 94 hit_total: 5100 miss_total: 1574 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 19 seconds llite.lustre-ffff9dfa85267800.statahead_stats= statahead total: 125 statahead wrong: 2 agl total: 120 list_total: 31 fname_total: 94 hit_total: 5100 miss_total: 1574 mdc.lustre-MDT0000-mdc-ffff9dfa85267800.batch_stats= snapshot_time: 1787911705.402905242 secs.nsecs start_time: 1787911612.689490980 secs.nsecs elapsed_time: 92.713414262 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 1 2 5 8: 6 15 20 16: 14 35 56 32: 17 43 100 mdc.lustre-MDT0001-mdc-ffff9dfa85267800.batch_stats= snapshot_time: 1787911705.402992449 secs.nsecs start_time: 1787911612.689683125 secs.nsecs elapsed_time: 92.713309324 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 5 12 12 8: 0 0 12 16: 23 58 71 32: 11 28 100 - open/close 755 (time 1787911718.13 total 10.01 last 75.40) - open/close 1486 (time 1787911728.13 total 20.02 last 73.06) - open/close 2193 (time 1787911738.16 total 30.04 last 70.53) - open/close 2943 (time 1787911748.16 total 40.04 last 74.98) - open/close 3590 (time 1787911758.16 total 50.05 last 64.68) - open/close 4327 (time 1787911768.16 total 60.05 last 73.69) - open/close 5102 (time 1787911778.17 total 70.06 last 77.42) - open/close 5751 (time 1787911788.18 total 80.06 last 64.87) - open/close 6497 (time 1787911798.18 total 90.07 last 74.59) - open/close 7223 (time 1787911808.18 total 100.07 last 72.59) - open/close 8018 (time 1787911818.18 total 110.07 last 79.50) - open/close 8756 (time 1787911828.19 total 120.08 last 73.73) - open/close 9502 (time 1787911838.20 total 130.08 last 74.57) total: 10000 open/close in 136.38 seconds: 73.32 ops/second llite.lustre-ffff9dfa85267800.statahead_batch_max=0 llite.lustre-ffff9dfa85267800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9dfa85267800.stats=clear mdc.lustre-MDT0001-mdc-ffff9dfa85267800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.767s user 0m0.344s sys 0m16.332s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9dfa85267800.statahead_batch_max=32 llite.lustre-ffff9dfa85267800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9dfa85267800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9dfa85267800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9dfa85267800.stats=clear mdc.lustre-MDT0001-mdc-ffff9dfa85267800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m40.539s user 0m0.331s sys 0m15.155s batched statahead RPC= hit count=9991 mdc.lustre-MDT0000-mdc-ffff9dfa85267800.batch_stats= snapshot_time: 1787912027.924853973 secs.nsecs start_time: 1787911916.918799567 secs.nsecs elapsed_time: 111.006054406 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9dfa85267800.batch_stats= snapshot_time: 1787912027.924917235 secs.nsecs start_time: 1787911916.918891247 secs.nsecs elapsed_time: 111.006025988 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9dfa85267800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9dfa85267800.batch_stats=0 llite.lustre-ffff9dfa85267800.statahead_max=2048 llite.lustre-ffff9dfa85267800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9dfa85267800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9dfa85267800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.80 seconds: 35.65 ops/second llite.lustre-ffff9dfa85267800.statahead_max=0 101 real 0m2.327s user 0m0.004s sys 0m0.316s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9dfa85267800.statahead_max=2048 llite.lustre-ffff9dfa85267800.statahead_max=2048 101 real 0m0.702s user 0m0.002s sys 0m0.178s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10090 miss_total: 10 swrong=0 ewrong=0 - open/close 773 (time 1787912335.25 total 10.00 last 77.28) total: 900 open/close in 11.63 seconds: 77.36 ops/second llite.lustre-ffff9dfa85267800.statahead_max=0 1001 real 0m20.806s user 0m0.036s sys 0m3.025s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff9dfa85267800.statahead_max=2048 llite.lustre-ffff9dfa85267800.statahead_max=2048 1001 real 0m7.343s user 0m0.031s sys 0m1.448s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11089 miss_total: 11 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff9dfa85267800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11089 miss_total: 11 mdc.lustre-MDT0000-mdc-ffff9dfa85267800.batch_stats= snapshot_time: 1787912396.833013474 secs.nsecs start_time: 1787912030.776828491 secs.nsecs elapsed_time: 366.056184983 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 12 12 8: 4 50 62 16: 0 0 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff9dfa85267800.batch_stats= snapshot_time: 1787912396.833091821 secs.nsecs start_time: 1787912030.777518416 secs.nsecs elapsed_time: 366.055573405 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 4 50 50 8: 0 0 50 16: 1 12 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 - open/close 692 (time 1787912409.64 total 10.01 last 69.16) - open/close 1489 (time 1787912419.64 total 20.01 last 79.70) - open/close 2282 (time 1787912429.65 total 30.01 last 79.25) - open/close 3122 (time 1787912439.69 total 40.06 last 83.64) - open/close 3967 (time 1787912449.70 total 50.07 last 84.41) - open/close 4746 (time 1787912459.71 total 60.07 last 77.87) - open/close 5507 (time 1787912469.72 total 70.08 last 76.02) - open/close 6244 (time 1787912479.74 total 80.10 last 73.57) - open/close 6932 (time 1787912489.74 total 90.10 last 68.78) - open/close 7730 (time 1787912499.74 total 100.10 last 79.77) - open/close 8543 (time 1787912509.76 total 110.12 last 81.18) - open/close 9385 (time 1787912519.77 total 120.14 last 84.05) total: 10000 open/close in 129.52 seconds: 77.21 ops/second llite.lustre-ffff9dfa85267800.statahead_batch_max=0 llite.lustre-ffff9dfa85267800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9dfa85267800.stats=clear mdc.lustre-MDT0001-mdc-ffff9dfa85267800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.143s user 0m0.353s sys 0m16.446s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9dfa85267800.statahead_batch_max=256 llite.lustre-ffff9dfa85267800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9dfa85267800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9dfa85267800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9dfa85267800.stats=clear mdc.lustre-MDT0001-mdc-ffff9dfa85267800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m50.023s user 0m0.300s sys 0m16.490s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9dfa85267800.batch_stats= snapshot_time: 1787912667.285558521 secs.nsecs start_time: 1787912605.100885276 secs.nsecs elapsed_time: 62.184673245 secs.nsecs subreqs per batch batches %% cum %% 1: 135 54 54 2: 22 8 63 4: 15 6 69 8: 12 4 74 16: 8 3 77 32: 14 5 83 64: 6 2 85 128: 0 0 85 256: 35 14 100 mdc.lustre-MDT0001-mdc-ffff9dfa85267800.batch_stats= snapshot_time: 1787912667.285634005 secs.nsecs start_time: 1787912605.101047245 secs.nsecs elapsed_time: 62.184586760 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9dfa85267800.statahead_batch_max=64 llite.lustre-ffff9dfa85267800.statahead_max=128