== sanity test 123ad: Verify batching statahead works correctly ========================================================== 09:24:58 (1788614698) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9f0ee7346000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f0ee7346000.batch_stats=0 llite.lustre-ffff9f0ee7346000.statahead_max=32 llite.lustre-ffff9f0ee7346000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9f0ee7346000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f0ee7346000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.71 seconds: 36.92 ops/second llite.lustre-ffff9f0ee7346000.statahead_max=0 101 real 0m2.395s user 0m0.007s sys 0m0.320s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9f0ee7346000.statahead_max=32 llite.lustre-ffff9f0ee7346000.statahead_max=32 101 real 0m1.104s user 0m0.003s sys 0m0.167s 'ls -l' 100 files with statahead: 1 sec statahead total: 132 statahead wrong: 2 agl total: 127 list_total: 31 fname_total: 90 hit_total: 4106 miss_total: 1293 swrong=2 ewrong=2 - open/close 728 (time 1788614727.13 total 10.00 last 72.79) total: 900 open/close in 12.29 seconds: 73.23 ops/second llite.lustre-ffff9f0ee7346000.statahead_max=0 1001 real 0m23.280s user 0m0.032s sys 0m3.077s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff9f0ee7346000.statahead_max=32 llite.lustre-ffff9f0ee7346000.statahead_max=32 1001 real 0m10.381s user 0m0.022s sys 0m1.443s 'ls -l' 1000 files with statahead: 11 sec statahead total: 133 statahead wrong: 2 agl total: 128 list_total: 32 fname_total: 90 hit_total: 5105 miss_total: 1294 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff9f0ee7346000.statahead_stats= statahead total: 133 statahead wrong: 2 agl total: 128 list_total: 32 fname_total: 90 hit_total: 5105 miss_total: 1294 mdc.lustre-MDT0000-mdc-ffff9f0ee7346000.batch_stats= snapshot_time: 1788614799.015665641 secs.nsecs start_time: 1788614703.498681589 secs.nsecs elapsed_time: 95.516984052 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 3 7 10 8: 4 10 20 16: 21 52 72 32: 11 27 100 mdc.lustre-MDT0001-mdc-ffff9f0ee7346000.batch_stats= snapshot_time: 1788614799.015743941 secs.nsecs start_time: 1788614703.498796987 secs.nsecs elapsed_time: 95.516946954 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 2 5 10 8: 1 2 12 16: 19 48 61 32: 15 38 100 - open/close 660 (time 1788614812.41 total 10.00 last 65.98) - open/close 1344 (time 1788614822.41 total 20.01 last 68.38) - open/close 1965 (time 1788614832.42 total 30.01 last 62.06) - open/close 2666 (time 1788614842.43 total 40.02 last 70.03) - open/close 3359 (time 1788614852.43 total 50.02 last 69.29) - open/close 4060 (time 1788614862.43 total 60.03 last 70.09) - open/close 4821 (time 1788614872.44 total 70.03 last 76.08) - open/close 5679 (time 1788614882.44 total 80.04 last 85.74) - open/close 6511 (time 1788614892.46 total 90.06 last 83.03) - open/close 7277 (time 1788614902.47 total 100.06 last 76.54) - open/close 8038 (time 1788614912.47 total 110.07 last 76.09) - open/close 8880 (time 1788614922.48 total 120.07 last 84.17) - open/close 9560 (time 1788614932.48 total 130.07 last 67.99) total: 10000 open/close in 137.21 seconds: 72.88 ops/second llite.lustre-ffff9f0ee7346000.statahead_batch_max=0 llite.lustre-ffff9f0ee7346000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f0ee7346000.stats=clear mdc.lustre-MDT0001-mdc-ffff9f0ee7346000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m4.209s user 0m0.385s sys 0m17.352s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9f0ee7346000.statahead_batch_max=32 llite.lustre-ffff9f0ee7346000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f0ee7346000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9f0ee7346000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9f0ee7346000.stats=clear mdc.lustre-MDT0001-mdc-ffff9f0ee7346000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m36.814s user 0m0.330s sys 0m15.107s batched statahead RPC= hit count=9975 mdc.lustre-MDT0000-mdc-ffff9f0ee7346000.batch_stats= snapshot_time: 1788615121.092871037 secs.nsecs start_time: 1788615014.053426299 secs.nsecs elapsed_time: 107.039444738 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9f0ee7346000.batch_stats= snapshot_time: 1788615121.092953200 secs.nsecs start_time: 1788615014.053512741 secs.nsecs elapsed_time: 107.039440459 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9f0ee7346000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f0ee7346000.batch_stats=0 llite.lustre-ffff9f0ee7346000.statahead_max=2048 llite.lustre-ffff9f0ee7346000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9f0ee7346000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9f0ee7346000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.55 seconds: 39.27 ops/second llite.lustre-ffff9f0ee7346000.statahead_max=0 101 real 0m2.280s user 0m0.006s sys 0m0.340s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9f0ee7346000.statahead_max=2048 llite.lustre-ffff9f0ee7346000.statahead_max=2048 101 real 0m0.956s user 0m0.004s sys 0m0.178s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10074 miss_total: 26 swrong=0 ewrong=0 - open/close 834 (time 1788615410.10 total 10.01 last 83.32) total: 900 open/close in 11.06 seconds: 81.38 ops/second llite.lustre-ffff9f0ee7346000.statahead_max=0 1001 real 0m24.551s user 0m0.040s sys 0m3.111s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff9f0ee7346000.statahead_max=2048 llite.lustre-ffff9f0ee7346000.statahead_max=2048 1001 real 0m6.419s user 0m0.028s sys 0m1.556s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11073 miss_total: 27 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff9f0ee7346000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11073 miss_total: 27 mdc.lustre-MDT0000-mdc-ffff9f0ee7346000.batch_stats= snapshot_time: 1788615472.525858765 secs.nsecs start_time: 1788615123.293311506 secs.nsecs elapsed_time: 349.232547259 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 25 25 8: 3 37 62 16: 0 0 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff9f0ee7346000.batch_stats= snapshot_time: 1788615472.525924030 secs.nsecs start_time: 1788615123.293480473 secs.nsecs elapsed_time: 349.232443557 secs.nsecs subreqs per batch batches %% cum %% 1: 1 12 12 2: 0 0 12 4: 2 25 37 8: 1 12 50 16: 1 12 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 - open/close 884 (time 1788615484.73 total 10.00 last 88.40) - open/close 1684 (time 1788615494.73 total 20.00 last 79.98) - open/close 2553 (time 1788615504.73 total 30.01 last 86.85) - open/close 3248 (time 1788615514.74 total 40.01 last 69.47) - open/close 3878 (time 1788615524.75 total 50.02 last 62.93) - open/close 4576 (time 1788615534.75 total 60.03 last 69.79) - open/close 5461 (time 1788615544.78 total 70.05 last 88.28) - open/close 6209 (time 1788615554.78 total 80.05 last 74.79) - open/close 6974 (time 1788615564.79 total 90.06 last 76.42) - open/close 7833 (time 1788615574.81 total 100.09 last 85.67) - open/close 8642 (time 1788615584.82 total 110.09 last 80.85) - open/close 9526 (time 1788615594.83 total 120.10 last 88.34) total: 10000 open/close in 126.92 seconds: 78.79 ops/second llite.lustre-ffff9f0ee7346000.statahead_batch_max=0 llite.lustre-ffff9f0ee7346000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f0ee7346000.stats=clear mdc.lustre-MDT0001-mdc-ffff9f0ee7346000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.039s user 0m0.373s sys 0m16.347s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9f0ee7346000.statahead_batch_max=256 llite.lustre-ffff9f0ee7346000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9f0ee7346000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9f0ee7346000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9f0ee7346000.stats=clear mdc.lustre-MDT0001-mdc-ffff9f0ee7346000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m54.237s user 0m0.322s sys 0m15.539s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9f0ee7346000.batch_stats= snapshot_time: 1788615730.828138332 secs.nsecs start_time: 1788615666.066686976 secs.nsecs elapsed_time: 64.761451356 secs.nsecs subreqs per batch batches %% cum %% 1: 117 46 46 2: 44 17 63 4: 17 6 70 8: 12 4 74 16: 9 3 78 32: 7 2 81 64: 12 4 85 128: 2 0 86 256: 34 13 100 mdc.lustre-MDT0001-mdc-ffff9f0ee7346000.batch_stats= snapshot_time: 1788615730.828303516 secs.nsecs start_time: 1788615666.066848555 secs.nsecs elapsed_time: 64.761454961 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9f0ee7346000.statahead_batch_max=64 llite.lustre-ffff9f0ee7346000.statahead_max=128