== sanity test 123ad: Verify batching statahead works correctly ========================================================== 13:51:57 (1785952317) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8aab48948800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8aab48948800.batch_stats=0 llite.lustre-ffff8aab48948800.statahead_max=32 llite.lustre-ffff8aab48948800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8aab48948800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8aab48948800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.50 seconds: 28.56 ops/second llite.lustre-ffff8aab48948800.statahead_max=0 101 real 0m3.244s user 0m0.003s sys 0m0.318s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8aab48948800.statahead_max=32 llite.lustre-ffff8aab48948800.statahead_max=32 101 real 0m1.500s user 0m0.003s sys 0m0.164s 'ls -l' 100 files with statahead: 2 sec statahead total: 130 statahead wrong: 2 agl total: 123 list_total: 33 fname_total: 84 hit_total: 4004 miss_total: 1567 swrong=2 ewrong=2 - open/close 701 (time 1785952350.38 total 10.00 last 70.08) total: 900 open/close in 14.96 seconds: 60.17 ops/second llite.lustre-ffff8aab48948800.statahead_max=0 1001 real 0m49.113s user 0m0.039s sys 0m3.281s 'ls -l' 1000 files without statahead: 49 sec llite.lustre-ffff8aab48948800.statahead_max=32 llite.lustre-ffff8aab48948800.statahead_max=32 1001 real 0m13.139s user 0m0.028s sys 0m1.520s 'ls -l' 1000 files with statahead: 13 sec statahead total: 131 statahead wrong: 2 agl total: 124 list_total: 34 fname_total: 84 hit_total: 5002 miss_total: 1569 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff8aab48948800.statahead_stats= statahead total: 131 statahead wrong: 2 agl total: 124 list_total: 34 fname_total: 84 hit_total: 5002 miss_total: 1569 mdc.lustre-MDT0000-mdc-ffff8aab48948800.batch_stats= snapshot_time: 1785952458.096737195 secs.nsecs start_time: 1785952323.401832192 secs.nsecs elapsed_time: 134.694905003 secs.nsecs subreqs per batch batches %% cum %% 1: 5 12 12 2: 1 2 15 4: 2 5 20 8: 0 0 20 16: 26 66 87 32: 5 12 100 mdc.lustre-MDT0001-mdc-ffff8aab48948800.batch_stats= snapshot_time: 1785952458.096811879 secs.nsecs start_time: 1785952323.401987900 secs.nsecs elapsed_time: 134.694823979 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 0 0 2 8: 4 10 12 16: 9 23 35 32: 25 64 100 - open/close 643 (time 1785952470.84 total 10.00 last 64.29) - open/close 1242 (time 1785952480.86 total 20.01 last 59.82) - open/close 1770 (time 1785952490.87 total 30.03 last 52.74) - open/close 2299 (time 1785952500.88 total 40.04 last 52.85) - open/close 2823 (time 1785952510.88 total 50.04 last 52.37) - open/close 3399 (time 1785952520.89 total 60.05 last 57.54) - open/close 4035 (time 1785952530.90 total 70.06 last 63.56) - open/close 5034 (time 1785952540.91 total 80.07 last 99.78) - open/close 5606 (time 1785952550.92 total 90.08 last 57.17) - open/close 6193 (time 1785952560.93 total 100.08 last 58.65) - open/close 6836 (time 1785952570.93 total 110.09 last 64.28) - open/close 7417 (time 1785952580.93 total 120.09 last 58.08) - open/close 8212 (time 1785952590.95 total 130.11 last 79.35) - open/close 8771 (time 1785952600.97 total 140.13 last 55.80) - open/close 9331 (time 1785952610.97 total 150.13 last 55.99) total: 10000 open/close in 158.51 seconds: 63.09 ops/second llite.lustre-ffff8aab48948800.statahead_batch_max=0 llite.lustre-ffff8aab48948800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8aab48948800.stats=clear mdc.lustre-MDT0001-mdc-ffff8aab48948800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m24.888s user 0m0.279s sys 0m15.794s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8aab48948800.statahead_batch_max=32 llite.lustre-ffff8aab48948800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8aab48948800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8aab48948800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8aab48948800.stats=clear mdc.lustre-MDT0001-mdc-ffff8aab48948800.stats=clear ls -l files in d123ad.sanity: 10001 real 2m22.246s user 0m0.263s sys 0m14.546s batched statahead RPC= hit count=9974 mdc.lustre-MDT0000-mdc-ffff8aab48948800.batch_stats= snapshot_time: 1785952878.509332207 secs.nsecs start_time: 1785952718.139507262 secs.nsecs elapsed_time: 160.369824945 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8aab48948800.batch_stats= snapshot_time: 1785952878.509390603 secs.nsecs start_time: 1785952718.139650286 secs.nsecs elapsed_time: 160.369740317 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8aab48948800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8aab48948800.batch_stats=0 llite.lustre-ffff8aab48948800.statahead_max=2048 llite.lustre-ffff8aab48948800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8aab48948800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8aab48948800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 1.63 seconds: 61.50 ops/second llite.lustre-ffff8aab48948800.statahead_max=0 101 real 0m2.245s user 0m0.007s sys 0m0.321s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8aab48948800.statahead_max=2048 llite.lustre-ffff8aab48948800.statahead_max=2048 101 real 0m1.175s user 0m0.003s sys 0m0.175s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10073 miss_total: 27 swrong=0 ewrong=0 - open/close 682 (time 1785953251.85 total 10.02 last 68.10) total: 900 open/close in 13.28 seconds: 67.79 ops/second llite.lustre-ffff8aab48948800.statahead_max=0 1001 real 0m23.212s user 0m0.026s sys 0m3.191s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff8aab48948800.statahead_max=2048 llite.lustre-ffff8aab48948800.statahead_max=2048 1001 real 0m7.485s user 0m0.017s sys 0m1.206s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11071 miss_total: 29 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff8aab48948800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11071 miss_total: 29 mdc.lustre-MDT0000-mdc-ffff8aab48948800.batch_stats= snapshot_time: 1785953321.572903068 secs.nsecs start_time: 1785952881.191523064 secs.nsecs elapsed_time: 440.381380004 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 25 25 8: 3 37 62 16: 0 0 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff8aab48948800.batch_stats= snapshot_time: 1785953321.572975988 secs.nsecs start_time: 1785952881.191652005 secs.nsecs elapsed_time: 440.381323983 secs.nsecs subreqs per batch batches %% cum %% 1: 1 12 12 2: 0 0 12 4: 2 25 37 8: 1 12 50 16: 1 12 62 32: 0 0 62 64: 1 12 75 128: 0 0 75 256: 2 25 100 - open/close 606 (time 1785953334.77 total 10.01 last 60.53) - open/close 1258 (time 1785953344.78 total 20.02 last 65.14) - open/close 1865 (time 1785953354.80 total 30.04 last 60.61) - open/close 2583 (time 1785953364.81 total 40.04 last 71.75) - open/close 3248 (time 1785953374.82 total 50.06 last 66.39) - open/close 3710 (time 1785953384.83 total 60.07 last 46.17) - open/close 4392 (time 1785953394.84 total 70.08 last 68.12) - open/close 4892 (time 1785953404.85 total 80.09 last 49.97) - open/close 5834 (time 1785953414.86 total 90.09 last 94.12) - open/close 6503 (time 1785953424.86 total 100.09 last 66.89) - open/close 7218 (time 1785953434.87 total 110.10 last 71.44) - open/close 7703 (time 1785953444.88 total 120.12 last 48.42) - open/close 8530 (time 1785953454.89 total 130.13 last 82.63) - open/close 9170 (time 1785953464.90 total 140.13 last 63.97) - open/close 9645 (time 1785953474.90 total 150.13 last 47.49) total: 10000 open/close in 156.47 seconds: 63.91 ops/second llite.lustre-ffff8aab48948800.statahead_batch_max=0 llite.lustre-ffff8aab48948800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8aab48948800.stats=clear mdc.lustre-MDT0001-mdc-ffff8aab48948800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m24.603s user 0m0.289s sys 0m15.219s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8aab48948800.statahead_batch_max=256 llite.lustre-ffff8aab48948800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8aab48948800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8aab48948800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8aab48948800.stats=clear mdc.lustre-MDT0001-mdc-ffff8aab48948800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m16.396s user 0m0.260s sys 0m14.096s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8aab48948800.batch_stats= snapshot_time: 1785953677.271548315 secs.nsecs start_time: 1785953581.829197341 secs.nsecs elapsed_time: 95.442350974 secs.nsecs subreqs per batch batches %% cum %% 1: 187 46 46 2: 71 17 64 4: 48 12 76 8: 22 5 82 16: 18 4 86 32: 13 3 90 64: 4 1 91 128: 1 0 91 256: 34 8 100 mdc.lustre-MDT0001-mdc-ffff8aab48948800.batch_stats= snapshot_time: 1785953677.273542733 secs.nsecs start_time: 1785953581.829326456 secs.nsecs elapsed_time: 95.444216277 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8aab48948800.statahead_batch_max=64 llite.lustre-ffff8aab48948800.statahead_max=128