== sanity test 123ad: Verify batching statahead works correctly ========================================================== 13:00:31 (1787850031) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8f9dc82a5800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8f9dc82a5800.batch_stats=0 llite.lustre-ffff8f9dc82a5800.statahead_max=32 llite.lustre-ffff8f9dc82a5800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8f9dc82a5800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8f9dc82a5800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.18 seconds: 31.45 ops/second llite.lustre-ffff8f9dc82a5800.statahead_max=0 101 real 0m2.757s user 0m0.007s sys 0m0.371s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8f9dc82a5800.statahead_max=32 llite.lustre-ffff8f9dc82a5800.statahead_max=32 101 real 0m1.126s user 0m0.007s sys 0m0.184s 'ls -l' 100 files with statahead: 1 sec statahead total: 144 statahead wrong: 2 agl total: 138 list_total: 31 fname_total: 93 hit_total: 3515 miss_total: 1383 swrong=2 ewrong=2 - open/close 661 (time 1787850061.96 total 10.00 last 66.09) total: 900 open/close in 13.29 seconds: 67.71 ops/second llite.lustre-ffff8f9dc82a5800.statahead_max=0 1001 real 0m22.721s user 0m0.034s sys 0m3.004s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff8f9dc82a5800.statahead_max=32 llite.lustre-ffff8f9dc82a5800.statahead_max=32 1001 real 0m9.528s user 0m0.029s sys 0m1.381s 'ls -l' 1000 files with statahead: 9 sec statahead total: 145 statahead wrong: 2 agl total: 139 list_total: 32 fname_total: 93 hit_total: 4511 miss_total: 1387 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 24 seconds llite.lustre-ffff8f9dc82a5800.statahead_stats= statahead total: 145 statahead wrong: 2 agl total: 139 list_total: 32 fname_total: 93 hit_total: 4511 miss_total: 1387 mdc.lustre-MDT0000-mdc-ffff8f9dc82a5800.batch_stats= snapshot_time: 1787850134.661374242 secs.nsecs start_time: 1787850037.046328034 secs.nsecs elapsed_time: 97.615046208 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 7 7 8: 3 7 15 16: 19 48 64 32: 14 35 100 mdc.lustre-MDT0001-mdc-ffff8f9dc82a5800.batch_stats= snapshot_time: 1787850134.661475910 secs.nsecs start_time: 1787850037.046425192 secs.nsecs elapsed_time: 97.615050718 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 5 5 4: 2 5 10 8: 2 5 15 16: 20 51 66 32: 13 33 100 - open/close 655 (time 1787850148.09 total 10.01 last 65.46) - open/close 1325 (time 1787850158.09 total 20.01 last 66.97) - open/close 1935 (time 1787850168.09 total 30.01 last 60.98) - open/close 2510 (time 1787850178.10 total 40.02 last 57.48) - open/close 3043 (time 1787850188.11 total 50.03 last 53.23) - open/close 3536 (time 1787850198.11 total 60.03 last 49.29) - open/close 4030 (time 1787850208.13 total 70.05 last 49.30) - open/close 4532 (time 1787850218.16 total 80.08 last 50.05) - open/close 5009 (time 1787850228.16 total 90.08 last 47.69) - open/close 5499 (time 1787850238.18 total 100.10 last 48.90) - open/close 6023 (time 1787850248.20 total 110.12 last 52.32) - open/close 6612 (time 1787850258.20 total 120.12 last 58.88) - open/close 7232 (time 1787850268.22 total 130.14 last 61.90) - open/close 7906 (time 1787850278.22 total 140.14 last 67.40) - open/close 8513 (time 1787850288.23 total 150.15 last 60.66) - open/close 9162 (time 1787850298.23 total 160.15 last 64.89) - open/close 9814 (time 1787850308.25 total 170.17 last 65.05) total: 10000 open/close in 173.65 seconds: 57.59 ops/second llite.lustre-ffff8f9dc82a5800.statahead_batch_max=0 llite.lustre-ffff8f9dc82a5800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8f9dc82a5800.stats=clear mdc.lustre-MDT0001-mdc-ffff8f9dc82a5800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m8.950s user 0m0.322s sys 0m14.962s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8f9dc82a5800.statahead_batch_max=32 llite.lustre-ffff8f9dc82a5800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8f9dc82a5800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8f9dc82a5800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8f9dc82a5800.stats=clear mdc.lustre-MDT0001-mdc-ffff8f9dc82a5800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m52.330s user 0m0.325s sys 0m14.265s batched statahead RPC= hit count=9985 mdc.lustre-MDT0000-mdc-ffff8f9dc82a5800.batch_stats= snapshot_time: 1787850517.457004540 secs.nsecs start_time: 1787850393.151767710 secs.nsecs elapsed_time: 124.305236830 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8f9dc82a5800.batch_stats= snapshot_time: 1787850517.457069277 secs.nsecs start_time: 1787850393.152131613 secs.nsecs elapsed_time: 124.304937664 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8f9dc82a5800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8f9dc82a5800.batch_stats=0 llite.lustre-ffff8f9dc82a5800.statahead_max=2048 llite.lustre-ffff8f9dc82a5800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8f9dc82a5800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8f9dc82a5800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.84 seconds: 35.25 ops/second llite.lustre-ffff8f9dc82a5800.statahead_max=0 101 real 0m2.770s user 0m0.003s sys 0m0.320s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8f9dc82a5800.statahead_max=2048 llite.lustre-ffff8f9dc82a5800.statahead_max=2048 101 real 0m0.836s user 0m0.005s sys 0m0.167s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10084 miss_total: 16 swrong=0 ewrong=0 - open/close 637 (time 1787850822.03 total 10.01 last 63.66) total: 900 open/close in 16.14 seconds: 55.77 ops/second llite.lustre-ffff8f9dc82a5800.statahead_max=0 1001 real 0m35.922s user 0m0.036s sys 0m3.077s 'ls -l' 1000 files without statahead: 36 sec llite.lustre-ffff8f9dc82a5800.statahead_max=2048 llite.lustre-ffff8f9dc82a5800.statahead_max=2048 1001 real 0m6.767s user 0m0.036s sys 0m1.467s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff8f9dc82a5800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11083 miss_total: 17 mdc.lustre-MDT0000-mdc-ffff8f9dc82a5800.batch_stats= snapshot_time: 1787850906.520218509 secs.nsecs start_time: 1787850520.714989897 secs.nsecs elapsed_time: 385.805228612 secs.nsecs subreqs per batch batches %% cum %% 1: 1 12 12 2: 2 25 37 4: 1 12 50 8: 1 12 62 16: 0 0 62 32: 1 12 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff8f9dc82a5800.batch_stats= snapshot_time: 1787850906.520289424 secs.nsecs start_time: 1787850520.715178934 secs.nsecs elapsed_time: 385.805110490 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 20 20 8: 3 30 50 16: 0 0 50 32: 2 20 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 702 (time 1787850919.07 total 10.00 last 70.19) - open/close 1397 (time 1787850929.07 total 20.00 last 69.48) - open/close 2269 (time 1787850939.07 total 30.00 last 87.20) - open/close 3013 (time 1787850949.08 total 40.01 last 74.33) - open/close 3792 (time 1787850959.09 total 50.02 last 77.84) - open/close 4545 (time 1787850969.09 total 60.02 last 75.29) - open/close 5342 (time 1787850979.10 total 70.03 last 79.62) - open/close 6035 (time 1787850989.11 total 80.04 last 69.23) - open/close 6801 (time 1787850999.12 total 90.05 last 76.58) - open/close 7547 (time 1787851009.13 total 100.06 last 74.51) - open/close 8322 (time 1787851019.13 total 110.06 last 77.46) - open/close 9022 (time 1787851029.13 total 120.07 last 69.99) - open/close 9742 (time 1787851039.14 total 130.07 last 71.95) total: 10000 open/close in 134.69 seconds: 74.24 ops/second llite.lustre-ffff8f9dc82a5800.statahead_batch_max=0 llite.lustre-ffff8f9dc82a5800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8f9dc82a5800.stats=clear mdc.lustre-MDT0001-mdc-ffff8f9dc82a5800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.836s user 0m0.346s sys 0m15.236s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8f9dc82a5800.statahead_batch_max=256 llite.lustre-ffff8f9dc82a5800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8f9dc82a5800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8f9dc82a5800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8f9dc82a5800.stats=clear mdc.lustre-MDT0001-mdc-ffff8f9dc82a5800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m1.509s user 0m0.323s sys 0m14.799s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8f9dc82a5800.batch_stats= snapshot_time: 1787851190.896251248 secs.nsecs start_time: 1787851116.789271512 secs.nsecs elapsed_time: 74.106979736 secs.nsecs subreqs per batch batches %% cum %% 1: 76 33 33 2: 25 11 45 4: 28 12 57 8: 24 10 68 16: 11 4 73 32: 15 6 79 64: 11 4 84 128: 0 0 84 256: 34 15 100 mdc.lustre-MDT0001-mdc-ffff8f9dc82a5800.batch_stats= snapshot_time: 1787851190.896315098 secs.nsecs start_time: 1787851116.789420162 secs.nsecs elapsed_time: 74.106894936 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8f9dc82a5800.statahead_batch_max=64 llite.lustre-ffff8f9dc82a5800.statahead_max=128