== sanity test 123ad: Verify batching statahead works correctly ========================================================== 12:29:58 (1787848198) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9078bb2c5000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9078bb2c5000.batch_stats=0 llite.lustre-ffff9078bb2c5000.statahead_max=32 llite.lustre-ffff9078bb2c5000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9078bb2c5000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9078bb2c5000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.56 seconds: 28.09 ops/second llite.lustre-ffff9078bb2c5000.statahead_max=0 101 real 0m2.767s user 0m0.004s sys 0m0.335s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9078bb2c5000.statahead_max=32 llite.lustre-ffff9078bb2c5000.statahead_max=32 101 real 0m1.058s user 0m0.005s sys 0m0.153s 'ls -l' 100 files with statahead: 1 sec statahead total: 150 statahead wrong: 2 agl total: 144 list_total: 31 fname_total: 119 hit_total: 4245 miss_total: 1288 swrong=2 ewrong=2 - open/close 703 (time 1787848229.44 total 10.01 last 70.23) total: 900 open/close in 13.01 seconds: 69.16 ops/second llite.lustre-ffff9078bb2c5000.statahead_max=0 1001 real 0m29.663s user 0m0.046s sys 0m3.264s 'ls -l' 1000 files without statahead: 30 sec llite.lustre-ffff9078bb2c5000.statahead_max=32 llite.lustre-ffff9078bb2c5000.statahead_max=32 1001 real 0m11.678s user 0m0.032s sys 0m1.500s 'ls -l' 1000 files with statahead: 12 sec statahead total: 151 statahead wrong: 2 agl total: 145 list_total: 32 fname_total: 119 hit_total: 5244 miss_total: 1289 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 28 seconds llite.lustre-ffff9078bb2c5000.statahead_stats= statahead total: 151 statahead wrong: 2 agl total: 145 list_total: 32 fname_total: 119 hit_total: 5244 miss_total: 1289 mdc.lustre-MDT0000-mdc-ffff9078bb2c5000.batch_stats= snapshot_time: 1787848317.722097622 secs.nsecs start_time: 1787848204.389905251 secs.nsecs elapsed_time: 113.332192371 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 2 5 7 8: 2 5 12 16: 23 58 71 32: 11 28 100 mdc.lustre-MDT0001-mdc-ffff9078bb2c5000.batch_stats= snapshot_time: 1787848317.722180136 secs.nsecs start_time: 1787848204.389985036 secs.nsecs elapsed_time: 113.332195100 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 2 5 10 8: 4 10 20 16: 19 47 67 32: 13 32 100 - open/close 590 (time 1787848331.05 total 10.03 last 58.82) - open/close 1132 (time 1787848341.07 total 20.04 last 54.13) - open/close 1684 (time 1787848351.08 total 30.06 last 55.12) - open/close 2250 (time 1787848361.08 total 40.06 last 56.60) - open/close 2848 (time 1787848371.09 total 50.07 last 59.73) - open/close 3462 (time 1787848381.10 total 60.08 last 61.34) - open/close 4285 (time 1787848391.12 total 70.10 last 82.15) - open/close 5043 (time 1787848401.13 total 80.11 last 75.74) - open/close 5764 (time 1787848411.15 total 90.13 last 71.93) - open/close 6439 (time 1787848421.16 total 100.14 last 67.44) - open/close 7122 (time 1787848431.19 total 110.17 last 68.08) - open/close 7894 (time 1787848441.21 total 120.19 last 77.09) - open/close 8647 (time 1787848451.21 total 130.19 last 75.28) - open/close 9307 (time 1787848461.22 total 140.20 last 65.96) - open/close 9989 (time 1787848471.22 total 150.20 last 68.14) total: 10000 open/close in 150.40 seconds: 66.49 ops/second llite.lustre-ffff9078bb2c5000.statahead_batch_max=0 llite.lustre-ffff9078bb2c5000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9078bb2c5000.stats=clear mdc.lustre-MDT0001-mdc-ffff9078bb2c5000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m17.322s user 0m0.412s sys 0m16.305s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9078bb2c5000.statahead_batch_max=32 llite.lustre-ffff9078bb2c5000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9078bb2c5000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9078bb2c5000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9078bb2c5000.stats=clear mdc.lustre-MDT0001-mdc-ffff9078bb2c5000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m3.617s user 0m0.383s sys 0m15.069s batched statahead RPC= hit count=9969 mdc.lustre-MDT0000-mdc-ffff9078bb2c5000.batch_stats= snapshot_time: 1787848698.965629319 secs.nsecs start_time: 1787848559.145453448 secs.nsecs elapsed_time: 139.820175871 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 2 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9078bb2c5000.batch_stats= snapshot_time: 1787848698.965703056 secs.nsecs start_time: 1787848559.146615541 secs.nsecs elapsed_time: 139.819087515 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9078bb2c5000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9078bb2c5000.batch_stats=0 llite.lustre-ffff9078bb2c5000.statahead_max=2048 llite.lustre-ffff9078bb2c5000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9078bb2c5000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9078bb2c5000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.50 seconds: 40.06 ops/second llite.lustre-ffff9078bb2c5000.statahead_max=0 101 real 0m3.544s user 0m0.003s sys 0m0.354s 'ls -l' 100 files without statahead: 4 sec llite.lustre-ffff9078bb2c5000.statahead_max=2048 llite.lustre-ffff9078bb2c5000.statahead_max=2048 101 real 0m1.083s user 0m0.002s sys 0m0.198s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10068 miss_total: 32 swrong=0 ewrong=0 - open/close 753 (time 1787849018.01 total 10.00 last 75.27) total: 900 open/close in 12.19 seconds: 73.80 ops/second llite.lustre-ffff9078bb2c5000.statahead_max=0 1001 real 0m25.129s user 0m0.040s sys 0m3.176s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff9078bb2c5000.statahead_max=2048 llite.lustre-ffff9078bb2c5000.statahead_max=2048 1001 real 0m8.117s user 0m0.035s sys 0m1.413s 'ls -l' 1000 files with statahead: 8 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11067 miss_total: 33 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff9078bb2c5000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11067 miss_total: 33 mdc.lustre-MDT0000-mdc-ffff9078bb2c5000.batch_stats= snapshot_time: 1787849088.744994844 secs.nsecs start_time: 1787848702.033997142 secs.nsecs elapsed_time: 386.710997702 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 0 0 14 4: 2 28 42 8: 2 28 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff9078bb2c5000.batch_stats= snapshot_time: 1787849088.745081211 secs.nsecs start_time: 1787848702.034177091 secs.nsecs elapsed_time: 386.710904120 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 10 10 8: 2 20 30 16: 3 30 60 32: 1 10 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 604 (time 1787849102.27 total 10.01 last 60.32) - open/close 1289 (time 1787849112.28 total 20.03 last 68.42) - open/close 2019 (time 1787849122.28 total 30.03 last 73.00) - open/close 2879 (time 1787849132.28 total 40.03 last 85.99) - open/close 3742 (time 1787849142.29 total 50.04 last 86.22) - open/close 4580 (time 1787849152.30 total 60.05 last 83.71) - open/close 5367 (time 1787849162.30 total 70.05 last 78.69) - open/close 6073 (time 1787849172.31 total 80.06 last 70.51) - open/close 6883 (time 1787849182.32 total 90.06 last 80.98) - open/close 7662 (time 1787849192.32 total 100.06 last 77.90) - open/close 8412 (time 1787849202.33 total 110.07 last 74.91) - open/close 9204 (time 1787849212.33 total 120.08 last 79.17) - open/close 9802 (time 1787849222.34 total 130.09 last 59.75) total: 10000 open/close in 132.95 seconds: 75.22 ops/second llite.lustre-ffff9078bb2c5000.statahead_batch_max=0 llite.lustre-ffff9078bb2c5000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9078bb2c5000.stats=clear mdc.lustre-MDT0001-mdc-ffff9078bb2c5000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m2.377s user 0m0.368s sys 0m15.436s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9078bb2c5000.statahead_batch_max=256 llite.lustre-ffff9078bb2c5000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9078bb2c5000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9078bb2c5000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9078bb2c5000.stats=clear mdc.lustre-MDT0001-mdc-ffff9078bb2c5000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.130s user 0m0.319s sys 0m14.306s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9078bb2c5000.batch_stats= snapshot_time: 1787849371.942776723 secs.nsecs start_time: 1787849298.447386608 secs.nsecs elapsed_time: 73.495390115 secs.nsecs subreqs per batch batches %% cum %% 1: 52 25 25 2: 18 8 34 4: 49 24 59 8: 13 6 65 16: 11 5 71 32: 14 6 78 64: 8 3 82 128: 1 0 82 256: 35 17 100 mdc.lustre-MDT0001-mdc-ffff9078bb2c5000.batch_stats= snapshot_time: 1787849371.942861607 secs.nsecs start_time: 1787849298.447514160 secs.nsecs elapsed_time: 73.495347447 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9078bb2c5000.statahead_batch_max=64 llite.lustre-ffff9078bb2c5000.statahead_max=128