== sanity test 123ad: Verify batching statahead works correctly ========================================================== 20:12:11 (1787789531) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8c0943f0a800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c0943f0a800.batch_stats=0 llite.lustre-ffff8c0943f0a800.statahead_max=32 llite.lustre-ffff8c0943f0a800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8c0943f0a800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c0943f0a800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.15 seconds: 46.61 ops/second llite.lustre-ffff8c0943f0a800.statahead_max=0 101 real 0m1.868s user 0m0.004s sys 0m0.298s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8c0943f0a800.statahead_max=32 llite.lustre-ffff8c0943f0a800.statahead_max=32 101 real 0m1.075s user 0m0.003s sys 0m0.157s 'ls -l' 100 files with statahead: 1 sec statahead total: 131 statahead wrong: 2 agl total: 125 list_total: 31 fname_total: 100 hit_total: 4094 miss_total: 1520 swrong=2 ewrong=2 total: 900 open/close in 9.03 seconds: 99.65 ops/second llite.lustre-ffff8c0943f0a800.statahead_max=0 1001 real 0m23.009s user 0m0.028s sys 0m2.851s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff8c0943f0a800.statahead_max=32 llite.lustre-ffff8c0943f0a800.statahead_max=32 1001 real 0m7.845s user 0m0.032s sys 0m1.338s 'ls -l' 1000 files with statahead: 8 sec statahead total: 132 statahead wrong: 2 agl total: 126 list_total: 32 fname_total: 100 hit_total: 5090 miss_total: 1524 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff8c0943f0a800.statahead_stats= statahead total: 132 statahead wrong: 2 agl total: 126 list_total: 32 fname_total: 100 hit_total: 5090 miss_total: 1524 mdc.lustre-MDT0000-mdc-ffff8c0943f0a800.batch_stats= snapshot_time: 1787789615.423014701 secs.nsecs start_time: 1787789535.801723964 secs.nsecs elapsed_time: 79.621290737 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 3 7 12 8: 0 0 12 16: 21 53 66 32: 13 33 100 mdc.lustre-MDT0001-mdc-ffff8c0943f0a800.batch_stats= snapshot_time: 1787789615.423489589 secs.nsecs start_time: 1787789535.801806508 secs.nsecs elapsed_time: 79.621683081 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 3 7 15 16: 16 41 56 32: 17 43 100 - open/close 1032 (time 1787789627.94 total 10.00 last 103.15) - open/close 2079 (time 1787789637.94 total 20.01 last 104.68) - open/close 3082 (time 1787789647.95 total 30.01 last 100.22) - open/close 3984 (time 1787789657.95 total 40.02 last 90.15) - open/close 5005 (time 1787789667.98 total 50.05 last 101.83) - open/close 6089 (time 1787789677.99 total 60.05 last 108.32) - open/close 7027 (time 1787789687.99 total 70.06 last 93.78) - open/close 7959 (time 1787789697.99 total 80.06 last 93.20) - open/close 8897 (time 1787789708.00 total 90.06 last 93.74) - open/close 9844 (time 1787789718.00 total 100.07 last 94.68) total: 10000 open/close in 101.46 seconds: 98.57 ops/second llite.lustre-ffff8c0943f0a800.statahead_batch_max=0 llite.lustre-ffff8c0943f0a800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c0943f0a800.stats=clear mdc.lustre-MDT0001-mdc-ffff8c0943f0a800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.126s user 0m0.305s sys 0m14.453s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8c0943f0a800.statahead_batch_max=32 llite.lustre-ffff8c0943f0a800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c0943f0a800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8c0943f0a800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8c0943f0a800.stats=clear mdc.lustre-MDT0001-mdc-ffff8c0943f0a800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m25.495s user 0m0.284s sys 0m13.512s batched statahead RPC= hit count=9982 mdc.lustre-MDT0000-mdc-ffff8c0943f0a800.batch_stats= snapshot_time: 1787789876.239120144 secs.nsecs start_time: 1787789781.894929656 secs.nsecs elapsed_time: 94.344190488 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8c0943f0a800.batch_stats= snapshot_time: 1787789876.239206446 secs.nsecs start_time: 1787789781.895016196 secs.nsecs elapsed_time: 94.344190250 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8c0943f0a800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c0943f0a800.batch_stats=0 llite.lustre-ffff8c0943f0a800.statahead_max=2048 llite.lustre-ffff8c0943f0a800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8c0943f0a800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8c0943f0a800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.02 seconds: 49.58 ops/second llite.lustre-ffff8c0943f0a800.statahead_max=0 101 real 0m1.690s user 0m0.010s sys 0m0.294s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff8c0943f0a800.statahead_max=2048 llite.lustre-ffff8c0943f0a800.statahead_max=2048 101 real 0m0.814s user 0m0.003s sys 0m0.183s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10081 miss_total: 19 swrong=0 ewrong=0 total: 900 open/close in 9.71 seconds: 92.71 ops/second llite.lustre-ffff8c0943f0a800.statahead_max=0 1001 real 0m18.221s user 0m0.039s sys 0m2.776s 'ls -l' 1000 files without statahead: 18 sec llite.lustre-ffff8c0943f0a800.statahead_max=2048 llite.lustre-ffff8c0943f0a800.statahead_max=2048 1001 real 0m5.811s user 0m0.023s sys 0m1.422s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 17 seconds llite.lustre-ffff8c0943f0a800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11080 miss_total: 20 mdc.lustre-MDT0000-mdc-ffff8c0943f0a800.batch_stats= snapshot_time: 1787790181.322118938 secs.nsecs start_time: 1787789878.104989693 secs.nsecs elapsed_time: 303.217129245 secs.nsecs subreqs per batch batches %% cum %% 1: 2 25 25 2: 0 0 25 4: 3 37 62 8: 0 0 62 16: 1 12 75 32: 0 0 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff8c0943f0a800.batch_stats= snapshot_time: 1787790181.322189850 secs.nsecs start_time: 1787789878.105144456 secs.nsecs elapsed_time: 303.217045394 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 10 10 8: 3 30 40 16: 2 20 60 32: 1 10 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 1033 (time 1787790193.48 total 10.00 last 103.27) - open/close 2066 (time 1787790203.49 total 20.01 last 103.24) - open/close 3134 (time 1787790213.49 total 30.02 last 106.72) - open/close 4076 (time 1787790223.50 total 40.02 last 94.16) - open/close 5068 (time 1787790233.50 total 50.02 last 99.17) - open/close 6183 (time 1787790243.50 total 60.02 last 111.50) - open/close 7187 (time 1787790253.51 total 70.03 last 100.31) - open/close 8222 (time 1787790263.51 total 80.04 last 103.47) - open/close 8974 (time 1787790273.53 total 90.05 last 75.09) - open/close 9746 (time 1787790283.54 total 100.06 last 77.15) total: 10000 open/close in 103.02 seconds: 97.07 ops/second llite.lustre-ffff8c0943f0a800.statahead_batch_max=0 llite.lustre-ffff8c0943f0a800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c0943f0a800.stats=clear mdc.lustre-MDT0001-mdc-ffff8c0943f0a800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m53.406s user 0m0.327s sys 0m14.472s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8c0943f0a800.statahead_batch_max=256 llite.lustre-ffff8c0943f0a800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8c0943f0a800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8c0943f0a800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8c0943f0a800.stats=clear mdc.lustre-MDT0001-mdc-ffff8c0943f0a800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m45.945s user 0m0.315s sys 0m14.941s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8c0943f0a800.batch_stats= snapshot_time: 1787790404.341567994 secs.nsecs start_time: 1787790347.281459149 secs.nsecs elapsed_time: 57.060108845 secs.nsecs subreqs per batch batches %% cum %% 1: 160 51 51 2: 37 12 63 4: 23 7 71 8: 23 7 78 16: 10 3 82 32: 14 4 86 64: 6 1 88 128: 0 0 88 256: 35 11 100 mdc.lustre-MDT0001-mdc-ffff8c0943f0a800.batch_stats= snapshot_time: 1787790404.341635369 secs.nsecs start_time: 1787790347.281648806 secs.nsecs elapsed_time: 57.059986563 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8c0943f0a800.statahead_batch_max=64 llite.lustre-ffff8c0943f0a800.statahead_max=128