== sanity test 123ad: Verify batching statahead works correctly ========================================================== 08:35:19 (1787747719) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9d0c92157800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d0c92157800.batch_stats=0 llite.lustre-ffff9d0c92157800.statahead_max=32 llite.lustre-ffff9d0c92157800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9d0c92157800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d0c92157800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.95 seconds: 33.94 ops/second llite.lustre-ffff9d0c92157800.statahead_max=0 101 real 0m2.638s user 0m0.004s sys 0m0.328s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9d0c92157800.statahead_max=32 llite.lustre-ffff9d0c92157800.statahead_max=32 101 real 0m1.253s user 0m0.007s sys 0m0.186s 'ls -l' 100 files with statahead: 2 sec statahead total: 156 statahead wrong: 2 agl total: 149 list_total: 30 fname_total: 125 hit_total: 3656 miss_total: 1398 swrong=2 ewrong=2 - open/close 690 (time 1787747748.42 total 10.00 last 68.99) total: 900 open/close in 12.97 seconds: 69.40 ops/second llite.lustre-ffff9d0c92157800.statahead_max=0 1001 real 0m24.193s user 0m0.034s sys 0m2.925s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff9d0c92157800.statahead_max=32 llite.lustre-ffff9d0c92157800.statahead_max=32 1001 real 0m9.901s user 0m0.028s sys 0m1.361s 'ls -l' 1000 files with statahead: 10 sec statahead total: 157 statahead wrong: 2 agl total: 150 list_total: 31 fname_total: 125 hit_total: 4653 miss_total: 1401 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff9d0c92157800.statahead_stats= statahead total: 157 statahead wrong: 2 agl total: 150 list_total: 31 fname_total: 125 hit_total: 4653 miss_total: 1401 mdc.lustre-MDT0000-mdc-ffff9d0c92157800.batch_stats= snapshot_time: 1787747819.733060626 secs.nsecs start_time: 1787747723.839194327 secs.nsecs elapsed_time: 95.893866299 secs.nsecs subreqs per batch batches %% cum %% 1: 3 8 8 2: 2 5 13 4: 0 0 13 8: 2 5 18 16: 26 70 89 32: 4 10 100 mdc.lustre-MDT0001-mdc-ffff9d0c92157800.batch_stats= snapshot_time: 1787747819.733212173 secs.nsecs start_time: 1787747723.839323678 secs.nsecs elapsed_time: 95.893888495 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 2 2 8: 5 12 15 16: 10 25 41 32: 23 58 100 - open/close 1097 (time 1787747832.13 total 10.00 last 109.67) - open/close 2030 (time 1787747842.15 total 20.02 last 93.09) - open/close 2958 (time 1787747852.16 total 30.03 last 92.72) - open/close 3773 (time 1787747862.16 total 40.03 last 81.49) - open/close 4692 (time 1787747872.16 total 50.04 last 91.85) - open/close 5615 (time 1787747882.17 total 60.04 last 92.26) - open/close 6472 (time 1787747892.17 total 70.05 last 85.65) - open/close 7354 (time 1787747902.18 total 80.06 last 88.15) - open/close 8133 (time 1787747912.19 total 90.06 last 77.84) - open/close 9107 (time 1787747922.19 total 100.07 last 97.33) - open/close 9903 (time 1787747932.20 total 110.08 last 79.54) total: 10000 open/close in 111.11 seconds: 90.00 ops/second llite.lustre-ffff9d0c92157800.statahead_batch_max=0 llite.lustre-ffff9d0c92157800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d0c92157800.stats=clear mdc.lustre-MDT0001-mdc-ffff9d0c92157800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m11.077s user 0m0.338s sys 0m14.334s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9d0c92157800.statahead_batch_max=32 llite.lustre-ffff9d0c92157800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d0c92157800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9d0c92157800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9d0c92157800.stats=clear mdc.lustre-MDT0001-mdc-ffff9d0c92157800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m52.545s user 0m0.279s sys 0m13.581s batched statahead RPC= hit count=9977 mdc.lustre-MDT0000-mdc-ffff9d0c92157800.batch_stats= snapshot_time: 1787748135.316505580 secs.nsecs start_time: 1787748012.158039769 secs.nsecs elapsed_time: 123.158465811 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9d0c92157800.batch_stats= snapshot_time: 1787748135.316571660 secs.nsecs start_time: 1787748012.158114116 secs.nsecs elapsed_time: 123.158457544 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9d0c92157800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d0c92157800.batch_stats=0 llite.lustre-ffff9d0c92157800.statahead_max=2048 llite.lustre-ffff9d0c92157800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9d0c92157800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9d0c92157800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.60 seconds: 38.51 ops/second llite.lustre-ffff9d0c92157800.statahead_max=0 101 real 0m2.929s user 0m0.008s sys 0m0.321s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9d0c92157800.statahead_max=2048 llite.lustre-ffff9d0c92157800.statahead_max=2048 101 real 0m0.967s user 0m0.003s sys 0m0.153s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10076 miss_total: 24 swrong=0 ewrong=0 - open/close 822 (time 1787748442.50 total 10.01 last 82.14) total: 900 open/close in 11.09 seconds: 81.19 ops/second llite.lustre-ffff9d0c92157800.statahead_max=0 1001 real 0m24.057s user 0m0.033s sys 0m2.911s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff9d0c92157800.statahead_max=2048 llite.lustre-ffff9d0c92157800.statahead_max=2048 1001 real 0m8.634s user 0m0.035s sys 0m1.377s 'ls -l' 1000 files with statahead: 9 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff9d0c92157800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 mdc.lustre-MDT0000-mdc-ffff9d0c92157800.batch_stats= snapshot_time: 1787748513.364291742 secs.nsecs start_time: 1787748139.236681068 secs.nsecs elapsed_time: 374.127610674 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 4 44 44 8: 1 11 55 16: 1 11 66 32: 0 0 66 64: 1 11 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff9d0c92157800.batch_stats= snapshot_time: 1787748513.364363409 secs.nsecs start_time: 1787748139.236955628 secs.nsecs elapsed_time: 374.127407781 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 5 55 55 16: 1 11 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 - open/close 898 (time 1787748526.26 total 10.00 last 89.78) - open/close 1750 (time 1787748536.27 total 20.01 last 85.14) - open/close 2623 (time 1787748546.28 total 30.02 last 87.21) - open/close 3620 (time 1787748556.28 total 40.02 last 99.65) - open/close 4423 (time 1787748566.30 total 50.04 last 80.20) - open/close 5257 (time 1787748576.30 total 60.04 last 83.39) - open/close 5989 (time 1787748586.30 total 70.04 last 73.18) - open/close 6727 (time 1787748596.31 total 80.05 last 73.73) - open/close 7477 (time 1787748606.32 total 90.06 last 74.94) - open/close 8222 (time 1787748616.33 total 100.07 last 74.40) - open/close 8821 (time 1787748626.36 total 110.10 last 59.73) - open/close 9582 (time 1787748636.37 total 120.11 last 76.04) total: 10000 open/close in 125.05 seconds: 79.97 ops/second llite.lustre-ffff9d0c92157800.statahead_batch_max=0 llite.lustre-ffff9d0c92157800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d0c92157800.stats=clear mdc.lustre-MDT0001-mdc-ffff9d0c92157800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.538s user 0m0.313s sys 0m14.356s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9d0c92157800.statahead_batch_max=256 llite.lustre-ffff9d0c92157800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9d0c92157800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9d0c92157800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9d0c92157800.stats=clear mdc.lustre-MDT0001-mdc-ffff9d0c92157800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m58.852s user 0m0.316s sys 0m13.777s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9d0c92157800.batch_stats= snapshot_time: 1787748791.202193750 secs.nsecs start_time: 1787748719.765638929 secs.nsecs elapsed_time: 71.436554821 secs.nsecs subreqs per batch batches %% cum %% 1: 122 44 44 2: 42 15 59 4: 26 9 68 8: 18 6 75 16: 13 4 79 32: 13 4 84 64: 7 2 87 128: 2 0 87 256: 34 12 100 mdc.lustre-MDT0001-mdc-ffff9d0c92157800.batch_stats= snapshot_time: 1787748791.202261824 secs.nsecs start_time: 1787748719.766422178 secs.nsecs elapsed_time: 71.435839646 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9d0c92157800.statahead_batch_max=64 llite.lustre-ffff9d0c92157800.statahead_max=128