== sanity test 123ad: Verify batching statahead works correctly ========================================================== 09:41:20 (1787665280) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff99d317e47000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff99d317e47000.batch_stats=0 llite.lustre-ffff99d317e47000.statahead_max=32 llite.lustre-ffff99d317e47000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff99d317e47000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff99d317e47000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.99 seconds: 33.42 ops/second llite.lustre-ffff99d317e47000.statahead_max=0 101 real 0m2.172s user 0m0.005s sys 0m0.366s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff99d317e47000.statahead_max=32 llite.lustre-ffff99d317e47000.statahead_max=32 101 real 0m1.086s user 0m0.005s sys 0m0.184s 'ls -l' 100 files with statahead: 1 sec statahead total: 129 statahead wrong: 2 agl total: 123 list_total: 32 fname_total: 68 hit_total: 3631 miss_total: 1565 swrong=2 ewrong=2 - open/close 743 (time 1787665309.22 total 10.00 last 74.27) total: 900 open/close in 12.02 seconds: 74.87 ops/second llite.lustre-ffff99d317e47000.statahead_max=0 1001 real 0m25.614s user 0m0.033s sys 0m3.457s 'ls -l' 1000 files without statahead: 26 sec llite.lustre-ffff99d317e47000.statahead_max=32 llite.lustre-ffff99d317e47000.statahead_max=32 1001 real 0m12.891s user 0m0.038s sys 0m1.692s 'ls -l' 1000 files with statahead: 12 sec statahead total: 130 statahead wrong: 2 agl total: 124 list_total: 33 fname_total: 68 hit_total: 4626 miss_total: 1570 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 25 seconds llite.lustre-ffff99d317e47000.statahead_stats= statahead total: 130 statahead wrong: 2 agl total: 124 list_total: 33 fname_total: 68 hit_total: 4626 miss_total: 1570 mdc.lustre-MDT0000-mdc-ffff99d317e47000.batch_stats= snapshot_time: 1787665388.455702877 secs.nsecs start_time: 1787665285.876437494 secs.nsecs elapsed_time: 102.579265383 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 5 5 4: 1 2 7 8: 4 10 17 16: 17 43 61 32: 15 38 100 mdc.lustre-MDT0001-mdc-ffff99d317e47000.batch_stats= snapshot_time: 1787665388.455755897 secs.nsecs start_time: 1787665285.876588545 secs.nsecs elapsed_time: 102.579167352 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 1 2 5 4: 1 2 7 8: 3 7 15 16: 22 55 70 32: 12 30 100 - open/close 823 (time 1787665400.58 total 10.01 last 82.24) - open/close 1621 (time 1787665410.58 total 20.01 last 79.75) - open/close 2388 (time 1787665420.59 total 30.02 last 76.66) - open/close 3155 (time 1787665430.62 total 40.05 last 76.46) - open/close 3882 (time 1787665440.62 total 50.05 last 72.68) - open/close 4442 (time 1787665450.63 total 60.06 last 55.95) - open/close 5021 (time 1787665460.63 total 70.06 last 57.89) - open/close 5726 (time 1787665470.64 total 80.07 last 70.42) - open/close 6536 (time 1787665480.66 total 90.09 last 80.84) - open/close 7210 (time 1787665490.67 total 100.10 last 67.38) - open/close 7847 (time 1787665500.67 total 110.10 last 63.67) - open/close 8351 (time 1787665510.67 total 120.10 last 50.40) - open/close 9065 (time 1787665520.68 total 130.11 last 71.37) - open/close 9705 (time 1787665530.68 total 140.11 last 63.95) total: 10000 open/close in 144.01 seconds: 69.44 ops/second llite.lustre-ffff99d317e47000.statahead_batch_max=0 llite.lustre-ffff99d317e47000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff99d317e47000.stats=clear mdc.lustre-MDT0001-mdc-ffff99d317e47000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m7.211s user 0m0.353s sys 0m16.980s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff99d317e47000.statahead_batch_max=32 llite.lustre-ffff99d317e47000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff99d317e47000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff99d317e47000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff99d317e47000.stats=clear mdc.lustre-MDT0001-mdc-ffff99d317e47000.stats=clear ls -l files in d123ad.sanity: 10001 real 2m0.171s user 0m0.313s sys 0m16.361s batched statahead RPC= hit count=9983 mdc.lustre-MDT0000-mdc-ffff99d317e47000.batch_stats= snapshot_time: 1787665748.010194532 secs.nsecs start_time: 1787665613.880520017 secs.nsecs elapsed_time: 134.129674515 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff99d317e47000.batch_stats= snapshot_time: 1787665748.010285713 secs.nsecs start_time: 1787665613.880662072 secs.nsecs elapsed_time: 134.129623641 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff99d317e47000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff99d317e47000.batch_stats=0 llite.lustre-ffff99d317e47000.statahead_max=2048 llite.lustre-ffff99d317e47000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff99d317e47000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff99d317e47000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 4.00 seconds: 25.00 ops/second llite.lustre-ffff99d317e47000.statahead_max=0 101 real 0m2.376s user 0m0.005s sys 0m0.343s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff99d317e47000.statahead_max=2048 llite.lustre-ffff99d317e47000.statahead_max=2048 101 real 0m0.901s user 0m0.010s sys 0m0.169s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10082 miss_total: 18 swrong=0 ewrong=0 - open/close 740 (time 1787666037.82 total 10.01 last 73.95) total: 900 open/close in 12.20 seconds: 73.77 ops/second llite.lustre-ffff99d317e47000.statahead_max=0 1001 real 0m20.062s user 0m0.035s sys 0m3.236s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff99d317e47000.statahead_max=2048 llite.lustre-ffff99d317e47000.statahead_max=2048 1001 real 0m6.916s user 0m0.031s sys 0m1.563s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 22 seconds llite.lustre-ffff99d317e47000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11081 miss_total: 19 mdc.lustre-MDT0000-mdc-ffff99d317e47000.batch_stats= snapshot_time: 1787666101.312455476 secs.nsecs start_time: 1787665750.690951267 secs.nsecs elapsed_time: 350.621504209 secs.nsecs subreqs per batch batches %% cum %% 1: 1 11 11 2: 1 11 22 4: 1 11 33 8: 2 22 55 16: 1 11 66 32: 1 11 77 64: 0 0 77 128: 0 0 77 256: 2 22 100 mdc.lustre-MDT0001-mdc-ffff99d317e47000.batch_stats= snapshot_time: 1787666101.312824787 secs.nsecs start_time: 1787665750.691113057 secs.nsecs elapsed_time: 350.621711730 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 11 11 4: 0 0 11 8: 4 44 55 16: 1 11 66 32: 0 0 66 64: 1 11 77 128: 0 0 77 256: 2 22 100 - open/close 909 (time 1787666114.29 total 10.01 last 90.85) - open/close 1724 (time 1787666124.30 total 20.01 last 81.45) - open/close 2536 (time 1787666134.31 total 30.02 last 81.15) - open/close 3277 (time 1787666144.31 total 40.02 last 74.05) - open/close 3938 (time 1787666154.32 total 50.04 last 66.02) - open/close 4412 (time 1787666164.32 total 60.04 last 47.40) - open/close 4905 (time 1787666174.33 total 70.04 last 49.26) - open/close 5462 (time 1787666184.34 total 80.05 last 55.65) - open/close 6058 (time 1787666194.35 total 90.06 last 59.53) - open/close 6707 (time 1787666204.35 total 100.06 last 64.89) - open/close 7535 (time 1787666214.36 total 110.07 last 82.75) - open/close 8277 (time 1787666224.37 total 120.08 last 74.15) - open/close 8800 (time 1787666234.37 total 130.08 last 52.28) - open/close 9634 (time 1787666244.37 total 140.08 last 83.37) total: 10000 open/close in 144.49 seconds: 69.21 ops/second llite.lustre-ffff99d317e47000.statahead_batch_max=0 llite.lustre-ffff99d317e47000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff99d317e47000.stats=clear mdc.lustre-MDT0001-mdc-ffff99d317e47000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m58.191s user 0m0.335s sys 0m16.619s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff99d317e47000.statahead_batch_max=256 llite.lustre-ffff99d317e47000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff99d317e47000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff99d317e47000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff99d317e47000.stats=clear mdc.lustre-MDT0001-mdc-ffff99d317e47000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.217s user 0m0.282s sys 0m15.394s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff99d317e47000.batch_stats= snapshot_time: 1787666393.795613925 secs.nsecs start_time: 1787666316.690724731 secs.nsecs elapsed_time: 77.104889194 secs.nsecs subreqs per batch batches %% cum %% 1: 149 44 44 2: 56 16 61 4: 35 10 71 8: 24 7 79 16: 13 3 82 32: 12 3 86 64: 10 2 89 128: 2 0 90 256: 33 9 100 mdc.lustre-MDT0001-mdc-ffff99d317e47000.batch_stats= snapshot_time: 1787666393.795690894 secs.nsecs start_time: 1787666316.690817166 secs.nsecs elapsed_time: 77.104873728 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff99d317e47000.statahead_batch_max=64 llite.lustre-ffff99d317e47000.statahead_max=128