== sanity test 123ad: Verify batching statahead works correctly ========================================================== 12:07:02 (1787846822) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9af707cb0000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9af707cb0000.batch_stats=0 llite.lustre-ffff9af707cb0000.statahead_max=32 llite.lustre-ffff9af707cb0000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9af707cb0000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9af707cb0000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.42 seconds: 41.30 ops/second llite.lustre-ffff9af707cb0000.statahead_max=0 101 real 0m1.505s user 0m0.003s sys 0m0.302s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9af707cb0000.statahead_max=32 llite.lustre-ffff9af707cb0000.statahead_max=32 101 real 0m0.979s user 0m0.004s sys 0m0.167s 'ls -l' 100 files with statahead: 1 sec statahead total: 139 statahead wrong: 2 agl total: 133 list_total: 31 fname_total: 88 hit_total: 3852 miss_total: 1761 swrong=2 ewrong=2 - open/close 892 (time 1787846849.13 total 10.01 last 89.15) total: 900 open/close in 10.10 seconds: 89.11 ops/second llite.lustre-ffff9af707cb0000.statahead_max=0 1001 real 0m23.924s user 0m0.030s sys 0m3.049s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff9af707cb0000.statahead_max=32 llite.lustre-ffff9af707cb0000.statahead_max=32 1001 real 0m9.554s user 0m0.031s sys 0m1.334s 'ls -l' 1000 files with statahead: 10 sec statahead total: 140 statahead wrong: 2 agl total: 134 list_total: 32 fname_total: 88 hit_total: 4846 miss_total: 1767 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff9af707cb0000.statahead_stats= statahead total: 140 statahead wrong: 2 agl total: 134 list_total: 32 fname_total: 88 hit_total: 4846 miss_total: 1767 mdc.lustre-MDT0000-mdc-ffff9af707cb0000.batch_stats= snapshot_time: 1787846915.209309969 secs.nsecs start_time: 1787846827.799818200 secs.nsecs elapsed_time: 87.409491769 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 5 12 17 16: 21 52 70 32: 12 30 100 mdc.lustre-MDT0001-mdc-ffff9af707cb0000.batch_stats= snapshot_time: 1787846915.209382369 secs.nsecs start_time: 1787846827.799966442 secs.nsecs elapsed_time: 87.409415927 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 2 2 4: 4 10 12 8: 1 2 15 16: 19 48 64 32: 14 35 100 - open/close 1031 (time 1787846928.03 total 10.00 last 103.08) - open/close 1887 (time 1787846938.04 total 20.01 last 85.49) - open/close 2760 (time 1787846948.05 total 30.02 last 87.24) - open/close 3636 (time 1787846958.06 total 40.03 last 87.54) - open/close 4526 (time 1787846968.06 total 50.04 last 88.92) - open/close 5472 (time 1787846978.07 total 60.04 last 94.56) - open/close 6388 (time 1787846988.07 total 70.04 last 91.59) - open/close 7334 (time 1787846998.07 total 80.05 last 94.56) - open/close 8227 (time 1787847008.07 total 90.05 last 89.29) - open/close 9223 (time 1787847018.08 total 100.05 last 99.56) total: 10000 open/close in 109.01 seconds: 91.73 ops/second llite.lustre-ffff9af707cb0000.statahead_batch_max=0 llite.lustre-ffff9af707cb0000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9af707cb0000.stats=clear mdc.lustre-MDT0001-mdc-ffff9af707cb0000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m12.010s user 0m0.324s sys 0m14.572s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9af707cb0000.statahead_batch_max=32 llite.lustre-ffff9af707cb0000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9af707cb0000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9af707cb0000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9af707cb0000.stats=clear mdc.lustre-MDT0001-mdc-ffff9af707cb0000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m36.533s user 0m0.300s sys 0m13.470s batched statahead RPC= hit count=9977 mdc.lustre-MDT0000-mdc-ffff9af707cb0000.batch_stats= snapshot_time: 1787847216.454462642 secs.nsecs start_time: 1787847107.257867187 secs.nsecs elapsed_time: 109.196595455 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9af707cb0000.batch_stats= snapshot_time: 1787847216.454535879 secs.nsecs start_time: 1787847107.258034991 secs.nsecs elapsed_time: 109.196500888 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9af707cb0000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9af707cb0000.batch_stats=0 llite.lustre-ffff9af707cb0000.statahead_max=2048 llite.lustre-ffff9af707cb0000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9af707cb0000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9af707cb0000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 1.93 seconds: 51.93 ops/second llite.lustre-ffff9af707cb0000.statahead_max=0 101 real 0m1.898s user 0m0.008s sys 0m0.292s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff9af707cb0000.statahead_max=2048 llite.lustre-ffff9af707cb0000.statahead_max=2048 101 real 0m0.959s user 0m0.005s sys 0m0.153s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10076 miss_total: 24 swrong=0 ewrong=0 - open/close 820 (time 1787847481.42 total 10.01 last 81.88) total: 900 open/close in 11.45 seconds: 78.62 ops/second llite.lustre-ffff9af707cb0000.statahead_max=0 1001 real 0m20.212s user 0m0.032s sys 0m2.988s 'ls -l' 1000 files without statahead: 20 sec llite.lustre-ffff9af707cb0000.statahead_max=2048 llite.lustre-ffff9af707cb0000.statahead_max=2048 1001 real 0m6.112s user 0m0.022s sys 0m1.310s 'ls -l' 1000 files with statahead: 6 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff9af707cb0000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 mdc.lustre-MDT0000-mdc-ffff9af707cb0000.batch_stats= snapshot_time: 1787847539.345026407 secs.nsecs start_time: 1787847218.875036010 secs.nsecs elapsed_time: 320.469990397 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 16 16 8: 3 50 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff9af707cb0000.batch_stats= snapshot_time: 1787847539.345094017 secs.nsecs start_time: 1787847218.875232012 secs.nsecs elapsed_time: 320.469862005 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 11 11 4: 1 11 22 8: 1 11 33 16: 2 22 55 32: 1 11 66 64: 0 0 66 128: 1 11 77 256: 2 22 100 - open/close 978 (time 1787847551.81 total 10.01 last 97.74) - open/close 2023 (time 1787847561.81 total 20.01 last 104.47) - open/close 3026 (time 1787847571.82 total 30.01 last 100.28) - open/close 4009 (time 1787847581.83 total 40.03 last 98.14) - open/close 4941 (time 1787847591.84 total 50.04 last 93.10) - open/close 5830 (time 1787847601.85 total 60.04 last 88.86) - open/close 6771 (time 1787847611.85 total 70.04 last 94.06) - open/close 7677 (time 1787847621.85 total 80.05 last 90.59) - open/close 8714 (time 1787847631.85 total 90.05 last 103.69) - open/close 9777 (time 1787847641.86 total 100.05 last 106.24) total: 10000 open/close in 102.24 seconds: 97.81 ops/second llite.lustre-ffff9af707cb0000.statahead_batch_max=0 llite.lustre-ffff9af707cb0000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9af707cb0000.stats=clear mdc.lustre-MDT0001-mdc-ffff9af707cb0000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m47.271s user 0m0.306s sys 0m14.465s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9af707cb0000.statahead_batch_max=256 llite.lustre-ffff9af707cb0000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9af707cb0000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9af707cb0000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9af707cb0000.stats=clear mdc.lustre-MDT0001-mdc-ffff9af707cb0000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m45.378s user 0m0.265s sys 0m14.088s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9af707cb0000.batch_stats= snapshot_time: 1787847751.501901913 secs.nsecs start_time: 1787847698.220761753 secs.nsecs elapsed_time: 53.281140160 secs.nsecs subreqs per batch batches %% cum %% 1: 61 30 30 2: 19 9 40 4: 32 16 56 8: 16 8 64 16: 6 3 68 32: 17 8 76 64: 11 5 82 128: 1 0 82 256: 34 17 100 mdc.lustre-MDT0001-mdc-ffff9af707cb0000.batch_stats= snapshot_time: 1787847751.501968489 secs.nsecs start_time: 1787847698.220828038 secs.nsecs elapsed_time: 53.281140451 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9af707cb0000.statahead_batch_max=64 llite.lustre-ffff9af707cb0000.statahead_max=128