== sanity test 123ad: Verify batching statahead works correctly ========================================================== 15:25:04 (1787858704) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff88cb109fa000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff88cb109fa000.batch_stats=0 llite.lustre-ffff88cb109fa000.statahead_max=32 llite.lustre-ffff88cb109fa000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff88cb109fa000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff88cb109fa000.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.24 seconds: 30.90 ops/second llite.lustre-ffff88cb109fa000.statahead_max=0 101 real 0m2.524s user 0m0.003s sys 0m0.326s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff88cb109fa000.statahead_max=32 llite.lustre-ffff88cb109fa000.statahead_max=32 101 real 0m1.453s user 0m0.002s sys 0m0.176s 'ls -l' 100 files with statahead: 1 sec statahead total: 126 statahead wrong: 2 agl total: 121 list_total: 32 fname_total: 81 hit_total: 4260 miss_total: 1301 swrong=2 ewrong=2 - open/close 623 (time 1787858735.68 total 10.01 last 62.25) total: 900 open/close in 14.53 seconds: 61.95 ops/second llite.lustre-ffff88cb109fa000.statahead_max=0 1001 real 0m23.543s user 0m0.031s sys 0m3.089s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff88cb109fa000.statahead_max=32 llite.lustre-ffff88cb109fa000.statahead_max=32 1001 real 0m7.749s user 0m0.022s sys 0m1.335s 'ls -l' 1000 files with statahead: 8 sec statahead total: 127 statahead wrong: 2 agl total: 122 list_total: 33 fname_total: 81 hit_total: 5258 miss_total: 1303 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 20 seconds llite.lustre-ffff88cb109fa000.statahead_stats= statahead total: 127 statahead wrong: 2 agl total: 122 list_total: 33 fname_total: 81 hit_total: 5258 miss_total: 1303 mdc.lustre-MDT0000-mdc-ffff88cb109fa000.batch_stats= snapshot_time: 1787858802.653036572 secs.nsecs start_time: 1787858710.591916359 secs.nsecs elapsed_time: 92.061120213 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 3 8 13 4: 1 2 16 8: 1 2 18 16: 23 62 81 32: 7 18 100 mdc.lustre-MDT0001-mdc-ffff88cb109fa000.batch_stats= snapshot_time: 1787858802.653838470 secs.nsecs start_time: 1787858710.592857747 secs.nsecs elapsed_time: 92.060980723 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 0 0 2 8: 4 10 12 16: 11 28 41 32: 23 58 100 - open/close 708 (time 1787858815.40 total 10.01 last 70.73) - open/close 1452 (time 1787858825.43 total 20.04 last 74.20) - open/close 2319 (time 1787858835.44 total 30.05 last 86.63) - open/close 3059 (time 1787858845.44 total 40.05 last 73.99) - open/close 3657 (time 1787858855.44 total 50.05 last 59.77) - open/close 4486 (time 1787858865.45 total 60.06 last 82.87) - open/close 5191 (time 1787858875.46 total 70.07 last 70.41) - open/close 5936 (time 1787858885.47 total 80.08 last 74.45) - open/close 6685 (time 1787858895.48 total 90.09 last 74.83) - open/close 7433 (time 1787858905.49 total 100.09 last 74.75) - open/close 8132 (time 1787858915.51 total 110.12 last 69.73) - open/close 8866 (time 1787858925.52 total 120.12 last 73.36) - open/close 9551 (time 1787858935.52 total 130.13 last 68.48) total: 10000 open/close in 137.86 seconds: 72.54 ops/second llite.lustre-ffff88cb109fa000.statahead_batch_max=0 llite.lustre-ffff88cb109fa000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff88cb109fa000.stats=clear mdc.lustre-MDT0001-mdc-ffff88cb109fa000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m53.760s user 0m0.327s sys 0m16.482s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff88cb109fa000.statahead_batch_max=32 llite.lustre-ffff88cb109fa000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff88cb109fa000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff88cb109fa000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff88cb109fa000.stats=clear mdc.lustre-MDT0001-mdc-ffff88cb109fa000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m37.730s user 0m0.306s sys 0m14.502s batched statahead RPC= hit count=9981 mdc.lustre-MDT0000-mdc-ffff88cb109fa000.batch_stats= snapshot_time: 1787859111.219294841 secs.nsecs start_time: 1787859004.541123204 secs.nsecs elapsed_time: 106.678171637 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff88cb109fa000.batch_stats= snapshot_time: 1787859111.219367958 secs.nsecs start_time: 1787859004.541209136 secs.nsecs elapsed_time: 106.678158822 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff88cb109fa000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff88cb109fa000.batch_stats=0 llite.lustre-ffff88cb109fa000.statahead_max=2048 llite.lustre-ffff88cb109fa000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff88cb109fa000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff88cb109fa000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.60 seconds: 38.39 ops/second llite.lustre-ffff88cb109fa000.statahead_max=0 101 real 0m2.151s user 0m0.006s sys 0m0.294s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff88cb109fa000.statahead_max=2048 llite.lustre-ffff88cb109fa000.statahead_max=2048 101 real 0m0.732s user 0m0.003s sys 0m0.217s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10080 miss_total: 20 swrong=0 ewrong=0 - open/close 769 (time 1787859414.22 total 10.01 last 76.80) total: 900 open/close in 11.65 seconds: 77.28 ops/second llite.lustre-ffff88cb109fa000.statahead_max=0 1001 real 0m21.312s user 0m0.037s sys 0m2.905s 'ls -l' 1000 files without statahead: 21 sec llite.lustre-ffff88cb109fa000.statahead_max=2048 llite.lustre-ffff88cb109fa000.statahead_max=2048 1001 real 0m5.339s user 0m0.024s sys 0m1.363s 'ls -l' 1000 files with statahead: 5 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff88cb109fa000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11079 miss_total: 21 mdc.lustre-MDT0000-mdc-ffff88cb109fa000.batch_stats= snapshot_time: 1787859474.069381813 secs.nsecs start_time: 1787859113.954786875 secs.nsecs elapsed_time: 360.114594938 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 1 14 28 4: 2 28 57 8: 1 14 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff88cb109fa000.batch_stats= snapshot_time: 1787859474.069615998 secs.nsecs start_time: 1787859113.954947238 secs.nsecs elapsed_time: 360.114668760 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 3 30 30 16: 3 30 60 32: 1 10 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 758 (time 1787859486.45 total 10.00 last 75.80) - open/close 1503 (time 1787859496.46 total 20.01 last 74.47) - open/close 2225 (time 1787859506.46 total 30.01 last 72.19) - open/close 2938 (time 1787859516.47 total 40.01 last 71.25) - open/close 3702 (time 1787859526.47 total 50.02 last 76.35) - open/close 4482 (time 1787859536.49 total 60.03 last 77.91) - open/close 5176 (time 1787859546.49 total 70.04 last 69.38) - open/close 5917 (time 1787859556.49 total 80.04 last 74.07) - open/close 6664 (time 1787859566.50 total 90.05 last 74.61) - open/close 7419 (time 1787859576.52 total 100.06 last 75.41) - open/close 8100 (time 1787859586.52 total 110.06 last 68.10) - open/close 8829 (time 1787859596.52 total 120.07 last 72.84) - open/close 9531 (time 1787859606.53 total 130.07 last 70.17) total: 10000 open/close in 137.46 seconds: 72.75 ops/second llite.lustre-ffff88cb109fa000.statahead_batch_max=0 llite.lustre-ffff88cb109fa000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff88cb109fa000.stats=clear mdc.lustre-MDT0001-mdc-ffff88cb109fa000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m58.959s user 0m0.318s sys 0m15.694s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff88cb109fa000.statahead_batch_max=256 llite.lustre-ffff88cb109fa000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff88cb109fa000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff88cb109fa000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff88cb109fa000.stats=clear mdc.lustre-MDT0001-mdc-ffff88cb109fa000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m44.047s user 0m0.312s sys 0m15.525s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff88cb109fa000.batch_stats= snapshot_time: 1787859735.224326711 secs.nsecs start_time: 1787859681.356591933 secs.nsecs elapsed_time: 53.867734778 secs.nsecs subreqs per batch batches %% cum %% 1: 183 51 51 2: 50 14 66 4: 33 9 75 8: 19 5 80 16: 7 1 82 32: 21 5 88 64: 4 1 90 128: 1 0 90 256: 34 9 100 mdc.lustre-MDT0001-mdc-ffff88cb109fa000.batch_stats= snapshot_time: 1787859735.224402474 secs.nsecs start_time: 1787859681.356671116 secs.nsecs elapsed_time: 53.867731358 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff88cb109fa000.statahead_batch_max=64 llite.lustre-ffff88cb109fa000.statahead_max=128