== sanity test 123ad: Verify batching statahead works correctly ========================================================== 03:54:31 (1789026871) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff95aa04f95000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff95aa04f95000.batch_stats=0 llite.lustre-ffff95aa04f95000.statahead_max=32 llite.lustre-ffff95aa04f95000.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff95aa04f95000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff95aa04f95000.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.82 seconds: 35.41 ops/second llite.lustre-ffff95aa04f95000.statahead_max=0 101 real 0m2.863s user 0m0.004s sys 0m0.319s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff95aa04f95000.statahead_max=32 llite.lustre-ffff95aa04f95000.statahead_max=32 101 real 0m1.340s user 0m0.005s sys 0m0.184s 'ls -l' 100 files with statahead: 1 sec statahead total: 126 statahead wrong: 2 agl total: 121 list_total: 31 fname_total: 95 hit_total: 4275 miss_total: 1293 swrong=2 ewrong=2 - open/close 637 (time 1789026900.30 total 10.01 last 63.61) total: 900 open/close in 14.17 seconds: 63.51 ops/second llite.lustre-ffff95aa04f95000.statahead_max=0 1001 real 0m24.215s user 0m0.030s sys 0m2.775s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff95aa04f95000.statahead_max=32 llite.lustre-ffff95aa04f95000.statahead_max=32 1001 real 0m8.846s user 0m0.030s sys 0m1.415s 'ls -l' 1000 files with statahead: 9 sec statahead total: 127 statahead wrong: 2 agl total: 122 list_total: 32 fname_total: 95 hit_total: 5273 miss_total: 1295 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff95aa04f95000.statahead_stats= statahead total: 127 statahead wrong: 2 agl total: 122 list_total: 32 fname_total: 95 hit_total: 5273 miss_total: 1295 mdc.lustre-MDT0000-mdc-ffff95aa04f95000.batch_stats= snapshot_time: 1789026971.533783111 secs.nsecs start_time: 1789026875.353413373 secs.nsecs elapsed_time: 96.180369738 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 0 0 5 4: 2 5 10 8: 2 5 15 16: 21 52 67 32: 13 32 100 mdc.lustre-MDT0001-mdc-ffff95aa04f95000.batch_stats= snapshot_time: 1789026971.534467491 secs.nsecs start_time: 1789026875.353561298 secs.nsecs elapsed_time: 96.180906193 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 2 5 5 4: 1 2 7 8: 4 10 17 16: 16 41 58 32: 16 41 100 - open/close 786 (time 1789026984.23 total 10.00 last 78.58) - open/close 1600 (time 1789026994.23 total 20.01 last 81.32) - open/close 2386 (time 1789027004.25 total 30.02 last 78.51) - open/close 3154 (time 1789027014.25 total 40.03 last 76.77) - open/close 3914 (time 1789027024.26 total 50.03 last 75.95) - open/close 4687 (time 1789027034.26 total 60.04 last 77.28) - open/close 5427 (time 1789027044.26 total 70.04 last 73.99) - open/close 6217 (time 1789027054.27 total 80.04 last 78.95) - open/close 6966 (time 1789027064.27 total 90.05 last 74.87) - open/close 7671 (time 1789027074.27 total 100.05 last 70.50) - open/close 8371 (time 1789027084.27 total 110.05 last 69.98) - open/close 9081 (time 1789027094.27 total 120.05 last 70.99) - open/close 9756 (time 1789027104.28 total 130.05 last 67.49) total: 10000 open/close in 133.42 seconds: 74.95 ops/second llite.lustre-ffff95aa04f95000.statahead_batch_max=0 llite.lustre-ffff95aa04f95000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff95aa04f95000.stats=clear mdc.lustre-MDT0001-mdc-ffff95aa04f95000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m5.699s user 0m0.319s sys 0m15.681s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff95aa04f95000.statahead_batch_max=32 llite.lustre-ffff95aa04f95000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff95aa04f95000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff95aa04f95000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff95aa04f95000.stats=clear mdc.lustre-MDT0001-mdc-ffff95aa04f95000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m45.935s user 0m0.338s sys 0m14.543s batched statahead RPC= hit count=9977 mdc.lustre-MDT0000-mdc-ffff95aa04f95000.batch_stats= snapshot_time: 1789027300.023267801 secs.nsecs start_time: 1789027182.930062123 secs.nsecs elapsed_time: 117.093205678 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 1 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff95aa04f95000.batch_stats= snapshot_time: 1789027300.023345819 secs.nsecs start_time: 1789027182.930406908 secs.nsecs elapsed_time: 117.092938911 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff95aa04f95000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff95aa04f95000.batch_stats=0 llite.lustre-ffff95aa04f95000.statahead_max=2048 llite.lustre-ffff95aa04f95000.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff95aa04f95000.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff95aa04f95000.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 2.40 seconds: 41.75 ops/second llite.lustre-ffff95aa04f95000.statahead_max=0 101 real 0m2.600s user 0m0.007s sys 0m0.294s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff95aa04f95000.statahead_max=2048 llite.lustre-ffff95aa04f95000.statahead_max=2048 101 real 0m0.903s user 0m0.005s sys 0m0.189s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10076 miss_total: 24 swrong=0 ewrong=0 - open/close 694 (time 1789027611.24 total 10.01 last 69.33) total: 900 open/close in 13.67 seconds: 65.81 ops/second llite.lustre-ffff95aa04f95000.statahead_max=0 1001 real 0m22.654s user 0m0.034s sys 0m3.014s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff95aa04f95000.statahead_max=2048 llite.lustre-ffff95aa04f95000.statahead_max=2048 1001 real 0m6.716s user 0m0.030s sys 0m1.590s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff95aa04f95000.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11075 miss_total: 25 mdc.lustre-MDT0000-mdc-ffff95aa04f95000.batch_stats= snapshot_time: 1789027675.350553118 secs.nsecs start_time: 1789027302.735158508 secs.nsecs elapsed_time: 372.615394610 secs.nsecs subreqs per batch batches %% cum %% 1: 2 28 28 2: 0 0 28 4: 2 28 57 8: 1 14 71 16: 0 0 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff95aa04f95000.batch_stats= snapshot_time: 1789027675.350638134 secs.nsecs start_time: 1789027302.735290746 secs.nsecs elapsed_time: 372.615347388 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 10 10 8: 2 20 30 16: 3 30 60 32: 1 10 70 64: 1 10 80 128: 0 0 80 256: 2 20 100 - open/close 771 (time 1789027687.87 total 10.00 last 77.08) - open/close 1545 (time 1789027697.88 total 20.01 last 77.33) - open/close 2255 (time 1789027707.88 total 30.01 last 70.99) - open/close 2986 (time 1789027717.89 total 40.02 last 73.04) - open/close 3686 (time 1789027727.91 total 50.04 last 69.86) - open/close 4410 (time 1789027737.91 total 60.05 last 72.34) - open/close 5161 (time 1789027747.93 total 70.07 last 74.99) - open/close 5899 (time 1789027757.94 total 80.07 last 73.74) - open/close 6641 (time 1789027767.96 total 90.09 last 74.05) - open/close 7448 (time 1789027777.96 total 100.10 last 80.69) - open/close 8251 (time 1789027787.96 total 110.10 last 80.29) - open/close 9034 (time 1789027797.96 total 120.10 last 78.30) - open/close 9809 (time 1789027807.97 total 130.11 last 77.40) total: 10000 open/close in 132.56 seconds: 75.44 ops/second llite.lustre-ffff95aa04f95000.statahead_batch_max=0 llite.lustre-ffff95aa04f95000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff95aa04f95000.stats=clear mdc.lustre-MDT0001-mdc-ffff95aa04f95000.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.789s user 0m0.372s sys 0m17.436s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff95aa04f95000.statahead_batch_max=256 llite.lustre-ffff95aa04f95000.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff95aa04f95000.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff95aa04f95000.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff95aa04f95000.stats=clear mdc.lustre-MDT0001-mdc-ffff95aa04f95000.stats=clear ls -l files in d123ad.sanity: 10001 real 1m1.476s user 0m0.357s sys 0m16.056s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff95aa04f95000.batch_stats= snapshot_time: 1789027947.949951982 secs.nsecs start_time: 1789027876.130018874 secs.nsecs elapsed_time: 71.819933108 secs.nsecs subreqs per batch batches %% cum %% 1: 104 44 44 2: 24 10 54 4: 25 10 65 8: 15 6 71 16: 12 5 76 32: 15 6 82 64: 4 1 84 128: 1 0 85 256: 35 14 100 mdc.lustre-MDT0001-mdc-ffff95aa04f95000.batch_stats= snapshot_time: 1789027947.950028304 secs.nsecs start_time: 1789027876.130315149 secs.nsecs elapsed_time: 71.819713155 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff95aa04f95000.statahead_batch_max=64 llite.lustre-ffff95aa04f95000.statahead_max=128