== sanity test 123ad: Verify batching statahead works correctly ========================================================== 05:13:53 (1788599633) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff8e5f82cf9800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8e5f82cf9800.batch_stats=0 llite.lustre-ffff8e5f82cf9800.statahead_max=32 llite.lustre-ffff8e5f82cf9800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff8e5f82cf9800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8e5f82cf9800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 2.63 seconds: 38.05 ops/second llite.lustre-ffff8e5f82cf9800.statahead_max=0 101 real 0m2.493s user 0m0.006s sys 0m0.297s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8e5f82cf9800.statahead_max=32 llite.lustre-ffff8e5f82cf9800.statahead_max=32 101 real 0m1.178s user 0m0.006s sys 0m0.182s 'ls -l' 100 files with statahead: 1 sec statahead total: 143 statahead wrong: 2 agl total: 137 list_total: 30 fname_total: 96 hit_total: 3657 miss_total: 1637 swrong=2 ewrong=2 - open/close 719 (time 1788599661.80 total 10.00 last 71.87) total: 900 open/close in 12.79 seconds: 70.38 ops/second llite.lustre-ffff8e5f82cf9800.statahead_max=0 1001 real 0m23.218s user 0m0.039s sys 0m3.287s 'ls -l' 1000 files without statahead: 23 sec llite.lustre-ffff8e5f82cf9800.statahead_max=32 llite.lustre-ffff8e5f82cf9800.statahead_max=32 1001 real 0m10.669s user 0m0.035s sys 0m1.475s 'ls -l' 1000 files with statahead: 11 sec statahead total: 144 statahead wrong: 2 agl total: 138 list_total: 31 fname_total: 96 hit_total: 4655 miss_total: 1639 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 24 seconds llite.lustre-ffff8e5f82cf9800.statahead_stats= statahead total: 144 statahead wrong: 2 agl total: 138 list_total: 31 fname_total: 96 hit_total: 4655 miss_total: 1639 mdc.lustre-MDT0000-mdc-ffff8e5f82cf9800.batch_stats= snapshot_time: 1788599736.975883043 secs.nsecs start_time: 1788599638.416066170 secs.nsecs elapsed_time: 98.559816873 secs.nsecs subreqs per batch batches %% cum %% 1: 4 10 10 2: 2 5 15 4: 1 2 18 8: 1 2 21 16: 22 57 78 32: 8 21 100 mdc.lustre-MDT0001-mdc-ffff8e5f82cf9800.batch_stats= snapshot_time: 1788599736.976193559 secs.nsecs start_time: 1788599638.416725848 secs.nsecs elapsed_time: 98.559467711 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 1 2 5 8: 4 10 15 16: 13 32 47 32: 21 52 100 - open/close 731 (time 1788599750.44 total 10.00 last 73.08) - open/close 1463 (time 1788599760.44 total 20.01 last 73.17) - open/close 2088 (time 1788599770.45 total 30.02 last 62.43) - open/close 2745 (time 1788599780.46 total 40.02 last 65.67) - open/close 3405 (time 1788599790.46 total 50.03 last 65.96) - open/close 4018 (time 1788599800.48 total 60.04 last 61.21) - open/close 4619 (time 1788599810.48 total 70.04 last 60.10) - open/close 5291 (time 1788599820.48 total 80.04 last 67.19) - open/close 5905 (time 1788599830.48 total 90.05 last 61.38) - open/close 6493 (time 1788599840.50 total 100.07 last 58.68) - open/close 7125 (time 1788599850.50 total 110.07 last 63.18) - open/close 7666 (time 1788599860.51 total 120.07 last 54.08) - open/close 8270 (time 1788599870.51 total 130.08 last 60.38) - open/close 8930 (time 1788599880.51 total 140.08 last 65.99) - open/close 9540 (time 1788599890.52 total 150.08 last 60.96) total: 10000 open/close in 156.40 seconds: 63.94 ops/second llite.lustre-ffff8e5f82cf9800.statahead_batch_max=0 llite.lustre-ffff8e5f82cf9800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8e5f82cf9800.stats=clear mdc.lustre-MDT0001-mdc-ffff8e5f82cf9800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m12.548s user 0m0.361s sys 0m15.062s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8e5f82cf9800.statahead_batch_max=32 llite.lustre-ffff8e5f82cf9800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8e5f82cf9800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8e5f82cf9800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8e5f82cf9800.stats=clear mdc.lustre-MDT0001-mdc-ffff8e5f82cf9800.stats=clear ls -l files in d123ad.sanity: 10001 real 2m1.568s user 0m0.341s sys 0m14.695s batched statahead RPC= hit count=9986 mdc.lustre-MDT0000-mdc-ffff8e5f82cf9800.batch_stats= snapshot_time: 1788600115.510036406 secs.nsecs start_time: 1788599979.989514418 secs.nsecs elapsed_time: 135.520521988 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff8e5f82cf9800.batch_stats= snapshot_time: 1788600115.510308024 secs.nsecs start_time: 1788599979.990006441 secs.nsecs elapsed_time: 135.520301583 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff8e5f82cf9800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8e5f82cf9800.batch_stats=0 llite.lustre-ffff8e5f82cf9800.statahead_max=2048 llite.lustre-ffff8e5f82cf9800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff8e5f82cf9800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff8e5f82cf9800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 3.10 seconds: 32.24 ops/second llite.lustre-ffff8e5f82cf9800.statahead_max=0 101 real 0m2.548s user 0m0.005s sys 0m0.355s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff8e5f82cf9800.statahead_max=2048 llite.lustre-ffff8e5f82cf9800.statahead_max=2048 101 real 0m0.974s user 0m0.004s sys 0m0.185s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10085 miss_total: 15 swrong=0 ewrong=0 - open/close 609 (time 1788600455.37 total 10.00 last 60.88) total: 900 open/close in 14.73 seconds: 61.09 ops/second llite.lustre-ffff8e5f82cf9800.statahead_max=0 1001 real 0m28.264s user 0m0.042s sys 0m3.191s 'ls -l' 1000 files without statahead: 28 sec llite.lustre-ffff8e5f82cf9800.statahead_max=2048 llite.lustre-ffff8e5f82cf9800.statahead_max=2048 1001 real 0m8.600s user 0m0.033s sys 0m1.354s 'ls -l' 1000 files with statahead: 9 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11084 miss_total: 16 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 25 seconds llite.lustre-ffff8e5f82cf9800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11084 miss_total: 16 mdc.lustre-MDT0000-mdc-ffff8e5f82cf9800.batch_stats= snapshot_time: 1788600536.957401368 secs.nsecs start_time: 1788600117.815819784 secs.nsecs elapsed_time: 419.141581584 secs.nsecs subreqs per batch batches %% cum %% 1: 2 25 25 2: 2 25 50 4: 1 12 62 8: 0 0 62 16: 1 12 75 32: 0 0 75 64: 0 0 75 128: 0 0 75 256: 2 25 100 mdc.lustre-MDT0001-mdc-ffff8e5f82cf9800.batch_stats= snapshot_time: 1788600536.957498261 secs.nsecs start_time: 1788600117.815976274 secs.nsecs elapsed_time: 419.141521987 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 4 44 44 16: 0 0 44 32: 1 11 55 64: 2 22 77 128: 0 0 77 256: 2 22 100 - open/close 682 (time 1788600550.38 total 10.02 last 68.05) - open/close 1403 (time 1788600560.38 total 20.03 last 72.07) - open/close 2023 (time 1788600570.40 total 30.05 last 61.87) - open/close 2623 (time 1788600580.41 total 40.05 last 59.97) - open/close 3364 (time 1788600590.41 total 50.05 last 74.10) - open/close 4020 (time 1788600600.41 total 60.06 last 65.57) - open/close 4579 (time 1788600610.42 total 70.07 last 55.85) - open/close 5202 (time 1788600620.43 total 80.07 last 62.24) - open/close 5855 (time 1788600630.45 total 90.10 last 65.14) - open/close 6504 (time 1788600640.48 total 100.12 last 64.75) - open/close 7221 (time 1788600650.48 total 110.13 last 71.66) - open/close 7941 (time 1788600660.49 total 120.14 last 71.92) - open/close 8600 (time 1788600670.50 total 130.14 last 65.87) - open/close 9302 (time 1788600680.51 total 140.15 last 70.12) - open/close 9937 (time 1788600690.51 total 150.16 last 63.47) total: 10000 open/close in 151.13 seconds: 66.17 ops/second llite.lustre-ffff8e5f82cf9800.statahead_batch_max=0 llite.lustre-ffff8e5f82cf9800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8e5f82cf9800.stats=clear mdc.lustre-MDT0001-mdc-ffff8e5f82cf9800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m9.609s user 0m0.342s sys 0m14.913s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff8e5f82cf9800.statahead_batch_max=256 llite.lustre-ffff8e5f82cf9800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff8e5f82cf9800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff8e5f82cf9800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff8e5f82cf9800.stats=clear mdc.lustre-MDT0001-mdc-ffff8e5f82cf9800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m56.218s user 0m0.343s sys 0m15.154s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff8e5f82cf9800.batch_stats= snapshot_time: 1788600840.063586135 secs.nsecs start_time: 1788600771.718162386 secs.nsecs elapsed_time: 68.345423749 secs.nsecs subreqs per batch batches %% cum %% 1: 143 47 47 2: 34 11 58 4: 33 10 69 8: 23 7 77 16: 10 3 80 32: 19 6 87 64: 4 1 88 128: 1 0 88 256: 34 11 100 mdc.lustre-MDT0001-mdc-ffff8e5f82cf9800.batch_stats= snapshot_time: 1788600840.063731154 secs.nsecs start_time: 1788600771.718241315 secs.nsecs elapsed_time: 68.345489839 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff8e5f82cf9800.statahead_batch_max=64 llite.lustre-ffff8e5f82cf9800.statahead_max=128