== sanity test 123ad: Verify batching statahead works correctly ========================================================== 01:53:50 (1786946030) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff9745463a9800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9745463a9800.batch_stats=0 llite.lustre-ffff9745463a9800.statahead_max=32 llite.lustre-ffff9745463a9800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff9745463a9800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9745463a9800.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.22 seconds: 45.06 ops/second llite.lustre-ffff9745463a9800.statahead_max=0 101 real 0m2.391s user 0m0.005s sys 0m0.364s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff9745463a9800.statahead_max=32 llite.lustre-ffff9745463a9800.statahead_max=32 101 real 0m0.948s user 0m0.006s sys 0m0.213s 'ls -l' 100 files with statahead: 1 sec statahead total: 111 statahead wrong: 2 agl total: 106 list_total: 31 fname_total: 78 hit_total: 4203 miss_total: 1291 swrong=2 ewrong=2 - open/close 800 (time 1786946058.75 total 10.01 last 79.95) total: 900 open/close in 11.81 seconds: 76.18 ops/second llite.lustre-ffff9745463a9800.statahead_max=0 1001 real 0m23.836s user 0m0.033s sys 0m3.335s 'ls -l' 1000 files without statahead: 24 sec llite.lustre-ffff9745463a9800.statahead_max=32 llite.lustre-ffff9745463a9800.statahead_max=32 1001 real 0m9.989s user 0m0.033s sys 0m1.564s 'ls -l' 1000 files with statahead: 10 sec statahead total: 112 statahead wrong: 2 agl total: 107 list_total: 32 fname_total: 78 hit_total: 5199 miss_total: 1295 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 18 seconds llite.lustre-ffff9745463a9800.statahead_stats= statahead total: 112 statahead wrong: 2 agl total: 107 list_total: 32 fname_total: 78 hit_total: 5199 miss_total: 1295 mdc.lustre-MDT0000-mdc-ffff9745463a9800.batch_stats= snapshot_time: 1786946126.438399589 secs.nsecs start_time: 1786946036.228394497 secs.nsecs elapsed_time: 90.210005092 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 2 5 7 8: 4 10 17 16: 19 47 65 32: 14 35 100 mdc.lustre-MDT0001-mdc-ffff9745463a9800.batch_stats= snapshot_time: 1786946126.438457572 secs.nsecs start_time: 1786946036.228707703 secs.nsecs elapsed_time: 90.209749869 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 2 5 7 4: 1 2 10 8: 3 7 17 16: 19 47 65 32: 14 35 100 - open/close 926 (time 1786946139.63 total 10.01 last 92.54) - open/close 1689 (time 1786946149.63 total 20.01 last 76.30) - open/close 2693 (time 1786946159.64 total 30.01 last 100.32) - open/close 3740 (time 1786946169.64 total 40.02 last 104.66) - open/close 4718 (time 1786946179.64 total 50.02 last 97.77) - open/close 5712 (time 1786946189.65 total 60.03 last 99.32) - open/close 6680 (time 1786946199.66 total 70.03 last 96.76) - open/close 7694 (time 1786946209.66 total 80.04 last 101.32) - open/close 8779 (time 1786946219.67 total 90.05 last 108.39) - open/close 9762 (time 1786946229.68 total 100.05 last 98.28) total: 10000 open/close in 102.14 seconds: 97.90 ops/second llite.lustre-ffff9745463a9800.statahead_batch_max=0 llite.lustre-ffff9745463a9800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9745463a9800.stats=clear mdc.lustre-MDT0001-mdc-ffff9745463a9800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m43.538s user 0m0.291s sys 0m19.116s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9745463a9800.statahead_batch_max=32 llite.lustre-ffff9745463a9800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9745463a9800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9745463a9800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9745463a9800.stats=clear mdc.lustre-MDT0001-mdc-ffff9745463a9800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m3.545s user 0m0.266s sys 0m15.854s batched statahead RPC= hit count=9987 mdc.lustre-MDT0000-mdc-ffff9745463a9800.batch_stats= snapshot_time: 1786946354.704051712 secs.nsecs start_time: 1786946281.615286309 secs.nsecs elapsed_time: 73.088765403 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 1 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff9745463a9800.batch_stats= snapshot_time: 1786946354.704119145 secs.nsecs start_time: 1786946281.615397202 secs.nsecs elapsed_time: 73.088721943 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff9745463a9800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9745463a9800.batch_stats=0 llite.lustre-ffff9745463a9800.statahead_max=2048 llite.lustre-ffff9745463a9800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff9745463a9800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff9745463a9800.batch_stats=0 striped dir -i1 -c2 -H crush /mnt/lustre/d123ad.sanity total: 100 open/close in 1.42 seconds: 70.55 ops/second llite.lustre-ffff9745463a9800.statahead_max=0 101 real 0m1.524s user 0m0.005s sys 0m0.318s 'ls -l' 100 files without statahead: 1 sec llite.lustre-ffff9745463a9800.statahead_max=2048 llite.lustre-ffff9745463a9800.statahead_max=2048 101 real 0m0.547s user 0m0.003s sys 0m0.211s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10086 miss_total: 14 swrong=0 ewrong=0 total: 900 open/close in 6.13 seconds: 146.94 ops/second llite.lustre-ffff9745463a9800.statahead_max=0 1001 real 0m11.489s user 0m0.030s sys 0m2.888s 'ls -l' 1000 files without statahead: 12 sec llite.lustre-ffff9745463a9800.statahead_max=2048 llite.lustre-ffff9745463a9800.statahead_max=2048 1001 real 0m3.534s user 0m0.017s sys 0m1.758s 'ls -l' 1000 files with statahead: 3 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 9 seconds llite.lustre-ffff9745463a9800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11085 miss_total: 15 mdc.lustre-MDT0000-mdc-ffff9745463a9800.batch_stats= snapshot_time: 1786946550.737722547 secs.nsecs start_time: 1786946356.268132240 secs.nsecs elapsed_time: 194.469590307 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 16 16 8: 3 50 66 16: 0 0 66 32: 0 0 66 64: 0 0 66 128: 0 0 66 256: 2 33 100 mdc.lustre-MDT0001-mdc-ffff9745463a9800.batch_stats= snapshot_time: 1786946550.737793554 secs.nsecs start_time: 1786946356.268290556 secs.nsecs elapsed_time: 194.469502998 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 1 11 11 8: 2 22 33 16: 2 22 55 32: 1 11 66 64: 0 0 66 128: 1 11 77 256: 2 22 100 - open/close 1913 (time 1786946562.21 total 10.00 last 191.29) - open/close 3639 (time 1786946572.22 total 20.00 last 172.57) - open/close 5785 (time 1786946582.22 total 30.00 last 214.57) - open/close 7805 (time 1786946592.22 total 40.01 last 201.96) total: 10000 open/close in 49.25 seconds: 203.04 ops/second llite.lustre-ffff9745463a9800.statahead_batch_max=0 llite.lustre-ffff9745463a9800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9745463a9800.stats=clear mdc.lustre-MDT0001-mdc-ffff9745463a9800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m33.390s user 0m0.267s sys 0m25.949s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff9745463a9800.statahead_batch_max=256 llite.lustre-ffff9745463a9800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff9745463a9800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff9745463a9800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff9745463a9800.stats=clear mdc.lustre-MDT0001-mdc-ffff9745463a9800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m29.633s user 0m0.229s sys 0m20.126s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff9745463a9800.batch_stats= snapshot_time: 1786946674.916229311 secs.nsecs start_time: 1786946638.269229872 secs.nsecs elapsed_time: 36.646999439 secs.nsecs subreqs per batch batches %% cum %% 1: 395 65 65 2: 74 12 77 4: 54 8 86 8: 20 3 89 16: 6 0 90 32: 17 2 93 64: 5 0 94 128: 0 0 94 256: 33 5 100 mdc.lustre-MDT0001-mdc-ffff9745463a9800.batch_stats= snapshot_time: 1786946674.916299882 secs.nsecs start_time: 1786946638.269376166 secs.nsecs elapsed_time: 36.646923716 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff9745463a9800.statahead_batch_max=64 llite.lustre-ffff9745463a9800.statahead_max=128