== sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 10:17:32 (1788531452)
osp.lustre-MDT0000-osp-MDT0001.rpc_stats
osp.lustre-MDT0001-osp-MDT0000.rpc_stats
total: 10 mkdir in 0.48 seconds: 20.93 ops/second
 - mkdir 147 (time 1788531471.59 total 10.11 last 14.53)
 - mkdir 297 (time 1788531481.68 total 20.20 last 14.87)
 - mkdir 450 (time 1788531491.71 total 30.24 last 15.25)
 - mkdir 605 (time 1788531501.80 total 40.32 last 15.37)
 - mkdir 752 (time 1788531511.81 total 50.33 last 14.69)
 - mkdir 906 (time 1788531521.90 total 60.42 last 15.26)
total: 1000 mkdir in 66.53 seconds: 15.03 ops/second
 - mkdir 140 (time 1788531472.00 total 10.07 last 13.91)
 - mkdir 288 (time 1788531482.02 total 20.08 last 14.77)
 - mkdir 438 (time 1788531492.06 total 30.12 last 14.95)
 - mkdir 588 (time 1788531502.13 total 40.19 last 14.89)
 - mkdir 739 (time 1788531512.15 total 50.21 last 15.07)
 - mkdir 893 (time 1788531522.16 total 60.22 last 15.39)
total: 1000 mkdir in 68.66 seconds: 14.56 ops/second
 - mkdir 148 (time 1788531473.92 total 10.03 last 14.75)
 - mkdir 300 (time 1788531483.99 total 20.10 last 15.10)
 - mkdir 447 (time 1788531494.00 total 30.10 last 14.69)
 - mkdir 607 (time 1788531504.02 total 40.13 last 15.96)
 - mkdir 749 (time 1788531514.07 total 50.18 last 14.13)
 - mkdir 905 (time 1788531524.07 total 60.18 last 15.60)
total: 1000 mkdir in 68.95 seconds: 14.50 ops/second
 - mkdir 142 (time 1788531474.43 total 10.02 last 14.17)
 - mkdir 295 (time 1788531484.48 total 20.07 last 15.23)
 - mkdir 448 (time 1788531494.50 total 30.09 last 15.27)
 - mkdir 604 (time 1788531504.50 total 40.09 last 15.59)
 - mkdir 751 (time 1788531514.51 total 50.10 last 14.68)
 - mkdir 902 (time 1788531524.59 total 60.18 last 14.99)
total: 1000 mkdir in 70.12 seconds: 14.26 ops/second
 - mkdir 142 (time 1788531472.96 total 10.04 last 14.14)
 - mkdir 300 (time 1788531482.98 total 20.06 last 15.77)
 - mkdir 442 (time 1788531493.01 total 30.09 last 14.16)
 - mkdir 590 (time 1788531503.01 total 40.09 last 14.80)
 - mkdir 736 (time 1788531513.11 total 50.18 last 14.46)
 - mkdir 884 (time 1788531523.13 total 60.21 last 14.76)
 - mkdir 990 (time 1788531533.28 total 70.36 last 10.44)
total: 1000 mkdir in 71.72 seconds: 13.94 ops/second
 - mkdir 147 (time 1788531474.19 total 10.08 last 14.59)
 - mkdir 300 (time 1788531484.23 total 20.12 last 15.24)
 - mkdir 452 (time 1788531494.27 total 30.16 last 15.14)
 - mkdir 605 (time 1788531504.33 total 40.22 last 15.21)
 - mkdir 747 (time 1788531514.34 total 50.22 last 14.19)
 - mkdir 900 (time 1788531524.37 total 60.25 last 15.25)
 - mkdir 996 (time 1788531534.40 total 70.29 last 9.57)
total: 1000 mkdir in 70.54 seconds: 14.18 ops/second
 - mkdir 139 (time 1788531473.04 total 10.02 last 13.87)
 - mkdir 288 (time 1788531483.05 total 20.03 last 14.88)
 - mkdir 424 (time 1788531493.08 total 30.06 last 13.56)
 - mkdir 579 (time 1788531503.09 total 40.07 last 15.49)
 - mkdir 725 (time 1788531513.13 total 50.11 last 14.53)
 - mkdir 878 (time 1788531523.18 total 60.16 last 15.24)
 - mkdir 983 (time 1788531533.28 total 70.26 last 10.39)
total: 1000 mkdir in 72.36 seconds: 13.82 ops/second
 - mkdir 146 (time 1788531474.59 total 10.02 last 14.58)
 - mkdir 295 (time 1788531484.62 total 20.06 last 14.84)
 - mkdir 446 (time 1788531494.65 total 30.08 last 15.07)
 - mkdir 601 (time 1788531504.72 total 40.15 last 15.39)
 - mkdir 747 (time 1788531514.76 total 50.19 last 14.54)
 - mkdir 894 (time 1788531524.76 total 60.20 last 14.69)
 - mkdir 991 (time 1788531534.79 total 70.22 last 9.68)
total: 1000 mkdir in 70.95 seconds: 14.09 ops/second
 - mkdir 138 (time 1788531474.07 total 10.02 last 13.77)
 - mkdir 286 (time 1788531484.10 total 20.05 last 14.76)
 - mkdir 432 (time 1788531494.12 total 30.07 last 14.57)
 - mkdir 584 (time 1788531504.12 total 40.07 last 15.19)
 - mkdir 731 (time 1788531514.16 total 50.11 last 14.65)
 - mkdir 885 (time 1788531524.20 total 60.15 last 15.34)
 - mkdir 983 (time 1788531534.31 total 70.26 last 9.70)
total: 1000 mkdir in 71.96 seconds: 13.90 ops/second
 - mkdir 133 (time 1788531474.13 total 10.03 last 13.27)
 - mkdir 277 (time 1788531484.14 total 20.04 last 14.38)
 - mkdir 419 (time 1788531494.15 total 30.05 last 14.19)
 - mkdir 572 (time 1788531504.19 total 40.09 last 15.24)
 - mkdir 714 (time 1788531514.20 total 50.10 last 14.19)
 - mkdir 867 (time 1788531524.21 total 60.11 last 15.29)
 - mkdir 966 (time 1788531534.26 total 70.16 last 9.85)
total: 1000 mkdir in 73.15 seconds: 13.67 ops/second
Time for file creation 83 sec for 8 parallel RPCs
rmdir(/mnt/lustre/d182b.sanity/1/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/0/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/2/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/6/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/7/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/8/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/3/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/4/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/5/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/9/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
Time for file removal 12 sec for 8 parallel RPCs
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=1
total: 10 mkdir in 0.82 seconds: 12.23 ops/second
 - mkdir 96 (time 1788532093.00 total 10.00 last 9.60)
 - mkdir 231 (time 1788532103.01 total 20.01 last 13.49)
 - mkdir 360 (time 1788532113.07 total 30.07 last 12.82)
 - mkdir 501 (time 1788532123.09 total 40.10 last 14.07)
 - mkdir 647 (time 1788532133.10 total 50.11 last 14.58)
 - mkdir 783 (time 1788532143.13 total 60.13 last 13.57)
 - mkdir 917 (time 1788532153.20 total 70.20 last 13.31)
total: 1000 mkdir in 75.90 seconds: 13.18 ops/second
 - mkdir 121 (time 1788532096.50 total 10.03 last 12.07)
 - mkdir 253 (time 1788532106.50 total 20.03 last 13.19)
 - mkdir 381 (time 1788532116.53 total 30.06 last 12.77)
 - mkdir 535 (time 1788532126.58 total 40.11 last 15.33)
 - mkdir 676 (time 1788532136.60 total 50.13 last 14.07)
 - mkdir 820 (time 1788532146.64 total 60.17 last 14.34)
 - mkdir 958 (time 1788532156.71 total 70.24 last 13.71)
total: 1000 mkdir in 73.88 seconds: 13.54 ops/second
 - mkdir 131 (time 1788532098.04 total 10.04 last 13.05)
 - mkdir 263 (time 1788532108.05 total 20.06 last 13.18)
 - mkdir 396 (time 1788532118.08 total 30.09 last 13.26)
 - mkdir 545 (time 1788532128.11 total 40.11 last 14.86)
 - mkdir 686 (time 1788532138.20 total 50.20 last 13.97)
 - mkdir 826 (time 1788532148.27 total 60.27 last 13.91)
 - mkdir 965 (time 1788532158.28 total 70.29 last 13.88)
total: 1000 mkdir in 73.81 seconds: 13.55 ops/second
 - mkdir 126 (time 1788532097.19 total 10.04 last 12.55)
 - mkdir 255 (time 1788532107.23 total 20.08 last 12.85)
 - mkdir 387 (time 1788532117.26 total 30.11 last 13.16)
 - mkdir 535 (time 1788532127.34 total 40.18 last 14.69)
 - mkdir 675 (time 1788532137.34 total 50.19 last 13.99)
 - mkdir 812 (time 1788532147.36 total 60.21 last 13.67)
 - mkdir 951 (time 1788532157.37 total 70.22 last 13.88)
total: 1000 mkdir in 75.01 seconds: 13.33 ops/second
 - mkdir 136 (time 1788532098.62 total 10.01 last 13.59)
 - mkdir 271 (time 1788532108.69 total 20.08 last 13.40)
 - mkdir 399 (time 1788532118.73 total 30.12 last 12.75)
 - mkdir 548 (time 1788532128.79 total 40.18 last 14.81)
 - mkdir 683 (time 1788532138.85 total 50.24 last 13.43)
 - mkdir 826 (time 1788532148.88 total 60.27 last 14.25)
 - mkdir 971 (time 1788532158.93 total 70.32 last 14.43)
total: 1000 mkdir in 73.88 seconds: 13.54 ops/second
 - mkdir 118 (time 1788532096.96 total 10.03 last 11.77)
 - mkdir 249 (time 1788532107.05 total 20.12 last 12.99)
 - mkdir 381 (time 1788532117.08 total 30.15 last 13.16)
 - mkdir 529 (time 1788532127.13 total 40.20 last 14.73)
 - mkdir 673 (time 1788532137.17 total 50.24 last 14.34)
 - mkdir 811 (time 1788532147.19 total 60.25 last 13.78)
 - mkdir 943 (time 1788532157.20 total 70.27 last 13.18)
total: 1000 mkdir in 76.08 seconds: 13.14 ops/second
 - mkdir 122 (time 1788532097.97 total 10.05 last 12.15)
 - mkdir 248 (time 1788532108.02 total 20.09 last 12.55)
 - mkdir 377 (time 1788532118.06 total 30.13 last 12.85)
 - mkdir 523 (time 1788532128.10 total 40.17 last 14.54)
 - mkdir 668 (time 1788532138.16 total 50.23 last 14.41)
 - mkdir 811 (time 1788532148.22 total 60.29 last 14.22)
 - mkdir 945 (time 1788532158.23 total 70.30 last 13.38)
total: 1000 mkdir in 76.80 seconds: 13.02 ops/second
 - mkdir 120 (time 1788532097.51 total 10.03 last 11.97)
 - mkdir 245 (time 1788532107.54 total 20.06 last 12.45)
 - mkdir 371 (time 1788532117.57 total 30.09 last 12.57)
 - mkdir 520 (time 1788532127.59 total 40.11 last 14.87)
 - mkdir 661 (time 1788532137.61 total 50.13 last 14.07)
 - mkdir 801 (time 1788532147.66 total 60.18 last 13.93)
 - mkdir 937 (time 1788532157.70 total 70.22 last 13.55)
total: 1000 mkdir in 77.63 seconds: 12.88 ops/second
 - mkdir 106 (time 1788532096.59 total 10.02 last 10.58)
 - mkdir 232 (time 1788532106.69 total 20.11 last 12.48)
 - mkdir 352 (time 1788532116.77 total 30.19 last 11.90)
 - mkdir 505 (time 1788532126.80 total 40.22 last 15.26)
 - mkdir 647 (time 1788532136.85 total 50.27 last 14.13)
 - mkdir 791 (time 1788532146.93 total 60.35 last 14.29)
 - mkdir 916 (time 1788532157.01 total 70.44 last 12.39)
total: 1000 mkdir in 79.67 seconds: 12.55 ops/second
 - mkdir 126 (time 1788532098.45 total 10.04 last 12.55)
 - mkdir 257 (time 1788532108.48 total 20.07 last 13.05)
 - mkdir 386 (time 1788532118.49 total 30.08 last 12.89)
 - mkdir 532 (time 1788532128.57 total 40.16 last 14.49)
 - mkdir 663 (time 1788532138.65 total 50.24 last 13.00)
 - mkdir 798 (time 1788532148.69 total 60.28 last 13.44)
 - mkdir 934 (time 1788532158.73 total 70.32 last 13.55)
total: 1000 mkdir in 78.19 seconds: 12.79 ops/second
Time for file creation 96 sec for 1 RPC sent at a time
 - unlinked 0 (time 1788532174 ; total 0 ; last 0)
total: 1000 unlinks in 33 seconds: 30.303030 unlinks/second
 - unlinked 0 (time 1788532176 ; total 1 ; last 1)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
 - unlinked 0 (time 1788532177 ; total 0 ; last 0)
total: 1000 unlinks in 32 seconds: 31.250000 unlinks/second
 - unlinked 0 (time 1788532176 ; total 0 ; last 0)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
 - unlinked 0 (time 1788532177 ; total 1 ; last 1)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
 - unlinked 0 (time 1788532177 ; total 0 ; last 0)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
 - unlinked 0 (time 1788532177 ; total 0 ; last 0)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
 - unlinked 0 (time 1788532177 ; total 0 ; last 0)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
 - unlinked 0 (time 1788532177 ; total 0 ; last 0)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
 - unlinked 0 (time 1788532177 ; total 0 ; last 0)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
Time for file removal 47 sec for 1 RPC sent at a time
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=8