== sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 00:58:51 (1788757131)
osp.lustre-MDT0000-osp-MDT0001.rpc_stats
osp.lustre-MDT0001-osp-MDT0000.rpc_stats
total: 10 mkdir in 0.53 seconds: 18.80 ops/second
 - mkdir 139 (time 1788757153.79 total 10.06 last 13.81)
 - mkdir 297 (time 1788757163.82 total 20.10 last 15.74)
 - mkdir 447 (time 1788757173.86 total 30.14 last 14.94)
 - mkdir 593 (time 1788757183.87 total 40.15 last 14.59)
 - mkdir 725 (time 1788757193.91 total 50.19 last 13.14)
 - mkdir 874 (time 1788757203.95 total 60.22 last 14.85)
total: 1000 mkdir in 68.99 seconds: 14.50 ops/second
 - mkdir 134 (time 1788757152.56 total 10.00 last 13.40)
 - mkdir 288 (time 1788757162.61 total 20.05 last 15.33)
 - mkdir 428 (time 1788757172.64 total 30.08 last 13.96)
 - mkdir 575 (time 1788757182.70 total 40.14 last 14.61)
 - mkdir 709 (time 1788757192.78 total 50.21 last 13.30)
 - mkdir 852 (time 1788757202.80 total 60.24 last 14.26)
 - mkdir 995 (time 1788757212.82 total 70.26 last 14.27)
total: 1000 mkdir in 70.84 seconds: 14.12 ops/second
 - mkdir 136 (time 1788757153.85 total 10.04 last 13.55)
 - mkdir 288 (time 1788757163.92 total 20.11 last 15.09)
 - mkdir 436 (time 1788757173.95 total 30.14 last 14.76)
 - mkdir 577 (time 1788757183.99 total 40.19 last 14.03)
 - mkdir 709 (time 1788757194.01 total 50.21 last 13.18)
 - mkdir 860 (time 1788757204.02 total 60.22 last 15.09)
 - mkdir 989 (time 1788757214.08 total 70.28 last 12.82)
total: 1000 mkdir in 71.71 seconds: 13.94 ops/second
 - mkdir 138 (time 1788757153.57 total 10.04 last 13.74)
 - mkdir 288 (time 1788757163.59 total 20.07 last 14.96)
 - mkdir 432 (time 1788757173.67 total 30.15 last 14.29)
 - mkdir 576 (time 1788757183.71 total 40.19 last 14.34)
 - mkdir 704 (time 1788757193.75 total 50.23 last 12.75)
 - mkdir 852 (time 1788757203.81 total 60.29 last 14.71)
 - mkdir 984 (time 1788757213.94 total 70.42 last 13.03)
total: 1000 mkdir in 72.59 seconds: 13.78 ops/second
 - mkdir 133 (time 1788757154.10 total 10.05 last 13.23)
 - mkdir 282 (time 1788757164.16 total 20.12 last 14.81)
 - mkdir 420 (time 1788757174.16 total 30.12 last 13.79)
 - mkdir 560 (time 1788757184.20 total 40.16 last 13.95)
 - mkdir 693 (time 1788757194.20 total 50.16 last 13.30)
 - mkdir 845 (time 1788757204.22 total 60.18 last 15.17)
 - mkdir 976 (time 1788757214.31 total 70.27 last 12.99)
total: 1000 mkdir in 73.52 seconds: 13.60 ops/second
 - mkdir 138 (time 1788757154.56 total 10.04 last 13.74)
 - mkdir 292 (time 1788757164.57 total 20.05 last 15.38)
 - mkdir 435 (time 1788757174.60 total 30.09 last 14.25)
 - mkdir 570 (time 1788757184.71 total 40.19 last 13.36)
 - mkdir 707 (time 1788757194.79 total 50.27 last 13.60)
 - mkdir 854 (time 1788757204.85 total 60.34 last 14.61)
 - mkdir 978 (time 1788757214.91 total 70.40 last 12.33)
total: 1000 mkdir in 73.32 seconds: 13.64 ops/second
 - mkdir 140 (time 1788757154.61 total 10.01 last 13.98)
 - mkdir 290 (time 1788757164.62 total 20.02 last 14.99)
 - mkdir 434 (time 1788757174.64 total 30.04 last 14.37)
 - mkdir 571 (time 1788757184.68 total 40.08 last 13.65)
 - mkdir 702 (time 1788757194.73 total 50.13 last 13.03)
 - mkdir 847 (time 1788757204.74 total 60.14 last 14.48)
 - mkdir 972 (time 1788757214.98 total 70.38 last 12.21)
total: 1000 mkdir in 73.79 seconds: 13.55 ops/second
 - mkdir 148 (time 1788757155.07 total 10.03 last 14.76)
 - mkdir 299 (time 1788757165.12 total 20.07 last 15.04)
 - mkdir 440 (time 1788757175.20 total 30.15 last 13.99)
 - mkdir 577 (time 1788757185.22 total 40.17 last 13.67)
 - mkdir 717 (time 1788757195.22 total 50.18 last 13.99)
 - mkdir 860 (time 1788757205.31 total 60.27 last 14.18)
 - mkdir 976 (time 1788757215.32 total 70.27 last 11.59)
total: 1000 mkdir in 73.75 seconds: 13.56 ops/second
 - mkdir 143 (time 1788757155.12 total 10.07 last 14.20)
 - mkdir 289 (time 1788757165.14 total 20.08 last 14.58)
 - mkdir 428 (time 1788757175.20 total 30.15 last 13.81)
 - mkdir 563 (time 1788757185.23 total 40.18 last 13.45)
 - mkdir 699 (time 1788757195.24 total 50.19 last 13.58)
 - mkdir 844 (time 1788757205.26 total 60.21 last 14.47)
 - mkdir 962 (time 1788757215.28 total 70.23 last 11.78)
total: 1000 mkdir in 74.85 seconds: 13.36 ops/second
 - mkdir 134 (time 1788757154.81 total 10.04 last 13.35)
 - mkdir 281 (time 1788757164.83 total 20.06 last 14.67)
 - mkdir 413 (time 1788757174.87 total 30.09 last 13.15)
 - mkdir 546 (time 1788757184.97 total 40.20 last 13.16)
 - mkdir 681 (time 1788757195.02 total 50.25 last 13.43)
 - mkdir 830 (time 1788757205.08 total 60.31 last 14.82)
 - mkdir 950 (time 1788757215.20 total 70.43 last 11.85)
total: 1000 mkdir in 76.07 seconds: 13.15 ops/second
Time for file creation 88 sec for 8 parallel RPCs
rmdir(/mnt/lustre/d182b.sanity/7/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/2/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/6/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/0/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/1/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/8/d-0) error: No such file or directory
total: 0 unlinks in 1 seconds: 0.000000 unlinks/second
rmdir(/mnt/lustre/d182b.sanity/9/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/4/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/3/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/5/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
Time for file removal 14 sec for 8 parallel RPCs
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=1
total: 10 mkdir in 0.35 seconds: 28.61 ops/second
 - mkdir 152 (time 1788757756.61 total 10.12 last 15.03)
 - mkdir 293 (time 1788757766.62 total 20.13 last 14.08)
 - mkdir 433 (time 1788757776.64 total 30.14 last 13.98)
 - mkdir 575 (time 1788757786.66 total 40.17 last 14.16)
 - mkdir 749 (time 1788757796.67 total 50.17 last 17.39)
 - mkdir 926 (time 1788757806.68 total 60.19 last 17.68)
total: 1000 mkdir in 64.76 seconds: 15.44 ops/second
 - mkdir 150 (time 1788757758.58 total 10.05 last 14.93)
 - mkdir 294 (time 1788757768.61 total 20.08 last 14.35)
 - mkdir 445 (time 1788757778.74 total 30.21 last 14.91)
 - mkdir 591 (time 1788757788.75 total 40.22 last 14.58)
 - mkdir 758 (time 1788757798.80 total 50.27 last 16.63)
 - mkdir 940 (time 1788757808.85 total 60.32 last 18.10)
total: 1000 mkdir in 64.72 seconds: 15.45 ops/second
 - mkdir 153 (time 1788757759.54 total 10.05 last 15.22)
 - mkdir 299 (time 1788757769.57 total 20.08 last 14.56)
 - mkdir 455 (time 1788757779.61 total 30.12 last 15.54)
 - mkdir 603 (time 1788757789.63 total 40.14 last 14.78)
 - mkdir 769 (time 1788757799.65 total 50.16 last 16.56)
 - mkdir 950 (time 1788757809.68 total 60.19 last 18.04)
total: 1000 mkdir in 64.07 seconds: 15.61 ops/second
 - mkdir 147 (time 1788757759.34 total 10.04 last 14.65)
 - mkdir 286 (time 1788757769.41 total 20.11 last 13.80)
 - mkdir 430 (time 1788757779.50 total 30.19 last 14.28)
 - mkdir 583 (time 1788757789.52 total 40.22 last 15.26)
 - mkdir 753 (time 1788757799.55 total 50.24 last 16.96)
 - mkdir 938 (time 1788757809.56 total 60.26 last 18.47)
total: 1000 mkdir in 65.71 seconds: 15.22 ops/second
 - mkdir 146 (time 1788757759.49 total 10.05 last 14.53)
 - mkdir 289 (time 1788757769.50 total 20.06 last 14.28)
 - mkdir 437 (time 1788757779.51 total 30.07 last 14.79)
 - mkdir 591 (time 1788757789.55 total 40.11 last 15.34)
 - mkdir 758 (time 1788757799.57 total 50.13 last 16.66)
 - mkdir 941 (time 1788757809.60 total 60.16 last 18.26)
total: 1000 mkdir in 65.84 seconds: 15.19 ops/second
 - mkdir 155 (time 1788757759.57 total 10.07 last 15.38)
 - mkdir 295 (time 1788757769.66 total 20.16 last 13.88)
 - mkdir 443 (time 1788757779.70 total 30.20 last 14.74)
 - mkdir 591 (time 1788757789.72 total 40.22 last 14.77)
 - mkdir 759 (time 1788757799.80 total 50.30 last 16.67)
 - mkdir 939 (time 1788757809.86 total 60.36 last 17.90)
total: 1000 mkdir in 65.99 seconds: 15.15 ops/second
 - mkdir 142 (time 1788757759.22 total 10.04 last 14.14)
 - mkdir 282 (time 1788757769.24 total 20.06 last 13.97)
 - mkdir 433 (time 1788757779.24 total 30.07 last 15.10)
 - mkdir 582 (time 1788757789.27 total 40.10 last 14.86)
 - mkdir 748 (time 1788757799.32 total 50.14 last 16.53)
 - mkdir 927 (time 1788757809.33 total 60.15 last 17.89)
total: 1000 mkdir in 66.47 seconds: 15.04 ops/second
 - mkdir 149 (time 1788757759.77 total 10.04 last 14.85)
 - mkdir 288 (time 1788757769.78 total 20.05 last 13.89)
 - mkdir 438 (time 1788757779.82 total 30.09 last 14.94)
 - mkdir 593 (time 1788757789.84 total 40.10 last 15.47)
 - mkdir 760 (time 1788757799.87 total 50.13 last 16.66)
 - mkdir 939 (time 1788757809.91 total 60.17 last 17.83)
total: 1000 mkdir in 65.99 seconds: 15.15 ops/second
 - mkdir 143 (time 1788757759.27 total 10.06 last 14.22)
 - mkdir 282 (time 1788757769.30 total 20.09 last 13.86)
 - mkdir 431 (time 1788757779.34 total 30.13 last 14.84)
 - mkdir 582 (time 1788757789.37 total 40.16 last 15.05)
 - mkdir 746 (time 1788757799.40 total 50.19 last 16.35)
 - mkdir 923 (time 1788757809.42 total 60.21 last 17.67)
total: 1000 mkdir in 67.18 seconds: 14.89 ops/second
 - mkdir 145 (time 1788757758.95 total 10.05 last 14.43)
 - mkdir 283 (time 1788757768.98 total 20.07 last 13.76)
 - mkdir 428 (time 1788757779.05 total 30.15 last 14.40)
 - mkdir 577 (time 1788757789.12 total 40.21 last 14.80)
 - mkdir 741 (time 1788757799.17 total 50.27 last 16.31)
 - mkdir 918 (time 1788757809.18 total 60.27 last 17.69)
total: 1000 mkdir in 67.55 seconds: 14.80 ops/second
Time for file creation 77 sec for 1 RPC sent at a time
 - unlinked 0 (time 1788757822 ; total 0 ; last 0)
total: 1000 unlinks in 24 seconds: 41.666668 unlinks/second
 - unlinked 0 (time 1788757823 ; total 0 ; last 0)
total: 1000 unlinks in 27 seconds: 37.037037 unlinks/second
 - unlinked 0 (time 1788757824 ; total 0 ; last 0)
total: 1000 unlinks in 28 seconds: 35.714287 unlinks/second
 - unlinked 0 (time 1788757824 ; total 0 ; last 0)
total: 1000 unlinks in 29 seconds: 34.482758 unlinks/second
 - unlinked 0 (time 1788757825 ; total 1 ; last 1)
total: 1000 unlinks in 29 seconds: 34.482758 unlinks/second
 - unlinked 0 (time 1788757824 ; total 0 ; last 0)
total: 1000 unlinks in 29 seconds: 34.482758 unlinks/second
 - unlinked 0 (time 1788757824 ; total 0 ; last 0)
total: 1000 unlinks in 30 seconds: 33.333332 unlinks/second
 - unlinked 0 (time 1788757825 ; total 0 ; last 0)
total: 1000 unlinks in 29 seconds: 34.482758 unlinks/second
 - unlinked 0 (time 1788757825 ; total 0 ; last 0)
total: 1000 unlinks in 29 seconds: 34.482758 unlinks/second
 - unlinked 0 (time 1788757824 ; total 0 ; last 0)
total: 1000 unlinks in 31 seconds: 32.258064 unlinks/second
Time for file removal 39 sec for 1 RPC sent at a time
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=8