== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:27:24 (1790742444) start mds service on oleg323-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1947032 Sep 29 13:30 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: kmemleak disabled oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: detected 4 online CPUs by sysfs oleg323-server: MODOPTS_LIBCFS= oleg323-server: Force libcfs to create 2 CPU partitions oleg323-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg323-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg323-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg323-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg323-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 209 (time 1790742528.79 total 10.31 last 20.28) - open/close 460 (time 1790742539.39 total 20.91 last 23.68) - open/close 746 (time 1790742549.40 total 30.92 last 28.57) - open/close 996 (time 1790742560.46 total 41.98 last 22.60) - open/close 1220 (time 1790742570.56 total 52.08 last 22.18) - open/close 1429 (time 1790742580.57 total 62.09 last 20.87) - open/close 1660 (time 1790742590.58 total 72.10 last 23.09) - open/close 1906 (time 1790742603.65 total 85.17 last 18.82) - open/close 2115 (time 1790742613.68 total 95.19 last 20.84) - open/close 2422 (time 1790742623.68 total 105.20 last 30.69) - open/close 2576 (time 1790742634.07 total 115.59 last 14.82) - open/close 2785 (time 1790742644.07 total 125.59 last 20.89) - open/close 3018 (time 1790742654.66 total 136.17 last 22.02) - open/close 3279 (time 1790742665.23 total 146.74 last 24.69) - open/close 3539 (time 1790742675.42 total 156.93 last 25.51) - open/close 3799 (time 1790742688.19 total 169.71 last 20.35) - open/close 4033 (time 1790742699.23 total 180.74 last 21.21) - open/close 4261 (time 1790742710.04 total 191.56 last 21.08) - open/close 4484 (time 1790742720.05 total 201.56 last 22.28) - open/close 4718 (time 1790742730.06 total 211.58 last 23.37) - open/close 4998 (time 1790742740.07 total 221.58 last 27.98) - open/close 5227 (time 1790742750.30 total 231.81 last 22.38) - open/close 5481 (time 1790742760.58 total 242.10 last 24.69) - open/close 5744 (time 1790742771.15 total 252.67 last 24.89) - open/close 5970 (time 1790742781.32 total 262.84 last 22.22) - open/close 6208 (time 1790742792.10 total 273.62 last 22.08) - open/close 6480 (time 1790742802.13 total 283.64 last 27.13) - open/close 6760 (time 1790742812.15 total 293.66 last 27.95) - open/close 6941 (time 1790742822.55 total 304.06 last 17.40) - open/close 7195 (time 1790742832.57 total 314.09 last 25.34) - open/close 7409 (time 1790742842.82 total 324.34 last 20.87) - open/close 7605 (time 1790742852.84 total 334.36 last 19.57) - open/close 7744 (time 1790742863.88 total 345.40 last 12.59) - open/close 7924 (time 1790742874.44 total 355.96 last 17.04) - open/close 8160 (time 1790742884.46 total 365.98 last 23.56) - open/close 8363 (time 1790742894.47 total 375.99 last 20.28) - open/close 8634 (time 1790742905.35 total 386.87 last 24.90) - open/close 8905 (time 1790742915.79 total 397.30 last 25.98) - open/close 9146 (time 1790742925.96 total 407.48 last 23.68) - open/close 9406 (time 1790742936.39 total 417.91 last 24.93) - open/close 9687 (time 1790742947.21 total 428.73 last 25.97) - open/close 9961 (time 1790742957.84 total 439.36 last 25.77) - open/close 10187 (time 1790742967.89 total 449.41 last 22.49) - open/close 10496 (time 1790742977.90 total 459.41 last 30.88) - open/close 10721 (time 1790742987.91 total 469.42 last 22.48) - open/close 10975 (time 1790742998.56 total 480.08 last 23.84) - open/close 11284 (time 1790743008.87 total 490.39 last 29.98) - open/close 11585 (time 1790743019.40 total 500.92 last 28.58) - open/close 11724 (time 1790743030.10 total 511.61 last 13.00) - open/close 11923 (time 1790743040.12 total 521.64 last 19.85) - open/close 12206 (time 1790743050.13 total 531.65 last 28.26) - open/close 12467 (time 1790743061.06 total 542.57 last 23.89) - open/close 12658 (time 1790743071.89 total 553.41 last 17.63) - open/close 12917 (time 1790743081.89 total 563.41 last 25.90) - open/close 13055 (time 1790743091.92 total 573.43 last 13.76) - open/close 13318 (time 1790743102.70 total 584.22 last 24.38) - open/close 13585 (time 1790743112.71 total 594.23 last 26.68) - open/close 13836 (time 1790743123.53 total 605.04 last 23.20) - open/close 14115 (time 1790743133.54 total 615.05 last 27.88) - open/close 14397 (time 1790743143.54 total 625.05 last 28.20) - open/close 14595 (time 1790743154.12 total 635.64 last 18.70) - open/close 14880 (time 1790743164.12 total 645.64 last 28.50) - open/close 15114 (time 1790743174.13 total 655.65 last 23.38) - open/close 15407 (time 1790743184.14 total 665.65 last 29.29) - open/close 15653 (time 1790743194.14 total 675.66 last 24.58) - open/close 15822 (time 1790743204.29 total 685.81 last 16.66) - open/close 15975 (time 1790743214.43 total 695.95 last 15.09) - open/close 16199 (time 1790743225.04 total 706.56 last 21.11) - open/close 16455 (time 1790743235.05 total 716.56 last 25.58) - open/close 16707 (time 1790743245.85 total 727.37 last 23.33) - open/close 16904 (time 1790743256.83 total 738.35 last 17.94) - open/close 17097 (time 1790743266.95 total 748.47 last 19.08) - open/close 17356 (time 1790743277.07 total 758.59 last 25.59) - open/close 17614 (time 1790743288.72 total 770.23 last 22.15) - open/close 17784 (time 1790743298.84 total 780.36 last 16.79) - open/close 18038 (time 1790743309.50 total 791.02 last 23.83) - open/close 18263 (time 1790743319.62 total 801.14 last 22.23) - open/close 18498 (time 1790743329.63 total 811.15 last 23.48) - open/close 18763 (time 1790743339.64 total 821.16 last 26.47) - open/close 18989 (time 1790743349.65 total 831.17 last 22.58) - open/close 19239 (time 1790743359.94 total 841.46 last 24.29) - open/close 19474 (time 1790743370.05 total 851.57 last 23.24) - open/close 19708 (time 1790743380.07 total 861.59 last 23.36) - open/close 19925 (time 1790743390.07 total 871.59 last 21.70) - open/close 20103 (time 1790743400.58 total 882.10 last 16.94) - open/close 20364 (time 1790743410.59 total 892.10 last 26.08) - open/close 20593 (time 1790743420.77 total 902.29 last 22.48) - open/close 20779 (time 1790743431.60 total 913.12 last 17.18) - open/close 21095 (time 1790743441.60 total 923.12 last 31.59) - open/close 21858 (time 1790743451.60 total 933.12 last 76.28) - open/close 22589 (time 1790743461.62 total 943.13 last 73.00) - open/close 23406 (time 1790743471.62 total 953.14 last 81.64) - open/close 24213 (time 1790743481.63 total 963.15 last 80.65) - open/close 25019 (time 1790743491.64 total 973.16 last 80.51) - open/close 25771 (time 1790743501.64 total 983.16 last 75.19) - open/close 26556 (time 1790743511.65 total 993.17 last 78.42) - open/close 27146 (time 1790743521.66 total 1003.18 last 58.94) - open/close 27763 (time 1790743531.70 total 1013.21 last 61.49) - open/close 28428 (time 1790743541.71 total 1023.23 last 66.43) - open/close 29119 (time 1790743551.71 total 1033.23 last 69.10) - open/close 29872 (time 1790743561.71 total 1043.23 last 75.27) - open/close 30614 (time 1790743571.71 total 1053.23 last 74.20) - open/close 31318 (time 1790743581.72 total 1063.23 last 70.36) - open/close 31977 (time 1790743591.73 total 1073.24 last 65.84) - open/close 32692 (time 1790743601.74 total 1083.26 last 71.40) - open/close 33349 (time 1790743611.76 total 1093.27 last 65.60) - open/close 33958 (time 1790743621.77 total 1103.28 last 60.83) - open/close 34729 (time 1790743631.78 total 1113.30 last 76.98) - open/close 35497 (time 1790743641.79 total 1123.31 last 76.74) - open/close 36313 (time 1790743651.79 total 1133.31 last 81.59) - open/close 37115 (time 1790743661.80 total 1143.31 last 80.15) - open/close 37887 (time 1790743671.80 total 1153.32 last 77.18) - open/close 38592 (time 1790743681.80 total 1163.32 last 70.49) - open/close 39316 (time 1790743691.81 total 1173.33 last 72.35) - open/close 40000 (time 1790743701.63 total 1183.15 last 69.63) - open/close 40716 (time 1790743711.65 total 1193.16 last 71.49) - open/close 41409 (time 1790743721.65 total 1203.17 last 69.27) - open/close 42193 (time 1790743731.65 total 1213.17 last 78.38) - open/close 42959 (time 1790743741.69 total 1223.20 last 76.35) - open/close 43678 (time 1790743751.69 total 1233.20 last 71.88) - open/close 44441 (time 1790743761.69 total 1243.21 last 76.28) - open/close 45164 (time 1790743771.70 total 1253.21 last 72.26) - open/close 45970 (time 1790743781.70 total 1263.22 last 80.57) - open/close 46774 (time 1790743791.71 total 1273.22 last 80.36) - open/close 47549 (time 1790743801.71 total 1283.23 last 77.47) - open/close 48312 (time 1790743811.74 total 1293.25 last 76.11) - open/close 49037 (time 1790743821.74 total 1303.26 last 72.45) - open/close 49636 (time 1790743831.76 total 1313.27 last 59.80) total: 50000 open/close in 1320.82 seconds: 37.86 ops/second - unlinked 0 (time 1790743843 ; total 0 ; last 0) - unlinked 10000 (time 1790743979 ; total 136 ; last 136) - unlinked 20000 (time 1790744122 ; total 279 ; last 143) - unlinked 30000 (time 1790744259 ; total 416 ; last 137) - unlinked 40000 (time 1790744396 ; total 553 ; last 137) total: 50000 unlinks in 687 seconds: 72.780205 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server unloading modules via unload_modules_local on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing unload_modules_local oleg323-server: kmemleak disabled modules unloaded. pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2