== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:27:15 (1788816435) start mds service on oleg209-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931936 Sep 7 16:07 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: kmemleak disabled oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: detected 4 online CPUs by sysfs oleg209-server: MODOPTS_LIBCFS= oleg209-server: Force libcfs to create 2 CPU partitions oleg209-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg209-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg209-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg209-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg209-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 207 (time 1788816505.88 total 10.00 last 20.70) - open/close 392 (time 1788816515.89 total 20.02 last 18.46) - open/close 563 (time 1788816526.75 total 30.88 last 15.75) - open/close 863 (time 1788816537.46 total 41.58 last 28.03) - open/close 1050 (time 1788816547.75 total 51.88 last 18.16) - open/close 1268 (time 1788816557.77 total 61.89 last 21.77) - open/close 1558 (time 1788816567.77 total 71.89 last 28.99) - open/close 1759 (time 1788816577.90 total 82.03 last 19.84) - open/close 1940 (time 1788816588.45 total 92.58 last 17.15) - open/close 2126 (time 1788816598.46 total 102.59 last 18.59) - open/close 2316 (time 1788816608.46 total 112.59 last 19.00) - open/close 2506 (time 1788816619.30 total 123.42 last 17.53) - open/close 2661 (time 1788816629.79 total 133.91 last 14.78) - open/close 2786 (time 1788816640.73 total 144.86 last 11.42) - open/close 2964 (time 1788816651.68 total 155.81 last 16.25) - open/close 3121 (time 1788816662.11 total 166.23 last 15.06) - open/close 3292 (time 1788816672.60 total 176.73 last 16.29) - open/close 3495 (time 1788816683.31 total 187.44 last 18.95) - open/close 3721 (time 1788816694.31 total 198.44 last 20.55) - open/close 3873 (time 1788816704.68 total 208.80 last 14.67) - open/close 4029 (time 1788816715.36 total 219.49 last 14.60) - open/close 4217 (time 1788816725.37 total 229.49 last 18.79) - open/close 4408 (time 1788816736.11 total 240.24 last 17.77) - open/close 4580 (time 1788816746.92 total 251.05 last 15.91) - open/close 4807 (time 1788816757.48 total 261.60 last 21.50) - open/close 5015 (time 1788816768.35 total 272.48 last 19.13) - open/close 5253 (time 1788816779.10 total 283.22 last 22.15) - open/close 5455 (time 1788816789.85 total 293.98 last 18.78) - open/close 5671 (time 1788816800.68 total 304.81 last 19.95) - open/close 5786 (time 1788816811.56 total 315.69 last 10.57) - open/close 5951 (time 1788816821.60 total 325.73 last 16.44) - open/close 6236 (time 1788816831.70 total 335.83 last 28.21) - open/close 6353 (time 1788816841.70 total 345.83 last 11.70) - open/close 6556 (time 1788816851.71 total 355.83 last 20.29) - open/close 6719 (time 1788816862.38 total 366.51 last 15.27) - open/close 6911 (time 1788816872.39 total 376.51 last 19.19) - open/close 7103 (time 1788816882.40 total 386.52 last 19.18) - open/close 7246 (time 1788816892.71 total 396.84 last 13.86) - open/close 7369 (time 1788816903.66 total 407.78 last 11.24) - open/close 7523 (time 1788816913.96 total 418.08 last 14.95) - open/close 7640 (time 1788816924.63 total 428.76 last 10.96) - open/close 7763 (time 1788816934.70 total 438.82 last 12.22) - open/close 7902 (time 1788816945.25 total 449.37 last 13.17) - open/close 8082 (time 1788816955.36 total 459.48 last 17.81) - open/close 8221 (time 1788816965.72 total 469.84 last 13.42) - open/close 8403 (time 1788816976.42 total 480.54 last 17.01) - open/close 8546 (time 1788816986.72 total 490.85 last 13.88) - open/close 8745 (time 1788816997.61 total 501.73 last 18.28) - open/close 8913 (time 1788817007.61 total 511.74 last 16.79) - open/close 9135 (time 1788817017.62 total 521.74 last 22.19) - open/close 9313 (time 1788817028.33 total 532.46 last 16.62) - open/close 9490 (time 1788817038.81 total 542.94 last 16.88) - open/close 9637 (time 1788817048.99 total 553.11 last 14.45) - open/close 9765 (time 1788817059.93 total 564.05 last 11.70) - open/close 9939 (time 1788817070.43 total 574.55 last 16.57) - open/close 10000 (time 1788817073.43 total 577.55 last 20.33) - open/close 10213 (time 1788817084.45 total 588.57 last 19.33) - open/close 10306 (time 1788817095.07 total 599.20 last 8.75) - open/close 10508 (time 1788817105.77 total 609.89 last 18.89) - open/close 10709 (time 1788817115.88 total 620.01 last 19.87) - open/close 10958 (time 1788817126.62 total 630.75 last 23.18) - open/close 11126 (time 1788817137.19 total 641.32 last 15.89) - open/close 11217 (time 1788817147.61 total 651.73 last 8.74) - open/close 11518 (time 1788817157.61 total 661.74 last 30.08) - open/close 12477 (time 1788817167.62 total 671.74 last 95.87) - open/close 13556 (time 1788817177.62 total 681.74 last 107.88) - open/close 14533 (time 1788817187.62 total 691.75 last 97.66) - open/close 15538 (time 1788817197.62 total 701.75 last 100.50) - open/close 16595 (time 1788817207.63 total 711.75 last 105.65) - open/close 17609 (time 1788817217.64 total 721.76 last 101.28) - open/close 18616 (time 1788817227.65 total 731.77 last 100.61) - open/close 19686 (time 1788817237.65 total 741.77 last 106.98) - open/close 20000 (time 1788817240.66 total 744.79 last 104.23) - open/close 21063 (time 1788817250.67 total 754.80 last 106.19) - open/close 22025 (time 1788817260.67 total 764.80 last 96.17) - open/close 23023 (time 1788817270.69 total 774.81 last 99.69) - open/close 24077 (time 1788817280.69 total 784.82 last 105.35) - open/close 25208 (time 1788817290.69 total 794.82 last 113.06) - open/close 26359 (time 1788817300.70 total 804.83 last 115.02) - open/close 27433 (time 1788817310.71 total 814.84 last 107.30) - open/close 28572 (time 1788817320.71 total 824.84 last 113.89) - open/close 29100 (time 1788817330.72 total 834.85 last 52.74) - open/close 29546 (time 1788817340.74 total 844.86 last 44.53) - open/close 30000 (time 1788817349.58 total 853.70 last 51.37) - open/close 30348 (time 1788817359.59 total 863.71 last 34.77) - open/close 30788 (time 1788817369.61 total 873.73 last 43.91) - open/close 31191 (time 1788817379.63 total 883.75 last 40.22) - open/close 31796 (time 1788817389.64 total 893.76 last 60.44) - open/close 32473 (time 1788817399.65 total 903.78 last 67.59) - open/close 33048 (time 1788817409.66 total 913.78 last 57.48) - open/close 33595 (time 1788817419.68 total 923.80 last 54.58) - open/close 34182 (time 1788817429.68 total 933.81 last 58.66) - open/close 34998 (time 1788817439.69 total 943.81 last 81.60) - open/close 35769 (time 1788817449.69 total 953.82 last 77.06) - open/close 36535 (time 1788817459.70 total 963.82 last 76.55) - open/close 37309 (time 1788817469.71 total 973.83 last 77.31) - open/close 38072 (time 1788817479.71 total 983.84 last 76.27) - open/close 38674 (time 1788817489.72 total 993.85 last 60.15) - open/close 39303 (time 1788817499.73 total 1003.86 last 62.84) - open/close 40000 (time 1788817509.50 total 1013.62 last 71.38) - open/close 40696 (time 1788817519.50 total 1023.62 last 69.59) - open/close 41441 (time 1788817529.50 total 1033.62 last 74.48) - open/close 42142 (time 1788817539.51 total 1043.63 last 70.05) - open/close 42812 (time 1788817549.57 total 1053.69 last 66.58) - open/close 43494 (time 1788817559.58 total 1063.71 last 68.11) - open/close 44226 (time 1788817569.59 total 1073.71 last 73.17) - open/close 44955 (time 1788817579.59 total 1083.72 last 72.85) - open/close 45624 (time 1788817589.59 total 1093.72 last 66.90) - open/close 46372 (time 1788817599.61 total 1103.73 last 74.72) - open/close 47054 (time 1788817609.61 total 1113.74 last 68.14) - open/close 47799 (time 1788817619.62 total 1123.75 last 74.44) - open/close 48498 (time 1788817629.64 total 1133.76 last 69.79) - open/close 49109 (time 1788817639.65 total 1143.77 last 61.06) - open/close 49806 (time 1788817649.66 total 1153.78 last 69.62) total: 50000 open/close in 1156.57 seconds: 43.23 ops/second - unlinked 0 (time 1788817656 ; total 0 ; last 0) - unlinked 10000 (time 1788817806 ; total 150 ; last 150) - unlinked 20000 (time 1788817979 ; total 323 ; last 173) - unlinked 30000 (time 1788818120 ; total 464 ; last 141) - unlinked 40000 (time 1788818271 ; total 615 ; last 151) total: 50000 unlinks in 780 seconds: 64.102562 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server unloading modules via unload_modules_local on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing unload_modules_local oleg209-server: kmemleak disabled modules unloaded. pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2