== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:11:16 (1789431076) start mds service on oleg123-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 11:26 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing load_modules_local oleg123-server: kmemleak disabled oleg123-server: Loading modules from /home/green/git/lustre-release/lustre oleg123-server: detected 4 online CPUs by sysfs oleg123-server: MODOPTS_LIBCFS= oleg123-server: Force libcfs to create 2 CPU partitions oleg123-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg123-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg123-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg123-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg123-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug -1 all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 112 (time 1789431180.88 total 10.40 last 10.77) - open/close 357 (time 1789431190.90 total 20.42 last 24.45) - open/close 654 (time 1789431201.60 total 31.12 last 27.75) - open/close 913 (time 1789431211.91 total 41.43 last 25.13) - open/close 1108 (time 1789431222.20 total 51.72 last 18.94) - open/close 1318 (time 1789431235.00 total 64.52 last 16.41) - open/close 1421 (time 1789431245.07 total 74.59 last 10.23) - open/close 1551 (time 1789431256.06 total 85.58 last 11.83) - open/close 1713 (time 1789431270.08 total 99.60 last 11.55) - open/close 1848 (time 1789431282.76 total 112.28 last 10.65) - open/close 2015 (time 1789431292.77 total 122.29 last 16.69) - open/close 2149 (time 1789431302.78 total 132.30 last 13.38) - open/close 2315 (time 1789431313.38 total 142.90 last 15.67) - open/close 2526 (time 1789431323.70 total 153.22 last 20.44) - open/close 2767 (time 1789431334.33 total 163.85 last 22.68) - open/close 2946 (time 1789431344.34 total 173.86 last 17.87) - open/close 3149 (time 1789431354.50 total 184.02 last 19.99) - open/close 3379 (time 1789431364.51 total 194.03 last 22.99) - open/close 3579 (time 1789431374.71 total 204.23 last 19.60) - open/close 3868 (time 1789431385.06 total 214.58 last 27.92) - open/close 4117 (time 1789431396.03 total 225.55 last 22.70) - open/close 4205 (time 1789431406.55 total 236.07 last 8.37) - open/close 4315 (time 1789431416.56 total 246.08 last 10.99) - open/close 4376 (time 1789431426.57 total 256.09 last 6.09) - open/close 4577 (time 1789431436.83 total 266.35 last 19.59) - open/close 4854 (time 1789431446.83 total 276.35 last 27.69) - open/close 5050 (time 1789431459.26 total 288.78 last 15.77) - open/close 5279 (time 1789431469.27 total 298.79 last 22.88) - open/close 5549 (time 1789431479.49 total 309.01 last 26.41) - open/close 5832 (time 1789431489.51 total 319.03 last 28.24) - open/close 6016 (time 1789431499.71 total 329.23 last 18.05) - open/close 6219 (time 1789431510.30 total 339.82 last 19.17) - open/close 6444 (time 1789431520.32 total 349.85 last 22.44) - open/close 6649 (time 1789431530.33 total 359.85 last 20.49) - open/close 6859 (time 1789431540.35 total 369.87 last 20.97) - open/close 7037 (time 1789431557.15 total 386.67 last 10.59) - open/close 7284 (time 1789431567.29 total 396.81 last 24.35) - open/close 7423 (time 1789431577.48 total 407.00 last 13.64) - open/close 7591 (time 1789431587.78 total 417.30 last 16.32) - open/close 7736 (time 1789431597.78 total 427.30 last 14.50) - open/close 7957 (time 1789431608.08 total 437.60 last 21.45) - open/close 8185 (time 1789431618.09 total 447.61 last 22.78) - open/close 8430 (time 1789431628.11 total 457.63 last 24.45) - open/close 8650 (time 1789431638.13 total 467.65 last 21.97) - open/close 8923 (time 1789431648.16 total 477.68 last 27.21) - open/close 9126 (time 1789431658.55 total 488.07 last 19.55) - open/close 9357 (time 1789431669.14 total 498.66 last 21.81) - open/close 9498 (time 1789431679.67 total 509.19 last 13.39) - open/close 9741 (time 1789431689.68 total 519.20 last 24.27) - open/close 9903 (time 1789431699.96 total 529.48 last 15.76) - open/close 10000 (time 1789431703.88 total 533.40 last 24.77) - open/close 10241 (time 1789431714.13 total 543.65 last 23.51) - open/close 10505 (time 1789431724.24 total 553.76 last 26.10) - open/close 10716 (time 1789431734.25 total 563.77 last 21.08) - open/close 10946 (time 1789431744.95 total 574.47 last 21.49) - open/close 11173 (time 1789431754.97 total 584.49 last 22.67) - open/close 11400 (time 1789431764.98 total 594.50 last 22.67) - open/close 11574 (time 1789431774.99 total 604.51 last 17.37) - open/close 11810 (time 1789431785.01 total 614.53 last 23.57) - open/close 12004 (time 1789431795.04 total 624.56 last 19.34) - open/close 12249 (time 1789431805.05 total 634.57 last 24.48) - open/close 12449 (time 1789431815.45 total 644.97 last 19.22) - open/close 12659 (time 1789431825.55 total 655.07 last 20.80) - open/close 12877 (time 1789431836.17 total 665.69 last 20.52) - open/close 13024 (time 1789431846.34 total 675.86 last 14.46) - open/close 13260 (time 1789431856.34 total 685.86 last 23.58) - open/close 13319 (time 1789431866.36 total 695.88 last 5.89) - open/close 13536 (time 1789431876.36 total 705.88 last 21.69) - open/close 13697 (time 1789431887.37 total 716.89 last 14.62) - open/close 13937 (time 1789431897.38 total 726.90 last 23.98) - open/close 14100 (time 1789431907.39 total 736.91 last 16.28) - open/close 14317 (time 1789431917.41 total 746.93 last 21.67) - open/close 14540 (time 1789431927.42 total 756.94 last 22.27) - open/close 14828 (time 1789431937.85 total 767.37 last 27.63) - open/close 15013 (time 1789431948.29 total 777.82 last 17.71) - open/close 15181 (time 1789431958.53 total 788.05 last 16.41) - open/close 15368 (time 1789431968.55 total 798.07 last 18.67) - open/close 15566 (time 1789431978.55 total 808.07 last 19.79) - open/close 15809 (time 1789431989.54 total 819.06 last 22.11) - open/close 15974 (time 1789432000.34 total 829.87 last 15.27) - open/close 16176 (time 1789432010.35 total 839.87 last 20.18) - open/close 16317 (time 1789432020.37 total 849.89 last 14.08) - open/close 16529 (time 1789432030.59 total 860.11 last 20.73) - open/close 16772 (time 1789432040.60 total 870.12 last 24.28) - open/close 16944 (time 1789432050.61 total 880.13 last 17.19) - open/close 17125 (time 1789432061.24 total 890.76 last 17.03) - open/close 17341 (time 1789432071.42 total 900.94 last 21.21) - open/close 17575 (time 1789432082.04 total 911.56 last 22.04) - open/close 17793 (time 1789432092.73 total 922.25 last 20.39) - open/close 18036 (time 1789432102.73 total 932.25 last 24.30) - open/close 18211 (time 1789432113.53 total 943.05 last 16.21) - open/close 18395 (time 1789432124.36 total 953.88 last 16.98) - open/close 18644 (time 1789432134.65 total 964.17 last 24.20) - open/close 18966 (time 1789432144.66 total 974.18 last 32.16) - open/close 19735 (time 1789432154.67 total 984.19 last 76.85) - open/close 20000 (time 1789432157.89 total 987.41 last 82.41) - open/close 20868 (time 1789432167.90 total 997.42 last 86.72) - open/close 21701 (time 1789432177.90 total 1007.42 last 83.28) - open/close 22406 (time 1789432187.91 total 1017.43 last 70.39) - open/close 23092 (time 1789432197.92 total 1027.44 last 68.56) - open/close 23804 (time 1789432207.92 total 1037.44 last 71.18) - open/close 24560 (time 1789432217.92 total 1047.44 last 75.60) - open/close 25279 (time 1789432227.93 total 1057.45 last 71.87) - open/close 26016 (time 1789432237.93 total 1067.45 last 73.66) - open/close 26718 (time 1789432247.93 total 1077.45 last 70.20) - open/close 27522 (time 1789432257.93 total 1087.46 last 80.39) - open/close 28325 (time 1789432267.94 total 1097.46 last 80.29) - open/close 29134 (time 1789432277.94 total 1107.46 last 80.86) - open/close 29825 (time 1789432287.98 total 1117.50 last 68.84) - open/close 30000 (time 1789432290.96 total 1120.48 last 58.63) - open/close 30712 (time 1789432300.96 total 1130.48 last 71.18) - open/close 31404 (time 1789432310.97 total 1140.49 last 69.19) - open/close 32101 (time 1789432320.97 total 1150.49 last 69.68) - open/close 32903 (time 1789432330.97 total 1160.49 last 80.19) - open/close 33641 (time 1789432340.99 total 1170.51 last 73.68) - open/close 34326 (time 1789432350.99 total 1180.51 last 68.47) - open/close 35015 (time 1789432361.00 total 1190.52 last 68.87) - open/close 35762 (time 1789432371.01 total 1200.53 last 74.62) - open/close 36572 (time 1789432381.01 total 1210.53 last 80.94) - open/close 37403 (time 1789432391.02 total 1220.54 last 83.06) - open/close 38164 (time 1789432401.03 total 1230.55 last 76.06) - open/close 38943 (time 1789432411.03 total 1240.55 last 77.90) - open/close 39551 (time 1789432421.04 total 1250.56 last 60.69) - open/close 40000 (time 1789432427.04 total 1256.56 last 74.86) - open/close 40749 (time 1789432437.05 total 1266.57 last 74.83) - open/close 41491 (time 1789432447.06 total 1276.58 last 74.13) - open/close 42109 (time 1789432457.08 total 1286.60 last 61.65) - open/close 42952 (time 1789432467.16 total 1296.68 last 83.69) - open/close 43660 (time 1789432477.19 total 1306.71 last 70.60) - open/close 44308 (time 1789432487.19 total 1316.72 last 64.74) - open/close 45026 (time 1789432497.20 total 1326.72 last 71.77) - open/close 45775 (time 1789432507.21 total 1336.73 last 74.83) - open/close 46548 (time 1789432517.21 total 1346.73 last 77.27) - open/close 47165 (time 1789432527.22 total 1356.74 last 61.66) - open/close 47815 (time 1789432537.22 total 1366.74 last 64.99) - open/close 48642 (time 1789432547.22 total 1376.74 last 82.70) - open/close 49447 (time 1789432557.23 total 1386.75 last 80.43) total: 50000 open/close in 1394.84 seconds: 35.85 ops/second - unlinked 0 (time 1789432569 ; total 0 ; last 0) - unlinked 10000 (time 1789432713 ; total 144 ; last 144) - unlinked 20000 (time 1789432862 ; total 293 ; last 149) - unlinked 30000 (time 1789433006 ; total 437 ; last 144) - unlinked 40000 (time 1789433152 ; total 583 ; last 146) total: 50000 unlinks in 733 seconds: 68.212822 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg123-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg123-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg123-server stop mds service on oleg123-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg123-server unloading modules via unload_modules_local on: 'oleg123-server' oleg123-server: oleg123-server.virtnet: executing unload_modules_local oleg123-server: kmemleak disabled modules unloaded. pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-server: ssh exited with exit code 2 pdsh@oleg123-client: oleg123-client: ssh exited with exit code 2