== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:53:28 (1789368808) start mds service on oleg107-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 14 01:55 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing load_modules_local oleg107-server: kmemleak disabled oleg107-server: Loading modules from /home/green/git/lustre-release/lustre oleg107-server: detected 4 online CPUs by sysfs oleg107-server: MODOPTS_LIBCFS= oleg107-server: Force libcfs to create 2 CPU partitions oleg107-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg107-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg107-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg107-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg107-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock 192.168.201.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 108 (time 1789368870.48 total 10.23 last 10.56) - open/close 233 (time 1789368880.49 total 20.23 last 12.50) - open/close 401 (time 1789368890.91 total 30.65 last 16.12) - open/close 674 (time 1789368901.21 total 40.96 last 26.48) - open/close 954 (time 1789368911.45 total 51.20 last 27.35) - open/close 1254 (time 1789368922.14 total 61.89 last 28.06) - open/close 1498 (time 1789368932.31 total 72.05 last 24.00) - open/close 1675 (time 1789368943.13 total 82.88 last 16.35) - open/close 1988 (time 1789368953.30 total 93.05 last 30.78) - open/close 2231 (time 1789368963.31 total 103.05 last 24.29) - open/close 2507 (time 1789368973.53 total 113.27 last 27.01) - open/close 2687 (time 1789368984.21 total 123.96 last 16.84) - open/close 2937 (time 1789368994.65 total 134.39 last 23.96) - open/close 3203 (time 1789369005.02 total 144.76 last 25.65) - open/close 3441 (time 1789369015.20 total 154.94 last 23.37) - open/close 3735 (time 1789369025.95 total 165.69 last 27.35) - open/close 3937 (time 1789369035.95 total 175.69 last 20.20) - open/close 4209 (time 1789369046.63 total 186.38 last 25.46) - open/close 4559 (time 1789369056.64 total 196.38 last 34.99) - open/close 4787 (time 1789369066.64 total 206.38 last 22.80) - open/close 4960 (time 1789369077.47 total 217.21 last 15.98) - open/close 5153 (time 1789369087.58 total 227.33 last 19.08) - open/close 5486 (time 1789369097.58 total 237.33 last 33.29) - open/close 5747 (time 1789369107.87 total 247.62 last 25.37) - open/close 6067 (time 1789369118.94 total 258.69 last 28.91) - open/close 6405 (time 1789369129.88 total 269.63 last 30.89) - open/close 6678 (time 1789369140.57 total 280.31 last 25.55) - open/close 6921 (time 1789369150.57 total 290.32 last 24.29) - open/close 7154 (time 1789369161.58 total 301.32 last 21.17) - open/close 7487 (time 1789369171.59 total 311.33 last 33.27) - open/close 7835 (time 1789369184.25 total 323.99 last 27.49) - open/close 8092 (time 1789369194.93 total 334.68 last 24.05) - open/close 8419 (time 1789369205.99 total 345.73 last 29.57) - open/close 8737 (time 1789369216.86 total 356.60 last 29.25) - open/close 9068 (time 1789369227.17 total 366.92 last 32.09) - open/close 9419 (time 1789369237.54 total 377.28 last 33.86) - open/close 9617 (time 1789369247.55 total 387.29 last 19.79) - open/close 9949 (time 1789369257.55 total 397.29 last 33.19) - open/close 10000 (time 1789369260.14 total 399.89 last 19.65) - open/close 10193 (time 1789369270.16 total 409.90 last 19.28) - open/close 10495 (time 1789369280.79 total 420.54 last 28.39) - open/close 10825 (time 1789369292.89 total 432.63 last 27.28) - open/close 11078 (time 1789369303.26 total 443.00 last 24.40) - open/close 12401 (time 1789369313.26 total 453.01 last 132.23) - open/close 14249 (time 1789369323.27 total 463.01 last 184.71) - open/close 16249 (time 1789369333.27 total 473.01 last 199.97) - open/close 18625 (time 1789369343.27 total 483.01 last 237.56) - open/close 20000 (time 1789369347.12 total 486.87 last 356.77) - open/close 22827 (time 1789369357.13 total 496.87 last 282.61) - open/close 24798 (time 1789369367.13 total 506.88 last 196.95) - open/close 27114 (time 1789369377.14 total 516.88 last 231.59) - open/close 30000 (time 1789369385.54 total 525.28 last 343.58) - open/close 33872 (time 1789369395.54 total 535.28 last 387.19) - open/close 38421 (time 1789369405.54 total 545.28 last 454.88) - open/close 40000 (time 1789369409.12 total 548.86 last 441.13) - open/close 43999 (time 1789369419.12 total 558.86 last 399.82) - open/close 48300 (time 1789369429.12 total 568.86 last 430.05) total: 50000 open/close in 573.13 seconds: 87.24 ops/second - unlinked 0 (time 1789369434 ; total 0 ; last 0) - unlinked 10000 (time 1789369451 ; total 17 ; last 17) - unlinked 20000 (time 1789369466 ; total 32 ; last 15) - unlinked 30000 (time 1789369483 ; total 49 ; last 17) - unlinked 40000 (time 1789369504 ; total 70 ; last 21) total: 50000 unlinks in 90 seconds: 555.555542 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server unloading modules via unload_modules_local on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing unload_modules_local oleg107-server: kmemleak disabled modules unloaded. pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2