== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:32:52 (1788867172) start mds service on oleg622-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 8 04:31 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg622-server' oleg622-server: oleg622-server.virtnet: executing load_modules_local oleg622-server: kmemleak disabled oleg622-server: Loading modules from /home/green/git/lustre-release/lustre oleg622-server: detected 4 online CPUs by sysfs oleg622-server: MODOPTS_LIBCFS= oleg622-server: Force libcfs to create 2 CPU partitions oleg622-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg622-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg622-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg622-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg622-server: oleg622-server.virtnet: executing set_default_debug -1 all pdsh@oleg622-client: oleg622-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg622-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg622-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg622-server: oleg622-server.virtnet: executing set_default_debug -1 all pdsh@oleg622-client: oleg622-server: ssh exited with exit code 1 Started lustre-OST0000 oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg622-client.virtnet: -o user_xattr,flock 192.168.206.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 188 (time 1788867215.27 total 10.12 last 18.57) - open/close 460 (time 1788867226.86 total 21.71 last 23.47) - open/close 634 (time 1788867237.10 total 31.95 last 16.99) - open/close 864 (time 1788867247.53 total 42.38 last 22.06) - open/close 1153 (time 1788867258.47 total 53.33 last 26.40) - open/close 1468 (time 1788867269.42 total 64.27 last 28.79) - open/close 1674 (time 1788867280.17 total 75.02 last 19.15) - open/close 1838 (time 1788867290.54 total 85.39 last 15.82) - open/close 2002 (time 1788867301.55 total 96.40 last 14.89) - open/close 2159 (time 1788867311.72 total 106.57 last 15.43) - open/close 2242 (time 1788867322.54 total 117.39 last 7.67) - open/close 2353 (time 1788867332.58 total 127.44 last 11.05) - open/close 2838 (time 1788867342.70 total 137.55 last 47.97) - open/close 3022 (time 1788867352.81 total 147.67 last 18.19) - open/close 3149 (time 1788867362.99 total 157.84 last 12.48) - open/close 3319 (time 1788867373.35 total 168.21 last 16.40) - open/close 3559 (time 1788867383.47 total 178.32 last 23.73) - open/close 3735 (time 1788867393.96 total 188.82 last 16.76) - open/close 3879 (time 1788867404.20 total 199.05 last 14.07) - open/close 4032 (time 1788867414.25 total 209.10 last 15.22) - open/close 4119 (time 1788867425.26 total 220.11 last 7.90) - open/close 4614 (time 1788867436.26 total 231.12 last 44.96) - open/close 4878 (time 1788867447.03 total 241.88 last 24.53) - open/close 5152 (time 1788867458.99 total 253.84 last 22.91) - open/close 5567 (time 1788867468.99 total 263.84 last 41.49) - open/close 5768 (time 1788867479.21 total 274.06 last 19.67) - open/close 5934 (time 1788867489.26 total 284.11 last 16.52) - open/close 6396 (time 1788867499.50 total 294.35 last 45.11) - open/close 6655 (time 1788867510.06 total 304.91 last 24.53) - open/close 7225 (time 1788867520.43 total 315.28 last 54.99) - open/close 7544 (time 1788867530.47 total 325.33 last 31.75) - open/close 7844 (time 1788867541.17 total 336.02 last 28.05) - open/close 8189 (time 1788867551.98 total 346.83 last 31.91) - open/close 8479 (time 1788867562.80 total 357.65 last 26.82) - open/close 8765 (time 1788867573.04 total 367.89 last 27.93) - open/close 9389 (time 1788867583.04 total 377.89 last 62.37) - open/close 10000 (time 1788867587.32 total 382.17 last 142.81) - open/close 11553 (time 1788867597.32 total 392.18 last 155.23) - open/close 13493 (time 1788867607.33 total 402.18 last 193.96) - open/close 15604 (time 1788867617.33 total 412.18 last 211.00) - open/close 17218 (time 1788867627.33 total 422.19 last 161.35) - open/close 18704 (time 1788867637.34 total 432.19 last 148.52) - open/close 20000 (time 1788867645.63 total 440.49 last 156.24) - open/close 21560 (time 1788867655.64 total 450.49 last 155.96) - open/close 23085 (time 1788867665.64 total 460.49 last 152.44) - open/close 24726 (time 1788867675.64 total 470.50 last 164.04) - open/close 26280 (time 1788867685.64 total 480.50 last 155.40) - open/close 27694 (time 1788867695.65 total 490.50 last 141.35) - open/close 29079 (time 1788867705.65 total 500.50 last 138.43) - open/close 30000 (time 1788867712.46 total 507.32 last 135.22) - open/close 31311 (time 1788867722.46 total 517.32 last 131.09) - open/close 33744 (time 1788867732.47 total 527.32 last 243.21) - open/close 35212 (time 1788867742.47 total 537.32 last 146.76) - open/close 36452 (time 1788867752.48 total 547.33 last 123.89) - open/close 37610 (time 1788867762.48 total 557.33 last 115.77) - open/close 38825 (time 1788867772.49 total 567.34 last 121.43) - open/close 40000 (time 1788867781.73 total 576.58 last 127.19) - open/close 41322 (time 1788867791.74 total 586.59 last 132.02) - open/close 42563 (time 1788867801.74 total 596.60 last 124.05) - open/close 43816 (time 1788867811.75 total 606.60 last 125.23) - open/close 44990 (time 1788867821.75 total 616.60 last 117.36) - open/close 46289 (time 1788867831.76 total 626.61 last 129.81) - open/close 47607 (time 1788867841.76 total 636.62 last 131.75) - open/close 48955 (time 1788867851.77 total 646.62 last 134.70) total: 50000 open/close in 654.10 seconds: 76.44 ops/second - unlinked 0 (time 1788867861 ; total 0 ; last 0) - unlinked 10000 (time 1788867919 ; total 58 ; last 58) - unlinked 20000 (time 1788867963 ; total 102 ; last 44) - unlinked 30000 (time 1788868005 ; total 144 ; last 42) - unlinked 40000 (time 1788868039 ; total 178 ; last 34) total: 50000 unlinks in 215 seconds: 232.558136 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg622-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg622-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg622-server stop mds service on oleg622-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg622-server unloading modules via unload_modules_local on: 'oleg622-server' oleg622-server: oleg622-server.virtnet: executing unload_modules_local oleg622-server: kmemleak disabled modules unloaded. pdsh@oleg622-client: oleg622-client: ssh exited with exit code 2 pdsh@oleg622-client: oleg622-server: ssh exited with exit code 2 pdsh@oleg622-client: oleg622-client: ssh exited with exit code 2 pdsh@oleg622-client: oleg622-server: ssh exited with exit code 2