== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:47:47 (1789368467) start mds service on oleg244-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 13 23:22 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg244-server' oleg244-server: oleg244-server.virtnet: executing load_modules_local oleg244-server: kmemleak disabled oleg244-server: Loading modules from /home/green/git/lustre-release/lustre oleg244-server: detected 4 online CPUs by sysfs oleg244-server: MODOPTS_LIBCFS= oleg244-server: Force libcfs to create 2 CPU partitions oleg244-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg244-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg244-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg244-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg244-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Started lustre-OST0000 oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg244-client.virtnet: -o user_xattr,flock 192.168.202.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 326 (time 1789368511.92 total 10.94 last 29.79) - open/close 461 (time 1789368522.03 total 21.06 last 13.35) - open/close 688 (time 1789368532.34 total 31.37 last 22.02) - open/close 1158 (time 1789368543.21 total 42.24 last 43.22) - open/close 1536 (time 1789368555.25 total 54.28 last 31.41) - open/close 1756 (time 1789368565.49 total 64.51 last 21.49) - open/close 2260 (time 1789368576.43 total 75.46 last 46.06) - open/close 2484 (time 1789368586.54 total 85.57 last 22.15) - open/close 3105 (time 1789368596.78 total 95.81 last 60.65) - open/close 3820 (time 1789368606.95 total 105.98 last 70.27) - open/close 4505 (time 1789368616.95 total 115.98 last 68.50) - open/close 5151 (time 1789368627.88 total 126.91 last 59.11) - open/close 5477 (time 1789368638.13 total 137.16 last 31.80) - open/close 5805 (time 1789368648.88 total 147.90 last 30.53) - open/close 6284 (time 1789368659.89 total 158.91 last 43.51) - open/close 6512 (time 1789368670.44 total 169.47 last 21.60) - open/close 6876 (time 1789368681.14 total 180.16 last 34.04) - open/close 7568 (time 1789368691.44 total 190.47 last 67.14) - open/close 7879 (time 1789368702.01 total 201.03 last 29.44) - open/close 8428 (time 1789368712.30 total 211.33 last 53.31) - open/close 9106 (time 1789368722.67 total 221.70 last 65.42) - open/close 9700 (time 1789368732.71 total 231.74 last 59.13) - open/close 10000 (time 1789368736.65 total 235.68 last 76.25) - open/close 10587 (time 1789368746.65 total 245.68 last 58.69) - open/close 11247 (time 1789368756.66 total 255.68 last 65.97) - open/close 11959 (time 1789368766.66 total 265.68 last 71.20) - open/close 12507 (time 1789368776.68 total 275.71 last 54.65) - open/close 13197 (time 1789368787.31 total 286.34 last 64.92) - open/close 13380 (time 1789368797.31 total 296.34 last 18.30) - open/close 13879 (time 1789368807.85 total 306.88 last 47.34) - open/close 14518 (time 1789368817.96 total 316.99 last 63.20) - open/close 15218 (time 1789368828.33 total 327.36 last 67.53) - open/close 17447 (time 1789368838.33 total 337.36 last 222.83) - open/close 20000 (time 1789368846.36 total 345.39 last 318.10) - open/close 23389 (time 1789368856.36 total 355.39 last 338.81) - open/close 26388 (time 1789368866.36 total 365.39 last 299.87) - open/close 29113 (time 1789368876.36 total 375.39 last 272.47) - open/close 30000 (time 1789368878.90 total 377.93 last 350.03) - open/close 32845 (time 1789368888.90 total 387.93 last 284.46) - open/close 36160 (time 1789368898.90 total 397.93 last 331.46) - open/close 39691 (time 1789368908.90 total 407.93 last 353.05) - open/close 42355 (time 1789368918.90 total 417.93 last 266.35) - open/close 45385 (time 1789368928.91 total 427.93 last 302.96) - open/close 48711 (time 1789368938.91 total 437.93 last 332.56) total: 50000 open/close in 441.33 seconds: 113.30 ops/second - unlinked 0 (time 1789368943 ; total 0 ; last 0) - unlinked 10000 (time 1789368961 ; total 18 ; last 18) - unlinked 20000 (time 1789368976 ; total 33 ; last 15) - unlinked 30000 (time 1789368990 ; total 47 ; last 14) - unlinked 40000 (time 1789369005 ; total 62 ; last 15) total: 50000 unlinks in 76 seconds: 657.894714 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg244-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg244-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg244-server stop mds service on oleg244-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg244-server unloading modules via unload_modules_local on: 'oleg244-server' oleg244-server: oleg244-server.virtnet: executing unload_modules_local oleg244-server: kmemleak disabled modules unloaded. pdsh@oleg244-client: oleg244-client: ssh exited with exit code 2 pdsh@oleg244-client: oleg244-server: ssh exited with exit code 2 pdsh@oleg244-client: oleg244-server: ssh exited with exit code 2 pdsh@oleg244-client: oleg244-client: ssh exited with exit code 2