== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:42:42 (1789188162) start mds service on oleg249-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931560 Sep 11 13:50 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing load_modules_local oleg249-server: kmemleak disabled oleg249-server: Loading modules from /home/green/git/lustre-release/lustre oleg249-server: detected 4 online CPUs by sysfs oleg249-server: MODOPTS_LIBCFS= oleg249-server: Force libcfs to create 2 CPU partitions oleg249-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg249-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg249-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg249-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg249-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg249-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg249-server: oleg249-server.virtnet: executing set_default_debug -1 all pdsh@oleg249-client: oleg249-server: ssh exited with exit code 1 Started lustre-OST0000 oleg249-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg249-client.virtnet: -o user_xattr,flock 192.168.202.149@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 255 (time 1789188209.59 total 10.49 last 24.32) - open/close 526 (time 1789188220.53 total 21.43 last 24.76) - open/close 718 (time 1789188230.53 total 31.43 last 19.20) - open/close 971 (time 1789188241.91 total 42.81 last 22.24) - open/close 1145 (time 1789188252.22 total 53.12 last 16.87) - open/close 1331 (time 1789188262.22 total 63.12 last 18.60) - open/close 1632 (time 1789188272.23 total 73.13 last 30.07) - open/close 1854 (time 1789188282.61 total 83.51 last 21.40) - open/close 2132 (time 1789188292.98 total 93.88 last 26.79) - open/close 2392 (time 1789188303.85 total 104.75 last 23.92) - open/close 2650 (time 1789188314.15 total 115.06 last 25.04) - open/close 2844 (time 1789188324.16 total 125.06 last 19.40) - open/close 3130 (time 1789188335.03 total 135.93 last 26.31) - open/close 3475 (time 1789188345.45 total 146.35 last 33.10) - open/close 3774 (time 1789188356.33 total 157.23 last 27.48) - open/close 3993 (time 1789188366.46 total 167.37 last 21.62) - open/close 4309 (time 1789188376.47 total 177.38 last 31.56) - open/close 4544 (time 1789188387.01 total 187.91 last 22.32) - open/close 4759 (time 1789188397.19 total 198.09 last 21.11) - open/close 4946 (time 1789188407.48 total 208.38 last 18.17) - open/close 5139 (time 1789188417.85 total 218.75 last 18.62) - open/close 5401 (time 1789188428.80 total 229.70 last 23.92) - open/close 5691 (time 1789188438.81 total 239.71 last 28.97) - open/close 5912 (time 1789188449.28 total 250.18 last 21.11) - open/close 6199 (time 1789188460.14 total 261.04 last 26.42) - open/close 6491 (time 1789188470.71 total 271.61 last 27.62) - open/close 6724 (time 1789188481.52 total 282.42 last 21.56) - open/close 6961 (time 1789188492.14 total 293.05 last 22.31) - open/close 7189 (time 1789188503.16 total 304.06 last 20.70) - open/close 7446 (time 1789188513.52 total 314.42 last 24.81) - open/close 7726 (time 1789188523.95 total 324.85 last 26.85) - open/close 8021 (time 1789188534.65 total 335.55 last 27.57) - open/close 8317 (time 1789188545.39 total 346.30 last 27.55) - open/close 8520 (time 1789188555.39 total 356.30 last 20.30) - open/close 8766 (time 1789188568.30 total 369.20 last 19.06) - open/close 8986 (time 1789188579.31 total 380.21 last 19.99) - open/close 9215 (time 1789188589.94 total 390.84 last 21.53) - open/close 9467 (time 1789188599.95 total 400.85 last 25.18) - open/close 9749 (time 1789188609.97 total 410.87 last 28.15) - open/close 9940 (time 1789188620.46 total 421.36 last 18.20) - open/close 10199 (time 1789188630.47 total 431.37 last 25.88) - open/close 10296 (time 1789188641.27 total 442.17 last 8.98) - open/close 11150 (time 1789188651.27 total 452.17 last 85.39) - open/close 13206 (time 1789188661.27 total 462.17 last 205.59) - open/close 15093 (time 1789188671.27 total 472.17 last 188.70) - open/close 16954 (time 1789188681.27 total 482.18 last 185.98) - open/close 19957 (time 1789188691.27 total 492.18 last 300.30) - open/close 23325 (time 1789188701.28 total 502.18 last 336.76) - open/close 27244 (time 1789188711.28 total 512.18 last 391.87) - open/close 30000 (time 1789188718.67 total 519.58 last 372.60) - open/close 32846 (time 1789188728.68 total 529.58 last 284.52) - open/close 35397 (time 1789188738.68 total 539.58 last 255.00) - open/close 38092 (time 1789188748.69 total 549.59 last 269.37) - open/close 40000 (time 1789188755.26 total 556.17 last 290.01) - open/close 42968 (time 1789188765.27 total 566.17 last 296.60) - open/close 45354 (time 1789188775.27 total 576.17 last 238.58) - open/close 47045 (time 1789188785.27 total 586.17 last 169.07) - open/close 48714 (time 1789188795.28 total 596.18 last 166.84) total: 50000 open/close in 602.66 seconds: 82.97 ops/second - unlinked 0 (time 1789188803 ; total 0 ; last 0) - unlinked 10000 (time 1789188832 ; total 29 ; last 29) - unlinked 20000 (time 1789188857 ; total 54 ; last 25) - unlinked 30000 (time 1789188877 ; total 74 ; last 20) - unlinked 40000 (time 1789188900 ; total 97 ; last 23) total: 50000 unlinks in 122 seconds: 409.836060 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg249-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg249-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg249-server stop mds service on oleg249-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg249-server unloading modules via unload_modules_local on: 'oleg249-server' oleg249-server: oleg249-server.virtnet: executing unload_modules_local oleg249-server: kmemleak disabled modules unloaded. pdsh@oleg249-client: oleg249-client: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-server: ssh exited with exit code 2 pdsh@oleg249-client: oleg249-client: ssh exited with exit code 2