== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:11:58 (1788883918) start mds service on oleg452-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' llite/lustre options: 'enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 8 11:27 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg452-server' oleg452-server: oleg452-server.virtnet: executing load_modules_local oleg452-server: kmemleak disabled oleg452-server: Loading modules from /home/green/git/lustre-release/lustre oleg452-server: detected 4 online CPUs by sysfs oleg452-server: MODOPTS_LIBCFS= oleg452-server: Force libcfs to create 2 CPU partitions oleg452-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg452-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg452-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg452-server: llite/lustre options: 'enable_flr_ec=1' oleg452-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg452-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg452-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg452-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg452-server: oleg452-server.virtnet: executing set_default_debug -1 all pdsh@oleg452-client: oleg452-server: ssh exited with exit code 1 Started lustre-OST0000 oleg452-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg452-client.virtnet: -o user_xattr,flock 192.168.204.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 221 (time 1788883972.59 total 10.22 last 21.63) - open/close 389 (time 1788883982.63 total 20.26 last 16.73) - open/close 712 (time 1788883993.19 total 30.82 last 30.58) - open/close 857 (time 1788884003.24 total 40.87 last 14.43) - open/close 1025 (time 1788884013.35 total 50.98 last 16.61) - open/close 1199 (time 1788884023.78 total 61.41 last 16.69) - open/close 1421 (time 1788884033.89 total 71.52 last 21.96) - open/close 1923 (time 1788884043.90 total 81.53 last 50.18) - open/close 2119 (time 1788884054.44 total 92.07 last 18.59) - open/close 2337 (time 1788884064.81 total 102.44 last 21.03) - open/close 2495 (time 1788884074.86 total 112.49 last 15.72) - open/close 2698 (time 1788884085.16 total 122.79 last 19.70) - open/close 3319 (time 1788884095.59 total 133.22 last 59.54) - open/close 3953 (time 1788884105.83 total 143.46 last 61.91) - open/close 4556 (time 1788884117.10 total 154.73 last 53.52) - open/close 4858 (time 1788884128.55 total 166.18 last 26.36) - open/close 5026 (time 1788884139.50 total 177.13 last 15.35) - open/close 5584 (time 1788884149.74 total 187.37 last 54.48) - open/close 6206 (time 1788884160.17 total 197.80 last 59.63) - open/close 6917 (time 1788884170.73 total 208.36 last 67.34) - open/close 7601 (time 1788884180.84 total 218.47 last 67.66) - open/close 8304 (time 1788884190.95 total 228.58 last 69.51) - open/close 8956 (time 1788884200.95 total 238.58 last 65.19) - open/close 9614 (time 1788884211.81 total 249.44 last 60.58) - open/close 10000 (time 1788884216.91 total 254.54 last 75.77) - open/close 10677 (time 1788884226.91 total 264.54 last 67.70) - open/close 11271 (time 1788884237.61 total 275.24 last 55.52) - open/close 11844 (time 1788884248.04 total 285.67 last 54.94) - open/close 12473 (time 1788884258.73 total 296.36 last 58.83) - open/close 13051 (time 1788884269.22 total 306.85 last 55.08) - open/close 13611 (time 1788884279.72 total 317.35 last 53.36) - open/close 14273 (time 1788884289.72 total 327.35 last 66.20) - open/close 16941 (time 1788884299.72 total 337.35 last 266.80) - open/close 20000 (time 1788884305.56 total 343.19 last 523.38) - open/close 25094 (time 1788884315.56 total 353.19 last 509.37) - open/close 30000 (time 1788884324.82 total 362.45 last 530.22) - open/close 34896 (time 1788884334.82 total 372.45 last 489.56) - open/close 39918 (time 1788884344.82 total 382.45 last 502.17) - open/close 45096 (time 1788884354.82 total 392.45 last 517.75) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 402.33 seconds: 123.36 ops/second - unlinked 0 (time 1788884366 ; total 0 ; last 0) - unlinked 10000 (time 1788884377 ; total 11 ; last 11) - unlinked 20000 (time 1788884388 ; total 22 ; last 11) - unlinked 30000 (time 1788884400 ; total 34 ; last 12) - unlinked 40000 (time 1788884411 ; total 45 ; last 11) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 55 seconds: 902.381836 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg452-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg452-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg452-server stop mds service on oleg452-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg452-server stop mds service on oleg452-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg452-server unloading modules via unload_modules_local on: 'oleg452-server' oleg452-server: oleg452-server.virtnet: executing unload_modules_local oleg452-server: kmemleak disabled modules unloaded. pdsh@oleg452-client: oleg452-client: ssh exited with exit code 2 pdsh@oleg452-client: oleg452-server: ssh exited with exit code 2 pdsh@oleg452-client: oleg452-server: ssh exited with exit code 2 pdsh@oleg452-client: oleg452-client: ssh exited with exit code 2