== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:58:50 (1789484330) start mds service on oleg354-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 15 02:31 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg354-server' oleg354-server: oleg354-server.virtnet: executing load_modules_local oleg354-server: kmemleak disabled oleg354-server: Loading modules from /home/green/git/lustre-release/lustre oleg354-server: detected 4 online CPUs by sysfs oleg354-server: MODOPTS_LIBCFS= oleg354-server: Force libcfs to create 2 CPU partitions oleg354-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg354-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg354-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg354-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg354-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg354-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 168 (time 1789484433.89 total 11.06 last 15.20) - open/close 404 (time 1789484444.51 total 21.68 last 22.22) - open/close 603 (time 1789484454.53 total 31.69 last 19.87) - open/close 753 (time 1789484465.11 total 42.28 last 14.17) - open/close 906 (time 1789484475.23 total 52.40 last 15.12) - open/close 1114 (time 1789484485.49 total 62.65 last 20.28) - open/close 1223 (time 1789484495.84 total 73.01 last 10.53) - open/close 1406 (time 1789484506.83 total 84.00 last 16.65) - open/close 1591 (time 1789484516.84 total 94.00 last 18.49) - open/close 1818 (time 1789484527.26 total 104.43 last 21.78) - open/close 2088 (time 1789484537.26 total 114.43 last 27.00) - open/close 2344 (time 1789484547.92 total 125.09 last 24.02) - open/close 2567 (time 1789484558.87 total 136.03 last 20.38) - open/close 2864 (time 1789484568.88 total 146.04 last 29.67) - open/close 3100 (time 1789484581.41 total 158.58 last 18.82) - open/close 3312 (time 1789484592.53 total 169.70 last 19.07) - open/close 3541 (time 1789484602.54 total 179.71 last 22.88) - open/close 3812 (time 1789484613.53 total 190.70 last 24.66) - open/close 4108 (time 1789484623.53 total 200.70 last 29.59) - open/close 4315 (time 1789484634.46 total 211.62 last 18.95) - open/close 4551 (time 1789484644.46 total 221.63 last 23.60) - open/close 4794 (time 1789484654.57 total 231.73 last 24.04) - open/close 5042 (time 1789484664.86 total 242.03 last 24.08) - open/close 5295 (time 1789484675.29 total 252.46 last 24.26) - open/close 5572 (time 1789484685.31 total 262.47 last 27.67) - open/close 5835 (time 1789484696.22 total 273.38 last 24.11) - open/close 6105 (time 1789484706.90 total 284.07 last 25.26) - open/close 6333 (time 1789484716.92 total 294.09 last 22.76) - open/close 6527 (time 1789484727.45 total 304.61 last 18.43) - open/close 6792 (time 1789484737.45 total 314.62 last 26.48) - open/close 7030 (time 1789484747.93 total 325.10 last 22.71) - open/close 7274 (time 1789484759.72 total 336.88 last 20.70) - open/close 7495 (time 1789484769.74 total 346.91 last 22.05) - open/close 7657 (time 1789484779.84 total 357.00 last 16.05) - open/close 7875 (time 1789484791.90 total 369.06 last 18.08) - open/close 8157 (time 1789484801.90 total 379.07 last 28.18) - open/close 8398 (time 1789484814.30 total 391.46 last 19.44) - open/close 8674 (time 1789484824.31 total 401.48 last 27.56) - open/close 8899 (time 1789484834.71 total 411.88 last 21.64) - open/close 9070 (time 1789484845.27 total 422.43 last 16.21) - open/close 9221 (time 1789484857.58 total 434.75 last 12.26) - open/close 9413 (time 1789484868.32 total 445.49 last 17.88) - open/close 9704 (time 1789484879.40 total 456.56 last 26.27) - open/close 9924 (time 1789484889.41 total 466.57 last 21.98) - open/close 10000 (time 1789484892.68 total 469.85 last 23.21) - open/close 10265 (time 1789484903.23 total 480.40 last 25.12) - open/close 10546 (time 1789484914.13 total 491.30 last 25.77) - open/close 10815 (time 1789484924.43 total 501.59 last 26.13) - open/close 11081 (time 1789484937.11 total 514.28 last 20.97) - open/close 11318 (time 1789484947.13 total 524.29 last 23.67) - open/close 11615 (time 1789484957.13 total 534.30 last 29.68) - open/close 11851 (time 1789484967.15 total 544.31 last 23.56) - open/close 12082 (time 1789484977.24 total 554.41 last 22.89) - open/close 12333 (time 1789484988.07 total 565.24 last 23.17) - open/close 12530 (time 1789484998.08 total 575.24 last 19.70) - open/close 12763 (time 1789485008.08 total 585.25 last 23.28) - open/close 13046 (time 1789485018.10 total 595.27 last 28.25) - open/close 13256 (time 1789485029.16 total 606.33 last 18.98) - open/close 13552 (time 1789485039.92 total 617.09 last 27.51) - open/close 13802 (time 1789485050.98 total 628.14 last 22.61) - open/close 14034 (time 1789485061.41 total 638.57 last 22.25) - open/close 14306 (time 1789485072.23 total 649.39 last 25.14) - open/close 14567 (time 1789485083.15 total 660.32 last 23.89) - open/close 14847 (time 1789485093.98 total 671.15 last 25.85) - open/close 15081 (time 1789485104.16 total 681.33 last 23.00) - open/close 15331 (time 1789485114.45 total 691.61 last 24.30) - open/close 15605 (time 1789485125.09 total 702.25 last 25.76) - open/close 15856 (time 1789485135.34 total 712.51 last 24.48) - open/close 16151 (time 1789485146.08 total 723.24 last 27.48) - open/close 16352 (time 1789485156.09 total 733.25 last 20.08) - open/close 16467 (time 1789485166.63 total 743.79 last 10.91) - open/close 16616 (time 1789485177.58 total 754.74 last 13.61) - open/close 16836 (time 1789485187.82 total 764.99 last 21.47) - open/close 17042 (time 1789485198.54 total 775.71 last 19.22) - open/close 17159 (time 1789485208.56 total 785.73 last 11.67) - open/close 17346 (time 1789485218.71 total 795.87 last 18.43) - open/close 17533 (time 1789485229.37 total 806.53 last 17.54) - open/close 17599 (time 1789485239.51 total 816.68 last 6.51) - open/close 17803 (time 1789485250.39 total 827.56 last 18.75) - open/close 17990 (time 1789485260.98 total 838.14 last 17.66) - open/close 18159 (time 1789485271.32 total 848.48 last 16.35) - open/close 18210 (time 1789485281.95 total 859.11 last 4.80) - open/close 18387 (time 1789485292.51 total 869.67 last 16.76) - open/close 18539 (time 1789485302.57 total 879.74 last 15.11) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-18673) error: No space left on device total: 18673 open/close in 884.12 seconds: 21.12 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9030: kill: (114604) - No such process - unlinked 0 (time 1789485314 ; total 0 ; last 0) - unlinked 10000 (time 1789485440 ; total 126 ; last 126) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-18673) error: No such file or directory total: 18673 unlinks in 190 seconds: 98.278946 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg354-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg354-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg354-server stop mds service on oleg354-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg354-server stop mds service on oleg354-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg354-server unloading modules via unload_modules_local on: 'oleg354-server' oleg354-server: oleg354-server.virtnet: executing unload_modules_local oleg354-server: kmemleak disabled modules unloaded. pdsh@oleg354-client: oleg354-client: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-client: ssh exited with exit code 2 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 2