== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:23:21 (1788935001) start mds service on oleg203-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 8 22:13 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: kmemleak disabled oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: detected 4 online CPUs by sysfs oleg203-server: MODOPTS_LIBCFS= oleg203-server: Force libcfs to create 2 CPU partitions oleg203-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg203-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg203-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg203-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg203-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg203-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 262 (time 1788935116.12 total 10.34 last 25.33) - open/close 521 (time 1788935126.74 total 20.96 last 24.39) - open/close 798 (time 1788935136.85 total 31.07 last 27.40) - open/close 1051 (time 1788935147.08 total 41.31 last 24.72) - open/close 1318 (time 1788935158.03 total 52.25 last 24.39) - open/close 1481 (time 1788935168.07 total 62.30 last 16.23) - open/close 1737 (time 1788935179.07 total 73.30 last 23.27) - open/close 1979 (time 1788935190.12 total 84.34 last 21.92) - open/close 2219 (time 1788935200.63 total 94.85 last 22.83) - open/close 2489 (time 1788935210.64 total 104.86 last 26.96) - open/close 2736 (time 1788935220.65 total 114.87 last 24.68) - open/close 3019 (time 1788935231.50 total 125.73 last 26.07) - open/close 3299 (time 1788935242.51 total 136.74 last 25.43) - open/close 3537 (time 1788935252.82 total 147.04 last 23.10) - open/close 3764 (time 1788935263.87 total 158.10 last 20.53) - open/close 4009 (time 1788935275.87 total 170.09 last 20.42) - open/close 4309 (time 1788935286.89 total 181.11 last 27.23) - open/close 4605 (time 1788935296.90 total 191.12 last 29.57) - open/close 4852 (time 1788935306.92 total 201.14 last 24.66) - open/close 5141 (time 1788935316.92 total 211.14 last 28.90) - open/close 5376 (time 1788935327.72 total 221.94 last 21.75) - open/close 5582 (time 1788935337.72 total 231.94 last 20.60) - open/close 5844 (time 1788935347.72 total 241.95 last 26.19) - open/close 6080 (time 1788935357.92 total 252.15 last 23.14) - open/close 6299 (time 1788935367.93 total 262.15 last 21.89) - open/close 6556 (time 1788935378.89 total 273.12 last 23.44) - open/close 6790 (time 1788935389.91 total 284.13 last 21.24) - open/close 7047 (time 1788935400.71 total 294.93 last 23.80) - open/close 7305 (time 1788935410.71 total 304.93 last 25.80) - open/close 7533 (time 1788935420.72 total 314.94 last 22.78) - open/close 7782 (time 1788935430.72 total 324.95 last 24.89) - open/close 8069 (time 1788935441.57 total 335.80 last 26.46) - open/close 8277 (time 1788935452.30 total 346.53 last 19.38) - open/close 8495 (time 1788935462.31 total 356.53 last 21.79) - open/close 8740 (time 1788935472.32 total 366.55 last 24.46) - open/close 8961 (time 1788935483.20 total 377.43 last 20.31) - open/close 9206 (time 1788935493.79 total 388.01 last 23.15) - open/close 9420 (time 1788935504.04 total 398.26 last 20.87) - open/close 9634 (time 1788935514.05 total 408.28 last 21.37) - open/close 9831 (time 1788935524.30 total 418.52 last 19.24) - open/close 10000 (time 1788935532.69 total 426.91 last 20.13) - open/close 10166 (time 1788935543.37 total 437.59 last 15.55) - open/close 10432 (time 1788935553.39 total 447.61 last 26.56) - open/close 10573 (time 1788935563.42 total 457.65 last 14.05) - open/close 10851 (time 1788935574.48 total 468.71 last 25.14) - open/close 11031 (time 1788935585.36 total 479.58 last 16.55) - open/close 11212 (time 1788935596.06 total 490.28 last 16.92) - open/close 11455 (time 1788935606.62 total 500.84 last 23.01) - open/close 11649 (time 1788935617.00 total 511.22 last 18.69) - open/close 11859 (time 1788935627.01 total 521.23 last 20.98) - open/close 12124 (time 1788935637.03 total 531.25 last 26.45) - open/close 12337 (time 1788935647.49 total 541.71 last 20.36) - open/close 12456 (time 1788935658.01 total 552.23 last 11.31) - open/close 12628 (time 1788935668.83 total 563.05 last 15.90) - open/close 12852 (time 1788935678.93 total 573.15 last 22.17) - open/close 13037 (time 1788935689.82 total 584.04 last 17.00) - open/close 13284 (time 1788935700.31 total 594.53 last 23.54) - open/close 13604 (time 1788935711.11 total 605.33 last 29.63) - open/close 13838 (time 1788935721.11 total 615.34 last 23.39) - open/close 14055 (time 1788935731.17 total 625.39 last 21.58) - open/close 14262 (time 1788935741.17 total 635.40 last 20.69) - open/close 14542 (time 1788935751.71 total 645.93 last 26.58) - open/close 14819 (time 1788935761.71 total 655.93 last 27.70) - open/close 15003 (time 1788935772.07 total 666.30 last 17.76) - open/close 15248 (time 1788935782.07 total 676.30 last 24.50) - open/close 15549 (time 1788935792.07 total 686.30 last 30.09) - open/close 15795 (time 1788935802.38 total 696.60 last 23.87) - open/close 16036 (time 1788935813.00 total 707.23 last 22.68) - open/close 16324 (time 1788935823.96 total 718.19 last 26.28) - open/close 16589 (time 1788935833.97 total 728.19 last 26.49) - open/close 16788 (time 1788935843.98 total 738.20 last 19.88) - open/close 17057 (time 1788935854.00 total 748.22 last 26.85) - open/close 17210 (time 1788935864.34 total 758.57 last 14.79) - open/close 17506 (time 1788935874.36 total 768.59 last 29.54) - open/close 17679 (time 1788935885.21 total 779.43 last 15.95) - open/close 17849 (time 1788935895.21 total 789.44 last 16.99) - open/close 18024 (time 1788935905.62 total 799.85 last 16.81) - open/close 18133 (time 1788935916.56 total 810.79 last 9.96) - open/close 18293 (time 1788935927.08 total 821.30 last 15.22) - open/close 18428 (time 1788935938.08 total 832.30 last 12.27) - open/close 18489 (time 1788935948.18 total 842.41 last 6.04) - open/close 18621 (time 1788935958.68 total 852.90 last 12.58) - open/close 18814 (time 1788935968.68 total 862.91 last 19.29) - open/close 18981 (time 1788935979.30 total 873.52 last 15.73) - open/close 19102 (time 1788935990.23 total 884.45 last 11.07) - open/close 19295 (time 1788936000.29 total 894.52 last 19.18) - open/close 19458 (time 1788936011.03 total 905.25 last 15.19) - open/close 19755 (time 1788936021.54 total 915.77 last 28.24) - open/close 19943 (time 1788936032.60 total 926.82 last 17.01) - open/close 20000 (time 1788936034.95 total 929.17 last 24.21) - open/close 20720 (time 1788936044.95 total 939.18 last 71.98) - open/close 21477 (time 1788936054.96 total 949.18 last 75.66) - open/close 22261 (time 1788936064.96 total 959.19 last 78.35) - open/close 23166 (time 1788936074.98 total 969.20 last 90.41) - open/close 24048 (time 1788936085.00 total 979.22 last 88.00) - open/close 25003 (time 1788936095.01 total 989.23 last 95.41) - open/close 26019 (time 1788936105.01 total 999.23 last 101.57) - open/close 26899 (time 1788936115.01 total 1009.24 last 87.98) - open/close 27621 (time 1788936125.02 total 1019.24 last 72.18) - open/close 28385 (time 1788936135.02 total 1029.24 last 76.39) - open/close 29242 (time 1788936145.02 total 1039.25 last 85.64) - open/close 29984 (time 1788936155.03 total 1049.25 last 74.16) - open/close 30727 (time 1788936165.04 total 1059.26 last 74.23) - open/close 31445 (time 1788936175.04 total 1069.26 last 71.79) - open/close 32160 (time 1788936185.04 total 1079.26 last 71.49) - open/close 32878 (time 1788936195.04 total 1089.27 last 71.79) - open/close 33528 (time 1788936205.05 total 1099.27 last 64.98) - open/close 34420 (time 1788936215.05 total 1109.28 last 89.13) - open/close 35152 (time 1788936225.06 total 1119.29 last 73.12) - open/close 35880 (time 1788936235.07 total 1129.30 last 72.73) - open/close 36978 (time 1788936245.08 total 1139.30 last 109.71) - open/close 37757 (time 1788936255.08 total 1149.31 last 77.87) - open/close 38527 (time 1788936265.11 total 1159.33 last 76.84) - open/close 39166 (time 1788936275.12 total 1169.34 last 63.83) - open/close 39885 (time 1788936285.12 total 1179.35 last 71.85) - open/close 40548 (time 1788936295.13 total 1189.35 last 66.26) - open/close 41418 (time 1788936305.14 total 1199.36 last 86.94) - open/close 41975 (time 1788936315.15 total 1209.37 last 55.65) - open/close 42766 (time 1788936325.15 total 1219.37 last 79.10) - open/close 43568 (time 1788936335.15 total 1229.37 last 80.20) - open/close 44474 (time 1788936345.15 total 1239.38 last 90.52) - open/close 45382 (time 1788936355.16 total 1249.39 last 90.74) - open/close 46110 (time 1788936365.17 total 1259.39 last 72.76) - open/close 46955 (time 1788936375.17 total 1269.40 last 84.46) - open/close 47933 (time 1788936385.18 total 1279.40 last 97.73) - open/close 48914 (time 1788936395.19 total 1289.41 last 98.03) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1297.46 seconds: 38.25 ops/second - unlinked 0 (time 1788936408 ; total 0 ; last 0) - unlinked 10000 (time 1788936549 ; total 141 ; last 141) - unlinked 20000 (time 1788936674 ; total 266 ; last 125) - unlinked 30000 (time 1788936797 ; total 389 ; last 123) - unlinked 40000 (time 1788936933 ; total 525 ; last 136) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 656 seconds: 75.657013 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg203-server unloading modules via unload_modules_local on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local oleg203-server: kmemleak disabled modules unloaded. pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2