== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:50:41 (1788670241) start mds service on oleg633-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1922328 Sep 5 15:39 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg633-server' oleg633-server: oleg633-server.virtnet: executing load_modules_local oleg633-server: kmemleak disabled oleg633-server: Loading modules from /home/green/git/lustre-release/lustre oleg633-server: detected 4 online CPUs by sysfs oleg633-server: MODOPTS_LIBCFS= oleg633-server: Force libcfs to create 2 CPU partitions oleg633-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg633-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg633-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg633-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg633-server: oleg633-server.virtnet: executing set_default_debug -1 all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg633-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg633-server: oleg633-server.virtnet: executing set_default_debug -1 all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg633-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg633-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg633-server: oleg633-server.virtnet: executing set_default_debug -1 all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-OST0000 oleg633-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 128 (time 1788670340.55 total 10.01 last 12.79) - open/close 381 (time 1788670351.43 total 20.89 last 23.25) - open/close 630 (time 1788670361.45 total 30.90 last 24.87) - open/close 868 (time 1788670372.49 total 41.95 last 21.55) - open/close 1077 (time 1788670382.93 total 52.39 last 20.02) - open/close 1312 (time 1788670393.47 total 62.93 last 22.29) - open/close 1539 (time 1788670403.85 total 73.31 last 21.87) - open/close 1744 (time 1788670414.28 total 83.73 last 19.66) - open/close 1989 (time 1788670424.77 total 94.23 last 23.34) - open/close 2174 (time 1788670435.46 total 104.91 last 17.32) - open/close 2387 (time 1788670445.46 total 114.92 last 21.28) - open/close 2565 (time 1788670456.13 total 125.58 last 16.70) - open/close 2751 (time 1788670466.63 total 136.09 last 17.71) - open/close 2907 (time 1788670476.63 total 146.09 last 15.59) - open/close 3036 (time 1788670486.65 total 156.11 last 12.88) - open/close 3290 (time 1788670496.67 total 166.12 last 25.36) - open/close 3595 (time 1788670507.59 total 177.05 last 27.92) - open/close 3852 (time 1788670517.61 total 187.06 last 25.65) - open/close 4090 (time 1788670528.51 total 197.97 last 21.83) - open/close 4344 (time 1788670538.88 total 208.34 last 24.48) - open/close 4586 (time 1788670548.89 total 218.35 last 24.19) - open/close 4752 (time 1788670559.56 total 229.01 last 15.56) - open/close 4974 (time 1788670569.85 total 239.31 last 21.56) - open/close 5227 (time 1788670580.61 total 250.07 last 23.52) - open/close 5373 (time 1788670590.78 total 260.24 last 14.35) - open/close 5512 (time 1788670600.96 total 270.42 last 13.66) - open/close 5717 (time 1788670611.07 total 280.53 last 20.27) - open/close 5924 (time 1788670621.25 total 290.71 last 20.34) - open/close 6135 (time 1788670631.49 total 300.95 last 20.62) - open/close 6341 (time 1788670641.55 total 311.01 last 20.47) - open/close 6577 (time 1788670652.38 total 321.84 last 21.79) - open/close 6824 (time 1788670662.39 total 331.84 last 24.68) - open/close 7129 (time 1788670673.30 total 342.76 last 27.95) - open/close 7236 (time 1788670685.21 total 354.67 last 8.98) - open/close 7349 (time 1788670696.08 total 365.53 last 10.40) - open/close 7496 (time 1788670707.01 total 376.47 last 13.44) - open/close 7776 (time 1788670717.98 total 387.44 last 25.53) - open/close 7999 (time 1788670727.99 total 397.45 last 22.28) - open/close 8219 (time 1788670738.00 total 407.45 last 21.99) - open/close 8482 (time 1788670748.00 total 417.46 last 26.28) - open/close 8710 (time 1788670758.47 total 427.93 last 21.78) - open/close 9016 (time 1788670769.06 total 438.51 last 28.90) - open/close 9340 (time 1788670779.86 total 449.32 last 29.99) - open/close 9573 (time 1788670789.87 total 459.33 last 23.28) - open/close 9788 (time 1788670800.20 total 469.65 last 20.82) - open/close 10000 (time 1788670808.51 total 477.97 last 25.51) - open/close 10230 (time 1788670818.59 total 488.05 last 22.81) - open/close 10508 (time 1788670829.12 total 498.58 last 26.40) - open/close 10765 (time 1788670839.13 total 508.58 last 25.69) - open/close 11008 (time 1788670850.13 total 519.59 last 22.08) - open/close 11259 (time 1788670861.18 total 530.63 last 22.72) - open/close 11501 (time 1788670871.56 total 541.02 last 23.30) - open/close 11741 (time 1788670881.56 total 551.02 last 23.99) - open/close 11960 (time 1788670891.57 total 561.03 last 21.88) - open/close 12189 (time 1788670901.60 total 571.05 last 22.85) - open/close 12435 (time 1788670911.63 total 581.09 last 24.52) - open/close 12640 (time 1788670921.87 total 591.33 last 20.02) - open/close 12871 (time 1788670931.97 total 601.43 last 22.86) - open/close 13081 (time 1788670941.99 total 611.45 last 20.97) - open/close 13298 (time 1788670953.24 total 622.70 last 19.28) - open/close 13466 (time 1788670963.73 total 633.19 last 16.01) - open/close 13719 (time 1788670974.43 total 643.88 last 23.66) - open/close 13966 (time 1788670984.72 total 654.17 last 24.01) - open/close 14231 (time 1788670994.73 total 664.19 last 26.45) - open/close 14473 (time 1788671004.82 total 674.27 last 24.00) - open/close 14775 (time 1788671014.82 total 684.28 last 30.18) - open/close 15021 (time 1788671024.84 total 694.29 last 24.56) - open/close 15288 (time 1788671035.35 total 704.80 last 25.41) - open/close 15456 (time 1788671045.35 total 714.81 last 16.80) - open/close 15723 (time 1788671055.35 total 724.81 last 26.69) - open/close 15924 (time 1788671066.25 total 735.71 last 18.45) - open/close 16110 (time 1788671077.19 total 746.65 last 17.00) - open/close 16312 (time 1788671087.20 total 756.66 last 20.17) - open/close 16508 (time 1788671097.21 total 766.66 last 19.60) - open/close 16765 (time 1788671107.21 total 776.67 last 25.70) - open/close 16967 (time 1788671118.15 total 787.61 last 18.46) - open/close 17524 (time 1788671128.15 total 797.61 last 55.68) - open/close 18325 (time 1788671138.16 total 807.61 last 80.08) - open/close 19088 (time 1788671148.16 total 817.62 last 76.25) - open/close 19859 (time 1788671158.17 total 827.63 last 77.04) - open/close 20639 (time 1788671168.18 total 837.64 last 77.90) - open/close 21456 (time 1788671178.19 total 847.65 last 81.67) - open/close 22291 (time 1788671188.19 total 857.65 last 83.48) - open/close 23145 (time 1788671198.19 total 867.65 last 85.38) - open/close 23825 (time 1788671208.20 total 877.66 last 67.94) - open/close 24753 (time 1788671218.20 total 887.66 last 92.77) - open/close 25565 (time 1788671228.23 total 897.68 last 81.02) - open/close 26340 (time 1788671238.23 total 907.69 last 77.46) - open/close 27122 (time 1788671248.24 total 917.69 last 78.17) - open/close 27876 (time 1788671258.25 total 927.70 last 75.31) - open/close 28630 (time 1788671268.26 total 937.72 last 75.32) - open/close 29337 (time 1788671278.26 total 947.72 last 70.66) - open/close 30000 (time 1788671285.74 total 955.20 last 88.67) - open/close 30839 (time 1788671295.74 total 965.20 last 83.88) - open/close 31656 (time 1788671305.75 total 975.21 last 81.64) - open/close 32361 (time 1788671315.77 total 985.23 last 70.36) - open/close 33018 (time 1788671325.78 total 995.24 last 65.61) - open/close 33725 (time 1788671335.80 total 1005.25 last 70.62) - open/close 34537 (time 1788671345.80 total 1015.26 last 81.14) - open/close 35328 (time 1788671355.81 total 1025.27 last 79.03) - open/close 36114 (time 1788671365.82 total 1035.28 last 78.55) - open/close 36813 (time 1788671375.82 total 1045.28 last 69.90) - open/close 37534 (time 1788671385.84 total 1055.29 last 71.98) - open/close 38292 (time 1788671395.85 total 1065.31 last 75.65) - open/close 39124 (time 1788671405.86 total 1075.32 last 83.17) - open/close 39891 (time 1788671415.87 total 1085.32 last 76.63) - open/close 40693 (time 1788671425.88 total 1095.33 last 80.13) - open/close 41435 (time 1788671435.88 total 1105.34 last 74.19) - open/close 42178 (time 1788671445.89 total 1115.34 last 74.23) - open/close 42962 (time 1788671455.89 total 1125.35 last 78.37) - open/close 43805 (time 1788671465.89 total 1135.35 last 84.29) - open/close 44539 (time 1788671475.90 total 1145.36 last 73.32) - open/close 45237 (time 1788671485.91 total 1155.37 last 69.75) - open/close 46067 (time 1788671495.92 total 1165.38 last 82.95) - open/close 46849 (time 1788671505.93 total 1175.39 last 78.11) - open/close 47708 (time 1788671515.93 total 1185.39 last 85.88) - open/close 48492 (time 1788671525.94 total 1195.40 last 78.33) - open/close 49375 (time 1788671535.94 total 1205.40 last 88.28) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1208.80 seconds: 41.06 ops/second - unlinked 0 (time 1788671544 ; total 0 ; last 0) - unlinked 10000 (time 1788671672 ; total 128 ; last 128) - unlinked 20000 (time 1788671801 ; total 257 ; last 129) - unlinked 30000 (time 1788671933 ; total 389 ; last 132) - unlinked 40000 (time 1788672061 ; total 517 ; last 128) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 641 seconds: 77.427460 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg633-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg633-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg633-server stop mds service on oleg633-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg633-server stop mds service on oleg633-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg633-server unloading modules via unload_modules_local on: 'oleg633-server' oleg633-server: oleg633-server.virtnet: executing unload_modules_local oleg633-server: kmemleak disabled modules unloaded. pdsh@oleg633-client: oleg633-server: ssh exited with exit code 2 pdsh@oleg633-client: oleg633-client: ssh exited with exit code 2 pdsh@oleg633-client: oleg633-server: ssh exited with exit code 2 pdsh@oleg633-client: oleg633-client: ssh exited with exit code 2