== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:21:58 (1788441718) start mds service on oleg434-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 3 02:21 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing load_modules_local oleg434-server: kmemleak disabled oleg434-server: Loading modules from /home/green/git/lustre-release/lustre oleg434-server: detected 4 online CPUs by sysfs oleg434-server: MODOPTS_LIBCFS= oleg434-server: Force libcfs to create 2 CPU partitions oleg434-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg434-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg434-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg434-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg434-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg434-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0000 oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 239 (time 1788441834.99 total 10.42 last 22.93) - open/close 403 (time 1788441845.00 total 20.43 last 16.39) - open/close 610 (time 1788441855.79 total 31.22 last 19.18) - open/close 838 (time 1788441866.38 total 41.81 last 21.53) - open/close 1017 (time 1788441876.86 total 52.29 last 17.09) - open/close 1208 (time 1788441887.82 total 63.26 last 17.41) - open/close 1457 (time 1788441902.34 total 77.77 last 17.16) - open/close 1661 (time 1788441912.35 total 87.78 last 20.38) - open/close 1834 (time 1788441923.32 total 98.75 last 15.76) - open/close 2046 (time 1788441933.55 total 108.98 last 20.72) - open/close 2334 (time 1788441943.56 total 118.99 last 28.77) - open/close 2373 (time 1788441954.36 total 129.79 last 3.61) - open/close 2538 (time 1788441964.98 total 140.41 last 15.54) - open/close 2766 (time 1788441975.00 total 150.43 last 22.75) - open/close 2887 (time 1788441985.73 total 161.16 last 11.28) - open/close 3008 (time 1788441996.47 total 171.90 last 11.26) - open/close 3219 (time 1788442006.64 total 182.07 last 20.75) - open/close 3404 (time 1788442017.37 total 192.80 last 17.25) - open/close 3663 (time 1788442027.38 total 202.81 last 25.87) - open/close 3876 (time 1788442037.39 total 212.82 last 21.28) - open/close 4096 (time 1788442047.40 total 222.83 last 21.97) - open/close 4246 (time 1788442057.98 total 233.41 last 14.18) - open/close 4315 (time 1788442067.99 total 243.42 last 6.90) - open/close 4556 (time 1788442078.00 total 253.43 last 24.07) - open/close 4824 (time 1788442088.02 total 263.45 last 26.74) - open/close 5113 (time 1788442098.82 total 274.25 last 26.78) - open/close 5304 (time 1788442108.86 total 284.29 last 19.02) - open/close 5478 (time 1788442119.67 total 295.10 last 16.09) - open/close 5616 (time 1788442129.78 total 305.21 last 13.65) - open/close 5777 (time 1788442140.35 total 315.78 last 15.23) - open/close 5937 (time 1788442150.36 total 325.79 last 15.99) - open/close 6061 (time 1788442160.37 total 335.80 last 12.39) - open/close 6163 (time 1788442172.72 total 348.15 last 8.26) - open/close 6286 (time 1788442182.74 total 358.17 last 12.27) - open/close 6514 (time 1788442192.75 total 368.18 last 22.79) - open/close 6776 (time 1788442202.76 total 378.19 last 26.18) - open/close 6872 (time 1788442213.64 total 389.07 last 8.82) - open/close 7089 (time 1788442224.40 total 399.83 last 20.17) - open/close 7193 (time 1788442235.00 total 410.43 last 9.81) - open/close 7386 (time 1788442245.00 total 420.43 last 19.29) - open/close 7649 (time 1788442255.02 total 430.45 last 26.25) - open/close 7788 (time 1788442265.05 total 440.48 last 13.86) - open/close 7939 (time 1788442275.05 total 450.48 last 15.09) - open/close 8073 (time 1788442285.06 total 460.49 last 13.39) - open/close 8225 (time 1788442295.50 total 470.93 last 14.55) - open/close 8353 (time 1788442305.51 total 480.94 last 12.80) - open/close 8479 (time 1788442315.52 total 490.95 last 12.58) - open/close 8649 (time 1788442326.07 total 501.50 last 16.11) - open/close 8846 (time 1788442336.07 total 511.50 last 19.69) - open/close 9046 (time 1788442346.68 total 522.11 last 18.86) - open/close 9230 (time 1788442357.12 total 532.55 last 17.62) - open/close 9397 (time 1788442367.81 total 543.24 last 15.63) - open/close 9575 (time 1788442378.87 total 554.30 last 16.08) - open/close 9772 (time 1788442389.56 total 564.99 last 18.44) - open/close 9954 (time 1788442399.87 total 575.30 last 17.66) - open/close 10014 (time 1788442410.37 total 585.80 last 5.71) - open/close 10196 (time 1788442421.06 total 596.49 last 17.04) - open/close 10414 (time 1788442431.36 total 606.79 last 21.16) - open/close 10460 (time 1788442441.65 total 617.08 last 4.47) - open/close 10669 (time 1788442452.73 total 628.16 last 18.87) - open/close 10939 (time 1788442462.74 total 638.17 last 26.97) - open/close 11186 (time 1788442472.90 total 648.33 last 24.31) - open/close 11420 (time 1788442482.90 total 658.33 last 23.39) - open/close 11691 (time 1788442493.96 total 669.39 last 24.50) - open/close 11945 (time 1788442504.70 total 680.13 last 23.65) - open/close 12209 (time 1788442515.53 total 690.96 last 24.39) - open/close 12412 (time 1788442525.87 total 701.30 last 19.62) - open/close 12671 (time 1788442535.93 total 711.36 last 25.75) - open/close 12954 (time 1788442545.94 total 721.37 last 28.27) - open/close 13037 (time 1788442559.85 total 735.28 last 5.97) - open/close 13274 (time 1788442570.11 total 745.54 last 23.10) - open/close 13518 (time 1788442580.72 total 756.15 last 22.99) - open/close 13778 (time 1788442591.30 total 766.73 last 24.57) - open/close 13910 (time 1788442601.31 total 776.74 last 13.20) - open/close 14127 (time 1788442611.32 total 786.75 last 21.68) - open/close 14240 (time 1788442621.95 total 797.38 last 10.63) - open/close 14524 (time 1788442631.96 total 807.39 last 28.38) - open/close 14784 (time 1788442641.98 total 817.41 last 25.96) - open/close 15012 (time 1788442651.99 total 827.42 last 22.78) - open/close 15195 (time 1788442662.00 total 837.43 last 18.27) - open/close 15328 (time 1788442672.70 total 848.13 last 12.44) - open/close 15541 (time 1788442682.70 total 858.13 last 21.29) - open/close 15808 (time 1788442693.05 total 868.48 last 25.81) - open/close 16016 (time 1788442704.66 total 880.09 last 17.91) - open/close 16175 (time 1788442715.25 total 890.68 last 15.02) - open/close 16425 (time 1788442725.37 total 900.80 last 24.72) - open/close 16560 (time 1788442735.37 total 910.80 last 13.50) - open/close 16736 (time 1788442746.22 total 921.65 last 16.21) - open/close 16947 (time 1788442756.86 total 932.29 last 19.83) - open/close 17109 (time 1788442766.87 total 942.30 last 16.19) - open/close 17291 (time 1788442779.85 total 955.28 last 14.03) - open/close 17369 (time 1788442789.96 total 965.39 last 7.71) - open/close 17528 (time 1788442802.93 total 978.36 last 12.26) - open/close 17655 (time 1788442815.57 total 991.00 last 10.05) - open/close 17932 (time 1788442826.23 total 1001.66 last 25.98) - open/close 18037 (time 1788442837.68 total 1013.11 last 9.17) - open/close 18145 (time 1788442848.44 total 1023.87 last 10.04) - open/close 18317 (time 1788442858.44 total 1033.87 last 17.19) - open/close 18948 (time 1788442868.46 total 1043.89 last 62.99) - open/close 19704 (time 1788442878.47 total 1053.90 last 75.52) - open/close 20000 (time 1788442882.21 total 1057.64 last 79.11) - open/close 20704 (time 1788442892.22 total 1067.65 last 70.35) - open/close 21423 (time 1788442902.23 total 1077.66 last 71.83) - open/close 22100 (time 1788442912.24 total 1087.67 last 67.63) - open/close 22790 (time 1788442922.26 total 1097.69 last 68.89) - open/close 23355 (time 1788442932.26 total 1107.69 last 56.47) - open/close 23945 (time 1788442942.26 total 1117.69 last 58.99) - open/close 24648 (time 1788442952.28 total 1127.71 last 70.18) - open/close 25396 (time 1788442962.29 total 1137.72 last 74.74) - open/close 26176 (time 1788442972.30 total 1147.73 last 77.91) - open/close 26896 (time 1788442982.30 total 1157.74 last 71.95) - open/close 27615 (time 1788442992.32 total 1167.75 last 71.81) - open/close 28380 (time 1788443002.32 total 1177.75 last 76.47) - open/close 29174 (time 1788443012.33 total 1187.76 last 79.36) - open/close 29756 (time 1788443022.33 total 1197.76 last 58.16) - open/close 30000 (time 1788443025.34 total 1200.77 last 81.05) - open/close 30769 (time 1788443035.36 total 1210.79 last 76.78) - open/close 31487 (time 1788443045.36 total 1220.80 last 71.75) - open/close 32216 (time 1788443055.37 total 1230.80 last 72.85) - open/close 32938 (time 1788443065.38 total 1240.81 last 72.16) - open/close 33651 (time 1788443075.39 total 1250.82 last 71.20) - open/close 34352 (time 1788443085.40 total 1260.83 last 70.03) - open/close 34992 (time 1788443095.41 total 1270.84 last 63.95) - open/close 35689 (time 1788443105.42 total 1280.85 last 69.62) - open/close 36332 (time 1788443115.44 total 1290.87 last 64.20) - open/close 37002 (time 1788443125.44 total 1300.87 last 66.99) - open/close 37591 (time 1788443135.45 total 1310.88 last 58.84) - open/close 38257 (time 1788443145.46 total 1320.89 last 66.50) - open/close 38989 (time 1788443155.47 total 1330.90 last 73.16) - open/close 39729 (time 1788443165.48 total 1340.91 last 73.93) - open/close 40000 (time 1788443169.26 total 1344.69 last 71.56) - open/close 40622 (time 1788443179.28 total 1354.71 last 62.13) - open/close 41371 (time 1788443189.29 total 1364.72 last 74.81) - open/close 42014 (time 1788443199.31 total 1374.74 last 64.17) - open/close 42681 (time 1788443209.31 total 1384.74 last 66.68) - open/close 43402 (time 1788443219.32 total 1394.75 last 72.06) - open/close 44104 (time 1788443229.33 total 1404.76 last 70.14) - open/close 44826 (time 1788443239.33 total 1414.76 last 72.18) - open/close 45473 (time 1788443249.33 total 1424.76 last 64.70) - open/close 46067 (time 1788443259.33 total 1434.76 last 59.36) - open/close 46693 (time 1788443269.37 total 1444.80 last 62.41) - open/close 47347 (time 1788443279.38 total 1454.81 last 65.34) - open/close 47890 (time 1788443289.40 total 1464.83 last 54.17) - open/close 48497 (time 1788443299.42 total 1474.85 last 60.61) - open/close 49028 (time 1788443309.43 total 1484.86 last 53.02) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1494.12 seconds: 33.22 ops/second - unlinked 0 (time 1788443324 ; total 0 ; last 0) - unlinked 10000 (time 1788443481 ; total 157 ; last 157) - unlinked 20000 (time 1788443625 ; total 301 ; last 144) - unlinked 30000 (time 1788443768 ; total 444 ; last 143) - unlinked 40000 (time 1788443915 ; total 591 ; last 147) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 726 seconds: 68.362259 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg434-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server unloading modules via unload_modules_local on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing unload_modules_local oleg434-server: kmemleak disabled modules unloaded. pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-client: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-client: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2