== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:51:22 (1787860282) start mds service on oleg625-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg625-server' oleg625-server: oleg625-server.virtnet: executing load_modules_local oleg625-server: Loading modules from /home/green/git/lustre-release/lustre oleg625-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg625-server: detected 4 online CPUs by sysfs oleg625-server: MODOPTS_LIBCFS= oleg625-server: Force libcfs to create 2 CPU partitions oleg625-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg625-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg625-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg625-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg625-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg625-server: oleg625-server.virtnet: executing set_default_debug -1 all pdsh@oleg625-client: oleg625-server: ssh exited with exit code 1 Started lustre-OST0000 oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg625-client.virtnet: -o user_xattr,flock 192.168.206.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 256 (time 1787860382.39 total 10.00 last 25.60) - open/close 521 (time 1787860392.91 total 20.52 last 25.19) - open/close 799 (time 1787860402.91 total 30.52 last 27.80) - open/close 1014 (time 1787860412.92 total 40.53 last 21.48) - open/close 1309 (time 1787860423.26 total 50.87 last 28.54) - open/close 1595 (time 1787860433.36 total 60.97 last 28.31) - open/close 1863 (time 1787860443.36 total 70.97 last 26.79) - open/close 2153 (time 1787860453.37 total 80.98 last 28.97) - open/close 2430 (time 1787860464.39 total 92.01 last 25.13) - open/close 2706 (time 1787860474.77 total 102.39 last 26.59) - open/close 2965 (time 1787860484.79 total 112.41 last 25.85) - open/close 3209 (time 1787860495.23 total 122.84 last 23.39) - open/close 3516 (time 1787860505.24 total 132.85 last 30.66) - open/close 3740 (time 1787860515.73 total 143.34 last 21.35) - open/close 3963 (time 1787860526.74 total 154.35 last 20.26) - open/close 4261 (time 1787860537.37 total 164.98 last 28.04) - open/close 4543 (time 1787860548.17 total 175.79 last 26.09) - open/close 4823 (time 1787860558.60 total 186.21 last 26.85) - open/close 5109 (time 1787860568.60 total 196.22 last 28.59) - open/close 5368 (time 1787860578.96 total 206.58 last 25.00) - open/close 5644 (time 1787860588.97 total 216.58 last 27.59) - open/close 5892 (time 1787860598.97 total 226.58 last 24.79) - open/close 6123 (time 1787860609.06 total 236.67 last 22.91) - open/close 6387 (time 1787860619.07 total 246.68 last 26.37) - open/close 6628 (time 1787860629.32 total 256.94 last 23.50) - open/close 6907 (time 1787860639.82 total 267.43 last 26.58) - open/close 7141 (time 1787860649.86 total 277.48 last 23.30) - open/close 7380 (time 1787860660.72 total 288.33 last 22.03) - open/close 7631 (time 1787860670.95 total 298.56 last 24.53) - open/close 7811 (time 1787860681.63 total 309.24 last 16.85) - open/close 8016 (time 1787860691.64 total 319.25 last 20.48) - open/close 8224 (time 1787860702.16 total 329.78 last 19.76) - open/close 8432 (time 1787860712.18 total 339.80 last 20.76) - open/close 8683 (time 1787860722.33 total 349.95 last 24.73) - open/close 8925 (time 1787860732.75 total 360.37 last 23.23) - open/close 9155 (time 1787860743.64 total 371.25 last 21.13) - open/close 9458 (time 1787860753.64 total 381.25 last 30.28) - open/close 9693 (time 1787860764.05 total 391.66 last 22.58) - open/close 9991 (time 1787860774.10 total 401.71 last 29.66) - open/close 10250 (time 1787860784.46 total 412.07 last 24.99) - open/close 10465 (time 1787860794.46 total 422.08 last 21.49) - open/close 10682 (time 1787860805.33 total 432.95 last 19.96) - open/close 10835 (time 1787860815.38 total 442.99 last 15.24) - open/close 10983 (time 1787860825.83 total 453.44 last 14.16) - open/close 11111 (time 1787860835.83 total 463.45 last 12.79) - open/close 11285 (time 1787860846.93 total 474.55 last 15.68) - open/close 11602 (time 1787860857.70 total 485.31 last 29.45) - open/close 11693 (time 1787860867.70 total 495.31 last 9.10) - open/close 11774 (time 1787860877.71 total 505.33 last 8.09) - open/close 12005 (time 1787860888.17 total 515.78 last 22.09) - open/close 12254 (time 1787860898.32 total 525.94 last 24.53) - open/close 12543 (time 1787860909.09 total 536.71 last 26.84) - open/close 12777 (time 1787860919.12 total 546.73 last 23.35) - open/close 13038 (time 1787860929.12 total 556.73 last 26.09) - open/close 13278 (time 1787860939.15 total 566.76 last 23.94) - open/close 13501 (time 1787860950.03 total 577.64 last 20.49) - open/close 13726 (time 1787860961.04 total 588.65 last 20.44) - open/close 13972 (time 1787860971.15 total 598.76 last 24.34) - open/close 14264 (time 1787860981.53 total 609.15 last 28.12) - open/close 14548 (time 1787860992.18 total 619.79 last 26.69) - open/close 14837 (time 1787861002.84 total 630.45 last 27.11) - open/close 15134 (time 1787861013.59 total 641.20 last 27.63) - open/close 15384 (time 1787861023.61 total 651.22 last 24.94) - open/close 15612 (time 1787861035.66 total 663.28 last 18.91) - open/close 15811 (time 1787861046.24 total 673.85 last 18.81) - open/close 16047 (time 1787861056.26 total 683.87 last 23.56) - open/close 16242 (time 1787861067.10 total 694.71 last 17.99) - open/close 16516 (time 1787861077.11 total 704.72 last 27.37) - open/close 16804 (time 1787861087.96 total 715.57 last 26.55) - open/close 17074 (time 1787861098.81 total 726.42 last 24.89) - open/close 17354 (time 1787861109.85 total 737.46 last 25.36) - open/close 17626 (time 1787861120.02 total 747.64 last 26.73) - open/close 17906 (time 1787861130.20 total 757.82 last 27.51) - open/close 18157 (time 1787861140.30 total 767.92 last 24.85) - open/close 18414 (time 1787861150.42 total 778.04 last 25.39) - open/close 18687 (time 1787861160.94 total 788.55 last 25.97) - open/close 18982 (time 1787861171.28 total 798.89 last 28.53) - open/close 19279 (time 1787861181.60 total 809.21 last 28.77) - open/close 19533 (time 1787861191.89 total 819.50 last 24.69) - open/close 19836 (time 1787861202.86 total 830.47 last 27.62) - open/close 20000 (time 1787861208.72 total 836.33 last 27.98) - open/close 20282 (time 1787861219.50 total 847.12 last 26.15) - open/close 20542 (time 1787861230.36 total 857.97 last 23.95) - open/close 20862 (time 1787861240.36 total 867.97 last 32.00) - open/close 21146 (time 1787861250.81 total 878.42 last 27.18) - open/close 21464 (time 1787861260.81 total 888.43 last 31.79) - open/close 21709 (time 1787861270.82 total 898.44 last 24.48) - open/close 22268 (time 1787861280.82 total 908.44 last 55.90) - open/close 23276 (time 1787861290.83 total 918.44 last 100.76) - open/close 24303 (time 1787861300.83 total 928.44 last 102.65) - open/close 25305 (time 1787861310.84 total 938.45 last 100.14) - open/close 26312 (time 1787861320.84 total 948.46 last 100.64) - open/close 27266 (time 1787861330.84 total 958.46 last 95.39) - open/close 28301 (time 1787861340.85 total 968.47 last 103.41) - open/close 29339 (time 1787861350.87 total 978.48 last 103.66) - open/close 30000 (time 1787861357.51 total 985.12 last 99.54) - open/close 30959 (time 1787861367.52 total 995.13 last 95.76) - open/close 31932 (time 1787861377.52 total 1005.14 last 97.27) - open/close 32874 (time 1787861387.53 total 1015.14 last 94.13) - open/close 33898 (time 1787861397.53 total 1025.14 last 102.39) - open/close 34822 (time 1787861407.54 total 1035.15 last 92.34) - open/close 35814 (time 1787861417.54 total 1045.15 last 99.17) - open/close 36810 (time 1787861427.56 total 1055.17 last 99.40) - open/close 37776 (time 1787861437.58 total 1065.19 last 96.44) - open/close 38672 (time 1787861447.58 total 1075.20 last 89.54) - open/close 39584 (time 1787861457.59 total 1085.20 last 91.16) - open/close 40000 (time 1787861462.11 total 1089.72 last 91.97) - open/close 40984 (time 1787861472.12 total 1099.73 last 98.34) - open/close 41963 (time 1787861482.12 total 1109.73 last 97.89) - open/close 42965 (time 1787861492.13 total 1119.74 last 100.09) - open/close 43923 (time 1787861502.14 total 1129.75 last 95.71) - open/close 44909 (time 1787861512.14 total 1139.76 last 98.54) - open/close 45868 (time 1787861522.15 total 1149.77 last 95.81) - open/close 46812 (time 1787861532.15 total 1159.77 last 94.40) - open/close 47800 (time 1787861542.16 total 1169.77 last 98.75) - open/close 48766 (time 1787861552.16 total 1179.77 last 96.58) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1188.71 seconds: 41.75 ops/second - unlinked 0 (time 1787861565 ; total 0 ; last 0) - unlinked 10000 (time 1787861675 ; total 110 ; last 110) - unlinked 20000 (time 1787861777 ; total 212 ; last 102) - unlinked 30000 (time 1787861837 ; total 272 ; last 60) - unlinked 40000 (time 1787861897 ; total 332 ; last 60) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 389 seconds: 127.586121 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg625-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg625-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg625-server stop mds service on oleg625-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg625-server stop mds service on oleg625-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg625-server unloading modules via unload_modules_local on: 'oleg625-server' oleg625-server: oleg625-server.virtnet: executing unload_modules_local oleg625-server: modules unloaded. pdsh@oleg625-client: oleg625-client: ssh exited with exit code 2 pdsh@oleg625-client: oleg625-server: ssh exited with exit code 2 pdsh@oleg625-client: oleg625-client: ssh exited with exit code 2 pdsh@oleg625-client: oleg625-server: ssh exited with exit code 2