== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:36:53 (1781588213) start mds service on oleg323-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg323-server: detected 4 online CPUs by sysfs oleg323-server: MODOPTS_LIBCFS= oleg323-server: Force libcfs to create 2 CPU partitions oleg323-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg323-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg323-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg323-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg323-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 225 (time 1781588317.65 total 10.02 last 22.47) - open/close 427 (time 1781588327.65 total 20.02 last 20.20) - open/close 634 (time 1781588337.66 total 30.03 last 20.68) - open/close 823 (time 1781588347.70 total 40.06 last 18.84) - open/close 1032 (time 1781588357.70 total 50.07 last 20.88) - open/close 1234 (time 1781588370.68 total 63.05 last 15.57) - open/close 1446 (time 1781588380.68 total 73.05 last 21.20) - open/close 1675 (time 1781588391.99 total 84.35 last 20.26) - open/close 1928 (time 1781588402.24 total 94.61 last 24.67) - open/close 2127 (time 1781588412.25 total 104.61 last 19.89) - open/close 2273 (time 1781588422.33 total 114.69 last 14.48) - open/close 2540 (time 1781588432.33 total 124.69 last 26.70) - open/close 2749 (time 1781588442.33 total 134.70 last 20.89) - open/close 3027 (time 1781588452.59 total 144.96 last 27.10) - open/close 3315 (time 1781588462.85 total 155.21 last 28.08) - open/close 3601 (time 1781588473.14 total 165.51 last 27.79) - open/close 3830 (time 1781588483.79 total 176.16 last 21.50) - open/close 4077 (time 1781588493.79 total 186.16 last 24.70) - open/close 4298 (time 1781588504.50 total 196.86 last 20.65) - open/close 4438 (time 1781588514.50 total 206.87 last 13.99) - open/close 4694 (time 1781588525.49 total 217.86 last 23.30) - open/close 4779 (time 1781588535.49 total 227.86 last 8.50) - open/close 4944 (time 1781588547.71 total 240.08 last 13.50) - open/close 5210 (time 1781588557.75 total 250.12 last 26.49) - open/close 5443 (time 1781588570.35 total 262.72 last 18.49) - open/close 5692 (time 1781588580.98 total 273.34 last 23.44) - open/close 5934 (time 1781588591.28 total 283.64 last 23.49) - open/close 6176 (time 1781588601.60 total 293.96 last 23.46) - open/close 6420 (time 1781588611.60 total 303.96 last 24.39) - open/close 6645 (time 1781588621.68 total 314.04 last 22.33) - open/close 6897 (time 1781588632.62 total 324.99 last 23.02) - open/close 7134 (time 1781588642.63 total 334.99 last 23.69) - open/close 7374 (time 1781588652.66 total 345.03 last 23.92) - open/close 7609 (time 1781588663.29 total 355.66 last 22.11) - open/close 7734 (time 1781588674.05 total 366.42 last 11.61) - open/close 8017 (time 1781588684.06 total 376.43 last 28.28) - open/close 8264 (time 1781588694.06 total 386.43 last 24.69) - open/close 8471 (time 1781588704.18 total 396.54 last 20.47) - open/close 8719 (time 1781588714.62 total 406.99 last 23.74) - open/close 8962 (time 1781588724.63 total 416.99 last 24.29) - open/close 9189 (time 1781588735.47 total 427.83 last 20.94) - open/close 9395 (time 1781588745.92 total 438.28 last 19.72) - open/close 9654 (time 1781588756.73 total 449.10 last 23.95) - open/close 9938 (time 1781588767.46 total 459.83 last 26.46) - open/close 10183 (time 1781588777.84 total 470.20 last 23.61) - open/close 10443 (time 1781588787.84 total 480.21 last 25.99) - open/close 10652 (time 1781588800.01 total 492.37 last 17.18) - open/close 10906 (time 1781588810.02 total 502.39 last 25.36) - open/close 11128 (time 1781588820.35 total 512.72 last 21.50) - open/close 11339 (time 1781588831.42 total 523.79 last 19.06) - open/close 11648 (time 1781588842.35 total 534.71 last 28.27) - open/close 11885 (time 1781588852.92 total 545.29 last 22.41) - open/close 12130 (time 1781588863.03 total 555.40 last 24.24) - open/close 12363 (time 1781588873.04 total 565.40 last 23.29) - open/close 12583 (time 1781588883.33 total 575.70 last 21.37) - open/close 12772 (time 1781588893.75 total 586.11 last 18.15) - open/close 12906 (time 1781588903.75 total 596.11 last 13.40) - open/close 13050 (time 1781588914.93 total 607.30 last 12.87) - open/close 13329 (time 1781588925.02 total 617.39 last 27.65) - open/close 13534 (time 1781588935.29 total 627.65 last 19.97) - open/close 13810 (time 1781588945.29 total 637.66 last 27.59) - open/close 14047 (time 1781588955.76 total 648.12 last 22.65) - open/close 14306 (time 1781588965.76 total 658.13 last 25.88) - open/close 14532 (time 1781588975.77 total 668.14 last 22.58) - open/close 14769 (time 1781588985.81 total 678.17 last 23.62) - open/close 15037 (time 1781588996.39 total 688.76 last 25.31) - open/close 15264 (time 1781589006.51 total 698.88 last 22.44) - open/close 15551 (time 1781589017.22 total 709.59 last 26.80) - open/close 15791 (time 1781589027.25 total 719.61 last 23.93) - open/close 15972 (time 1781589037.26 total 729.62 last 18.08) - open/close 16230 (time 1781589047.27 total 739.64 last 25.76) - open/close 16423 (time 1781589057.46 total 749.83 last 18.94) - open/close 16661 (time 1781589067.47 total 759.84 last 23.78) - open/close 16794 (time 1781589077.74 total 770.10 last 12.95) - open/close 16999 (time 1781589088.19 total 780.56 last 19.61) - open/close 17247 (time 1781589098.61 total 790.98 last 23.81) - open/close 17464 (time 1781589108.67 total 801.03 last 21.58) - open/close 17722 (time 1781589119.43 total 811.80 last 23.97) - open/close 17966 (time 1781589129.46 total 821.83 last 24.33) - open/close 18190 (time 1781589140.42 total 832.78 last 20.44) - open/close 18469 (time 1781589150.42 total 842.79 last 27.89) - open/close 18692 (time 1781589160.43 total 852.79 last 22.28) - open/close 18932 (time 1781589170.76 total 863.13 last 23.23) - open/close 19195 (time 1781589180.79 total 873.15 last 26.23) - open/close 19408 (time 1781589190.84 total 883.20 last 21.20) - open/close 19609 (time 1781589201.04 total 893.41 last 19.70) - open/close 20000 (time 1781589207.52 total 899.89 last 60.35) - open/close 20693 (time 1781589217.53 total 909.90 last 69.22) - open/close 21357 (time 1781589227.56 total 919.92 last 66.22) - open/close 21955 (time 1781589237.57 total 929.94 last 59.72) - open/close 22689 (time 1781589247.58 total 939.95 last 73.32) - open/close 23417 (time 1781589257.60 total 949.96 last 72.72) - open/close 24170 (time 1781589267.61 total 959.97 last 75.23) - open/close 24865 (time 1781589277.62 total 969.99 last 69.38) - open/close 25626 (time 1781589287.65 total 980.01 last 75.90) - open/close 26352 (time 1781589297.66 total 990.03 last 72.51) - open/close 27082 (time 1781589307.66 total 1000.03 last 72.99) - open/close 27815 (time 1781589317.67 total 1010.04 last 73.25) - open/close 28534 (time 1781589327.67 total 1020.04 last 71.89) - open/close 29221 (time 1781589337.69 total 1030.05 last 68.61) - open/close 29918 (time 1781589347.69 total 1040.05 last 69.67) - open/close 30655 (time 1781589357.69 total 1050.06 last 73.67) - open/close 31350 (time 1781589367.70 total 1060.06 last 69.46) - open/close 32084 (time 1781589377.72 total 1070.09 last 73.25) - open/close 32764 (time 1781589387.74 total 1080.10 last 67.87) - open/close 33430 (time 1781589397.75 total 1090.12 last 66.52) - open/close 34111 (time 1781589407.79 total 1100.16 last 67.82) - open/close 34912 (time 1781589417.80 total 1110.16 last 80.07) - open/close 35593 (time 1781589427.81 total 1120.17 last 68.02) - open/close 36261 (time 1781589437.82 total 1130.18 last 66.73) - open/close 37014 (time 1781589447.82 total 1140.19 last 75.28) - open/close 37728 (time 1781589457.82 total 1150.19 last 71.37) - open/close 38495 (time 1781589467.83 total 1160.20 last 76.66) - open/close 39218 (time 1781589477.83 total 1170.20 last 72.30) - open/close 39980 (time 1781589487.84 total 1180.21 last 76.10) - open/close 40732 (time 1781589497.86 total 1190.22 last 75.11) - open/close 41477 (time 1781589507.86 total 1200.22 last 74.50) - open/close 42245 (time 1781589517.86 total 1210.22 last 76.80) - open/close 42995 (time 1781589527.86 total 1220.23 last 74.94) - open/close 43808 (time 1781589537.87 total 1230.24 last 81.22) - open/close 44579 (time 1781589547.89 total 1240.26 last 76.98) - open/close 45338 (time 1781589557.89 total 1250.26 last 75.87) - open/close 46022 (time 1781589567.91 total 1260.28 last 68.28) - open/close 46744 (time 1781589577.92 total 1270.29 last 72.12) - open/close 47424 (time 1781589587.92 total 1280.29 last 67.99) - open/close 48079 (time 1781589597.93 total 1290.29 last 65.47) - open/close 48822 (time 1781589607.95 total 1300.32 last 74.12) - open/close 49497 (time 1781589617.95 total 1310.32 last 67.49) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1312.08 seconds: 37.83 ops/second - unlinked 0 (time 1781589623 ; total 0 ; last 0) - unlinked 10000 (time 1781589770 ; total 147 ; last 147) - unlinked 20000 (time 1781589914 ; total 291 ; last 144) - unlinked 30000 (time 1781590060 ; total 437 ; last 146) - unlinked 40000 (time 1781590204 ; total 581 ; last 144) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 713 seconds: 69.608696 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg323-server unloading modules via unload_modules_local on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing unload_modules_local oleg323-server: modules unloaded. pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2