== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:01:36 (1787846496) start mds service on oleg106-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg106-server' oleg106-server: oleg106-server.virtnet: executing load_modules_local oleg106-server: Loading modules from /home/green/git/lustre-release/lustre oleg106-server: detected 4 online CPUs by sysfs oleg106-server: MODOPTS_LIBCFS= oleg106-server: Force libcfs to create 2 CPU partitions oleg106-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg106-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg106-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg106-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg106-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg106-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0000 oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 228 (time 1787846616.85 total 11.14 last 20.47) - open/close 441 (time 1787846629.07 total 23.36 last 17.42) - open/close 602 (time 1787846639.90 total 34.19 last 14.88) - open/close 833 (time 1787846649.93 total 44.22 last 23.02) - open/close 995 (time 1787846662.43 total 56.72 last 12.96) - open/close 1062 (time 1787846673.50 total 67.79 last 6.05) - open/close 1110 (time 1787846684.20 total 78.49 last 4.49) - open/close 1221 (time 1787846695.12 total 89.41 last 10.17) - open/close 1374 (time 1787846705.13 total 99.42 last 15.29) - open/close 1452 (time 1787846715.23 total 109.52 last 7.72) - open/close 1583 (time 1787846726.24 total 120.53 last 11.90) - open/close 1666 (time 1787846736.92 total 131.21 last 7.77) - open/close 1844 (time 1787846749.79 total 144.08 last 13.83) - open/close 2036 (time 1787846760.24 total 154.53 last 18.38) - open/close 2262 (time 1787846770.90 total 165.20 last 21.18) - open/close 2524 (time 1787846781.85 total 176.14 last 23.95) - open/close 2668 (time 1787846791.85 total 186.14 last 14.40) - open/close 2743 (time 1787846802.19 total 196.48 last 7.25) - open/close 2878 (time 1787846813.30 total 207.59 last 12.15) - open/close 3092 (time 1787846823.64 total 217.93 last 20.70) - open/close 3311 (time 1787846833.66 total 227.95 last 21.85) - open/close 3539 (time 1787846843.73 total 238.02 last 22.64) - open/close 3663 (time 1787846854.36 total 248.65 last 11.67) - open/close 3880 (time 1787846866.86 total 261.15 last 17.35) - open/close 3978 (time 1787846879.25 total 273.54 last 7.91) - open/close 4131 (time 1787846889.25 total 283.54 last 15.29) - open/close 4215 (time 1787846901.33 total 295.62 last 6.96) - open/close 4487 (time 1787846912.35 total 306.64 last 24.68) - open/close 4650 (time 1787846922.78 total 317.07 last 15.63) - open/close 4922 (time 1787846933.35 total 327.64 last 25.73) - open/close 5155 (time 1787846943.95 total 338.24 last 21.98) - open/close 5385 (time 1787846957.00 total 351.29 last 17.62) - open/close 5638 (time 1787846969.69 total 363.98 last 19.94) - open/close 5742 (time 1787846979.82 total 374.11 last 10.27) - open/close 5963 (time 1787846990.82 total 385.11 last 20.09) - open/close 6161 (time 1787847001.68 total 395.97 last 18.23) - open/close 6348 (time 1787847011.70 total 405.99 last 18.66) - open/close 6664 (time 1787847021.71 total 416.00 last 31.56) - open/close 6873 (time 1787847031.72 total 426.01 last 20.88) - open/close 7141 (time 1787847042.02 total 436.31 last 26.02) - open/close 7273 (time 1787847052.71 total 447.00 last 12.35) - open/close 7403 (time 1787847063.18 total 457.47 last 12.42) - open/close 7565 (time 1787847073.68 total 467.97 last 15.43) - open/close 7671 (time 1787847083.69 total 477.98 last 10.59) - open/close 7784 (time 1787847094.42 total 488.72 last 10.52) - open/close 7895 (time 1787847104.43 total 498.72 last 11.10) - open/close 8117 (time 1787847115.09 total 509.38 last 20.82) - open/close 8266 (time 1787847125.58 total 519.87 last 14.20) - open/close 8556 (time 1787847135.59 total 529.88 last 28.98) - open/close 8740 (time 1787847145.60 total 539.89 last 18.38) - open/close 9007 (time 1787847155.61 total 549.90 last 26.67) - open/close 9296 (time 1787847165.62 total 559.91 last 28.87) - open/close 9540 (time 1787847176.48 total 570.77 last 22.49) - open/close 9829 (time 1787847186.61 total 580.90 last 28.52) - open/close 10000 (time 1787847194.77 total 589.06 last 20.95) - open/close 10197 (time 1787847207.46 total 601.75 last 15.52) - open/close 10421 (time 1787847218.28 total 612.57 last 20.70) - open/close 10691 (time 1787847229.16 total 623.45 last 24.82) - open/close 10904 (time 1787847240.10 total 634.39 last 19.48) - open/close 11133 (time 1787847250.62 total 644.91 last 21.76) - open/close 11359 (time 1787847261.42 total 655.71 last 20.93) - open/close 11449 (time 1787847272.10 total 666.39 last 8.43) - open/close 11583 (time 1787847284.94 total 679.23 last 10.43) - open/close 11787 (time 1787847294.95 total 689.24 last 20.39) - open/close 11934 (time 1787847306.02 total 700.31 last 13.28) - open/close 12096 (time 1787847316.09 total 710.38 last 16.08) - open/close 12315 (time 1787847326.10 total 720.39 last 21.89) - open/close 12486 (time 1787847336.22 total 730.51 last 16.90) - open/close 12766 (time 1787847346.22 total 740.52 last 27.98) - open/close 12951 (time 1787847356.32 total 750.61 last 18.32) - open/close 13187 (time 1787847366.32 total 760.61 last 23.60) - open/close 13445 (time 1787847376.34 total 770.63 last 25.77) - open/close 13567 (time 1787847386.93 total 781.22 last 11.51) - open/close 13871 (time 1787847397.99 total 792.28 last 27.50) - open/close 14050 (time 1787847408.14 total 802.43 last 17.62) - open/close 14200 (time 1787847418.15 total 812.44 last 14.99) - open/close 14340 (time 1787847428.15 total 822.45 last 13.99) - open/close 14587 (time 1787847438.17 total 832.46 last 24.67) - open/close 14866 (time 1787847449.11 total 843.40 last 25.50) - open/close 15104 (time 1787847459.80 total 854.09 last 22.26) - open/close 15413 (time 1787847470.56 total 864.85 last 28.71) - open/close 15691 (time 1787847481.49 total 875.78 last 25.44) - open/close 15916 (time 1787847491.50 total 885.79 last 22.48) - open/close 16195 (time 1787847502.29 total 896.58 last 25.86) - open/close 16466 (time 1787847513.37 total 907.66 last 24.45) - open/close 16701 (time 1787847523.40 total 917.69 last 23.44) - open/close 16979 (time 1787847533.66 total 927.95 last 27.10) - open/close 17209 (time 1787847543.67 total 937.96 last 22.97) - open/close 17433 (time 1787847553.68 total 947.97 last 22.38) - open/close 17670 (time 1787847563.89 total 958.18 last 23.21) - open/close 17928 (time 1787847574.16 total 968.45 last 25.11) - open/close 18228 (time 1787847584.73 total 979.02 last 28.40) - open/close 18442 (time 1787847594.73 total 989.02 last 21.39) - open/close 18671 (time 1787847604.82 total 999.11 last 22.70) - open/close 18941 (time 1787847615.35 total 1009.64 last 25.63) - open/close 19683 (time 1787847625.35 total 1019.64 last 74.19) - open/close 20000 (time 1787847628.65 total 1022.94 last 96.14) - open/close 20967 (time 1787847638.66 total 1032.95 last 96.64) - open/close 22090 (time 1787847648.66 total 1042.95 last 112.20) - open/close 22971 (time 1787847658.69 total 1052.98 last 87.91) - open/close 23947 (time 1787847668.69 total 1062.98 last 97.53) - open/close 24936 (time 1787847678.70 total 1072.99 last 98.83) - open/close 25963 (time 1787847688.71 total 1083.00 last 102.60) - open/close 26891 (time 1787847698.71 total 1093.01 last 92.76) - open/close 27936 (time 1787847708.72 total 1103.01 last 104.48) - open/close 28884 (time 1787847718.72 total 1113.01 last 94.79) - open/close 29924 (time 1787847728.72 total 1123.01 last 103.99) - open/close 30866 (time 1787847738.72 total 1133.02 last 94.15) - open/close 31756 (time 1787847748.73 total 1143.02 last 88.99) - open/close 32797 (time 1787847758.73 total 1153.02 last 104.07) - open/close 33816 (time 1787847768.73 total 1163.02 last 101.85) - open/close 34842 (time 1787847778.74 total 1173.03 last 102.54) - open/close 35800 (time 1787847788.75 total 1183.04 last 95.74) - open/close 36700 (time 1787847798.75 total 1193.04 last 89.99) - open/close 37671 (time 1787847808.76 total 1203.05 last 96.98) - open/close 38583 (time 1787847818.77 total 1213.06 last 91.15) - open/close 39432 (time 1787847828.77 total 1223.06 last 84.85) - open/close 40000 (time 1787847836.80 total 1231.09 last 70.73) - open/close 40899 (time 1787847846.81 total 1241.10 last 89.85) - open/close 41632 (time 1787847856.82 total 1251.11 last 73.21) - open/close 42626 (time 1787847866.83 total 1261.12 last 99.32) - open/close 43560 (time 1787847876.83 total 1271.12 last 93.36) - open/close 44251 (time 1787847886.84 total 1281.13 last 69.04) - open/close 45087 (time 1787847896.85 total 1291.14 last 83.55) - open/close 45857 (time 1787847906.85 total 1301.14 last 77.00) - open/close 46492 (time 1787847916.85 total 1311.14 last 63.49) - open/close 47276 (time 1787847926.85 total 1321.14 last 78.37) - open/close 48078 (time 1787847936.88 total 1331.17 last 79.98) - open/close 48880 (time 1787847946.91 total 1341.20 last 80.00) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1350.17 seconds: 36.76 ops/second - unlinked 0 (time 1787847961 ; total 0 ; last 0) - unlinked 10000 (time 1787848092 ; total 131 ; last 131) - unlinked 20000 (time 1787848203 ; total 242 ; last 111) - unlinked 30000 (time 1787848267 ; total 306 ; last 64) - unlinked 40000 (time 1787848352 ; total 391 ; last 85) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 460 seconds: 107.893478 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg106-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg106-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg106-server stop mds service on oleg106-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg106-server stop mds service on oleg106-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg106-server unloading modules via unload_modules_local on: 'oleg106-server' oleg106-server: oleg106-server.virtnet: executing unload_modules_local oleg106-server: modules unloaded. pdsh@oleg106-client: oleg106-server: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-client: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-client: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-server: ssh exited with exit code 2