== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:21:04 (1787563264) start mds service on oleg653-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg653-server' oleg653-server: oleg653-server.virtnet: executing load_modules_local oleg653-server: Loading modules from /home/green/git/lustre-release/lustre oleg653-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg653-server: detected 4 online CPUs by sysfs oleg653-server: MODOPTS_LIBCFS= oleg653-server: Force libcfs to create 2 CPU partitions oleg653-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg653-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg653-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg653-server: oleg653-server.virtnet: executing set_default_debug -1 all pdsh@oleg653-client: oleg653-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg653-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg653-server: oleg653-server.virtnet: executing set_default_debug -1 all pdsh@oleg653-client: oleg653-server: ssh exited with exit code 1 Started lustre-OST0000 oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 231 (time 1787563351.53 total 10.20 last 22.65) - open/close 452 (time 1787563361.53 total 20.20 last 22.09) - open/close 691 (time 1787563372.41 total 31.08 last 21.97) - open/close 802 (time 1787563382.42 total 41.09 last 11.08) - open/close 957 (time 1787563392.46 total 51.13 last 15.44) - open/close 1079 (time 1787563403.09 total 61.76 last 11.48) - open/close 1207 (time 1787563414.98 total 73.65 last 10.76) - open/close 1421 (time 1787563424.99 total 83.65 last 21.40) - open/close 1641 (time 1787563435.06 total 93.73 last 21.84) - open/close 1789 (time 1787563445.17 total 103.84 last 14.64) - open/close 1973 (time 1787563455.79 total 114.46 last 17.33) - open/close 2217 (time 1787563466.16 total 124.82 last 23.54) - open/close 2414 (time 1787563476.16 total 134.83 last 19.69) - open/close 2675 (time 1787563486.17 total 144.84 last 26.06) - open/close 2908 (time 1787563496.80 total 155.47 last 21.93) - open/close 3122 (time 1787563508.08 total 166.74 last 18.98) - open/close 3390 (time 1787563518.77 total 177.44 last 25.06) - open/close 3583 (time 1787563529.48 total 188.15 last 18.03) - open/close 3833 (time 1787563539.51 total 198.17 last 24.93) - open/close 3913 (time 1787563549.51 total 208.18 last 8.00) - open/close 3995 (time 1787563560.44 total 219.11 last 7.50) - open/close 4140 (time 1787563570.45 total 229.12 last 14.49) - open/close 4340 (time 1787563580.45 total 239.12 last 20.00) - open/close 4572 (time 1787563591.35 total 250.02 last 21.29) - open/close 4822 (time 1787563604.20 total 262.87 last 19.46) - open/close 5065 (time 1787563614.21 total 272.88 last 24.28) - open/close 5220 (time 1787563624.44 total 283.11 last 15.15) - open/close 5337 (time 1787563634.44 total 293.11 last 11.70) - open/close 5511 (time 1787563644.87 total 303.54 last 16.68) - open/close 5634 (time 1787563655.34 total 314.01 last 11.75) - open/close 5733 (time 1787563665.45 total 324.12 last 9.79) - open/close 5807 (time 1787563675.58 total 334.24 last 7.31) - open/close 5968 (time 1787563685.59 total 344.26 last 16.07) - open/close 6097 (time 1787563695.82 total 354.49 last 12.61) - open/close 6267 (time 1787563705.83 total 364.49 last 16.99) - open/close 6501 (time 1787563715.85 total 374.51 last 23.35) - open/close 6662 (time 1787563726.79 total 385.46 last 14.71) - open/close 6887 (time 1787563737.67 total 396.34 last 20.69) - open/close 7128 (time 1787563747.69 total 406.36 last 24.04) - open/close 7321 (time 1787563757.99 total 416.66 last 18.74) - open/close 7506 (time 1787563768.81 total 427.48 last 17.10) - open/close 7712 (time 1787563778.82 total 437.49 last 20.58) - open/close 7927 (time 1787563788.85 total 447.52 last 21.43) - open/close 8104 (time 1787563799.34 total 458.01 last 16.88) - open/close 8317 (time 1787563809.59 total 468.26 last 20.77) - open/close 8530 (time 1787563819.92 total 478.59 last 20.62) - open/close 8771 (time 1787563829.93 total 488.59 last 24.09) - open/close 8966 (time 1787563839.99 total 498.66 last 19.38) - open/close 9112 (time 1787563850.00 total 508.67 last 14.58) - open/close 9371 (time 1787563860.28 total 518.95 last 25.19) - open/close 9630 (time 1787563870.29 total 528.95 last 25.89) - open/close 9853 (time 1787563880.58 total 539.25 last 21.66) - open/close 10000 (time 1787563887.27 total 545.93 last 21.99) - open/close 10218 (time 1787563897.68 total 556.34 last 20.95) - open/close 10483 (time 1787563908.07 total 566.74 last 25.49) - open/close 10741 (time 1787563918.83 total 577.50 last 23.98) - open/close 10943 (time 1787563928.84 total 587.51 last 20.18) - open/close 11179 (time 1787563939.38 total 598.05 last 22.39) - open/close 11331 (time 1787563949.95 total 608.62 last 14.38) - open/close 11408 (time 1787563960.91 total 619.58 last 7.03) - open/close 11527 (time 1787563970.93 total 629.59 last 11.88) - open/close 11742 (time 1787563981.31 total 639.98 last 20.70) - open/close 11955 (time 1787563991.34 total 650.01 last 21.25) - open/close 12143 (time 1787564003.65 total 662.32 last 15.27) - open/close 12380 (time 1787564014.13 total 672.79 last 22.62) - open/close 12531 (time 1787564025.09 total 683.76 last 13.77) - open/close 12766 (time 1787564035.10 total 693.76 last 23.49) - open/close 12965 (time 1787564045.63 total 704.30 last 18.89) - open/close 13081 (time 1787564055.65 total 714.31 last 11.58) - open/close 13155 (time 1787564065.97 total 724.64 last 7.16) - open/close 13344 (time 1787564078.22 total 736.89 last 15.43) - open/close 13523 (time 1787564089.27 total 747.94 last 16.21) - open/close 13675 (time 1787564099.59 total 758.26 last 14.72) - open/close 13928 (time 1787564110.67 total 769.33 last 22.85) - open/close 14120 (time 1787564120.68 total 779.35 last 19.17) - open/close 14375 (time 1787564130.69 total 789.36 last 25.47) - open/close 14580 (time 1787564141.35 total 800.02 last 19.23) - open/close 14768 (time 1787564151.37 total 810.04 last 18.76) - open/close 14998 (time 1787564162.04 total 820.70 last 21.56) - open/close 15197 (time 1787564172.79 total 831.46 last 18.50) - open/close 15453 (time 1787564183.54 total 842.21 last 23.81) - open/close 15695 (time 1787564194.18 total 852.85 last 22.74) - open/close 15961 (time 1787564204.20 total 862.87 last 26.56) - open/close 16189 (time 1787564214.86 total 873.52 last 21.40) - open/close 16390 (time 1787564224.87 total 883.54 last 20.07) - open/close 16613 (time 1787564234.88 total 893.55 last 22.28) - open/close 16852 (time 1787564245.88 total 904.54 last 21.73) - open/close 16988 (time 1787564255.89 total 914.56 last 13.58) - open/close 17155 (time 1787564266.04 total 924.70 last 16.46) - open/close 17414 (time 1787564276.04 total 934.70 last 25.90) - open/close 17568 (time 1787564286.04 total 944.71 last 15.39) - open/close 17787 (time 1787564298.97 total 957.63 last 16.95) - open/close 18009 (time 1787564309.61 total 968.28 last 20.86) - open/close 18195 (time 1787564320.70 total 979.36 last 16.77) - open/close 18326 (time 1787564330.86 total 989.52 last 12.89) - open/close 18982 (time 1787564340.87 total 999.53 last 65.54) - open/close 19647 (time 1787564350.88 total 1009.55 last 66.41) - open/close 20000 (time 1787564356.44 total 1015.10 last 63.53) - open/close 20689 (time 1787564366.45 total 1025.12 last 68.80) - open/close 21441 (time 1787564376.46 total 1035.12 last 75.16) - open/close 22148 (time 1787564386.46 total 1045.13 last 70.65) - open/close 22783 (time 1787564396.48 total 1055.15 last 63.36) - open/close 23435 (time 1787564406.49 total 1065.16 last 65.14) - open/close 24076 (time 1787564416.50 total 1075.17 last 64.05) - open/close 24673 (time 1787564426.51 total 1085.18 last 59.63) - open/close 25348 (time 1787564436.53 total 1095.19 last 67.40) - open/close 26026 (time 1787564446.54 total 1105.21 last 67.71) - open/close 26656 (time 1787564456.54 total 1115.21 last 63.00) - open/close 27322 (time 1787564466.54 total 1125.21 last 66.57) - open/close 28060 (time 1787564476.55 total 1135.22 last 73.76) - open/close 28795 (time 1787564486.55 total 1145.22 last 73.46) - open/close 29527 (time 1787564496.56 total 1155.23 last 73.18) - open/close 30000 (time 1787564503.13 total 1161.79 last 72.01) - open/close 30694 (time 1787564513.14 total 1171.81 last 69.30) - open/close 31421 (time 1787564523.15 total 1181.82 last 72.61) - open/close 32114 (time 1787564533.17 total 1191.84 last 69.16) - open/close 32807 (time 1787564543.18 total 1201.84 last 69.28) - open/close 33485 (time 1787564553.19 total 1211.85 last 67.73) - open/close 34128 (time 1787564563.26 total 1221.93 last 63.83) - open/close 34850 (time 1787564573.27 total 1231.93 last 72.15) - open/close 35547 (time 1787564583.27 total 1241.93 last 69.70) - open/close 36201 (time 1787564593.27 total 1251.94 last 65.39) - open/close 36844 (time 1787564603.27 total 1261.94 last 64.28) - open/close 37495 (time 1787564613.29 total 1271.95 last 65.00) - open/close 38221 (time 1787564623.32 total 1281.98 last 72.39) - open/close 38842 (time 1787564633.32 total 1291.98 last 62.10) - open/close 39504 (time 1787564643.33 total 1302.00 last 66.10) - open/close 40000 (time 1787564651.41 total 1310.08 last 61.40) - open/close 40588 (time 1787564661.41 total 1320.08 last 58.79) - open/close 41293 (time 1787564671.43 total 1330.10 last 70.37) - open/close 41980 (time 1787564681.44 total 1340.11 last 68.64) - open/close 42628 (time 1787564691.45 total 1350.12 last 64.71) - open/close 43303 (time 1787564701.46 total 1360.13 last 67.42) - open/close 43956 (time 1787564711.47 total 1370.14 last 65.27) - open/close 44659 (time 1787564721.48 total 1380.15 last 70.20) - open/close 45333 (time 1787564731.49 total 1390.16 last 67.33) - open/close 46033 (time 1787564741.50 total 1400.17 last 69.95) - open/close 46720 (time 1787564751.51 total 1410.18 last 68.63) - open/close 47357 (time 1787564761.52 total 1420.19 last 63.63) - open/close 47999 (time 1787564771.53 total 1430.19 last 64.16) - open/close 48654 (time 1787564781.53 total 1440.20 last 65.47) - open/close 49367 (time 1787564791.54 total 1450.21 last 71.21) total: 50000 open/close in 1459.53 seconds: 34.26 ops/second - unlinked 0 (time 1787564804 ; total 0 ; last 0) - unlinked 10000 (time 1787564960 ; total 156 ; last 156) - unlinked 20000 (time 1787565109 ; total 305 ; last 149) - unlinked 30000 (time 1787565254 ; total 450 ; last 145) - unlinked 40000 (time 1787565402 ; total 598 ; last 148) total: 50000 unlinks in 746 seconds: 67.024132 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg653-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg653-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg653-server stop mds service on oleg653-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg653-server unloading modules via unload_modules_local on: 'oleg653-server' oleg653-server: oleg653-server.virtnet: executing unload_modules_local oleg653-server: modules unloaded. pdsh@oleg653-client: oleg653-client: ssh exited with exit code 2 pdsh@oleg653-client: oleg653-server: ssh exited with exit code 2 pdsh@oleg653-client: oleg653-client: ssh exited with exit code 2 pdsh@oleg653-client: oleg653-server: ssh exited with exit code 2