== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:41:06 (1787848866) start mds service on oleg336-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing load_modules_local oleg336-server: Loading modules from /home/green/git/lustre-release/lustre oleg336-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg336-server: detected 4 online CPUs by sysfs oleg336-server: MODOPTS_LIBCFS= oleg336-server: Force libcfs to create 2 CPU partitions oleg336-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg336-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg336-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg336-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 241 (time 1787848939.58 total 10.49 last 22.97) - open/close 456 (time 1787848949.77 total 20.68 last 21.10) - open/close 602 (time 1787848959.79 total 30.70 last 14.58) - open/close 801 (time 1787848969.86 total 40.77 last 19.76) - open/close 1013 (time 1787848979.88 total 50.79 last 21.17) - open/close 1222 (time 1787848989.90 total 60.81 last 20.86) - open/close 1482 (time 1787849000.54 total 71.45 last 24.43) - open/close 1734 (time 1787849011.01 total 81.92 last 24.07) - open/close 1973 (time 1787849021.71 total 92.62 last 22.34) - open/close 2181 (time 1787849031.72 total 102.63 last 20.77) - open/close 2430 (time 1787849042.19 total 113.10 last 23.78) - open/close 2720 (time 1787849053.20 total 124.11 last 26.36) - open/close 2975 (time 1787849063.88 total 134.79 last 23.88) - open/close 3250 (time 1787849074.84 total 145.75 last 25.08) - open/close 3541 (time 1787849084.84 total 155.75 last 29.09) - open/close 3688 (time 1787849094.88 total 165.79 last 14.64) - open/close 3980 (time 1787849104.89 total 175.80 last 29.17) - open/close 4189 (time 1787849115.27 total 186.18 last 20.14) - open/close 4434 (time 1787849125.27 total 196.18 last 24.50) - open/close 4672 (time 1787849135.39 total 206.30 last 23.50) - open/close 4946 (time 1787849145.99 total 216.90 last 25.86) - open/close 5219 (time 1787849156.00 total 226.91 last 27.27) - open/close 5421 (time 1787849166.28 total 237.19 last 19.65) - open/close 5621 (time 1787849176.71 total 247.62 last 19.17) - open/close 5832 (time 1787849187.02 total 257.93 last 20.47) - open/close 6036 (time 1787849197.27 total 268.18 last 19.90) - open/close 6172 (time 1787849207.30 total 278.21 last 13.55) - open/close 6375 (time 1787849217.81 total 288.72 last 19.31) - open/close 6600 (time 1787849227.97 total 298.88 last 22.15) - open/close 6796 (time 1787849238.00 total 308.91 last 19.55) - open/close 7060 (time 1787849248.71 total 319.62 last 24.64) - open/close 7310 (time 1787849259.27 total 330.18 last 23.66) - open/close 7610 (time 1787849269.29 total 340.20 last 29.95) - open/close 7826 (time 1787849280.27 total 351.18 last 19.67) - open/close 8087 (time 1787849290.28 total 361.19 last 26.09) - open/close 8201 (time 1787849300.86 total 371.77 last 10.77) - open/close 8356 (time 1787849311.87 total 382.78 last 14.08) - open/close 8617 (time 1787849322.44 total 393.35 last 24.70) - open/close 8847 (time 1787849334.73 total 405.64 last 18.72) - open/close 9083 (time 1787849344.84 total 415.75 last 23.33) - open/close 9338 (time 1787849355.15 total 426.06 last 24.75) - open/close 9583 (time 1787849365.15 total 436.06 last 24.48) - open/close 9847 (time 1787849375.70 total 446.61 last 25.04) - open/close 10000 (time 1787849381.44 total 452.35 last 26.61) - open/close 10294 (time 1787849391.45 total 462.36 last 29.38) - open/close 10523 (time 1787849401.47 total 472.38 last 22.85) - open/close 10786 (time 1787849411.49 total 482.40 last 26.26) - open/close 11003 (time 1787849421.92 total 492.83 last 20.81) - open/close 11247 (time 1787849434.89 total 505.80 last 18.81) - open/close 11501 (time 1787849446.35 total 517.26 last 22.16) - open/close 11641 (time 1787849457.44 total 528.35 last 12.62) - open/close 11841 (time 1787849468.30 total 539.21 last 18.42) - open/close 12020 (time 1787849478.80 total 549.71 last 17.05) - open/close 12201 (time 1787849489.15 total 560.06 last 17.48) - open/close 12428 (time 1787849500.12 total 571.03 last 20.69) - open/close 12578 (time 1787849510.13 total 581.04 last 14.99) - open/close 12688 (time 1787849520.65 total 591.56 last 10.46) - open/close 12759 (time 1787849531.55 total 602.46 last 6.51) - open/close 12841 (time 1787849541.57 total 612.48 last 8.19) - open/close 13033 (time 1787849551.95 total 622.86 last 18.48) - open/close 13227 (time 1787849561.97 total 632.88 last 19.37) - open/close 13365 (time 1787849571.98 total 642.89 last 13.79) - open/close 13575 (time 1787849582.04 total 652.95 last 20.86) - open/close 13771 (time 1787849592.65 total 663.56 last 18.48) - open/close 14081 (time 1787849603.09 total 674.00 last 29.70) - open/close 14412 (time 1787849614.03 total 684.94 last 30.24) - open/close 14651 (time 1787849624.05 total 694.96 last 23.86) - open/close 14884 (time 1787849634.96 total 705.87 last 21.37) - open/close 15153 (time 1787849645.48 total 716.39 last 25.56) - open/close 15400 (time 1787849655.48 total 726.39 last 24.68) - open/close 15681 (time 1787849665.87 total 736.78 last 27.06) - open/close 15864 (time 1787849675.88 total 746.79 last 18.28) - open/close 16086 (time 1787849686.92 total 757.83 last 20.11) - open/close 16312 (time 1787849697.17 total 768.08 last 22.05) - open/close 16567 (time 1787849707.17 total 778.08 last 25.48) - open/close 16792 (time 1787849718.09 total 789.00 last 20.60) - open/close 16978 (time 1787849728.14 total 799.05 last 18.51) - open/close 17243 (time 1787849738.70 total 809.61 last 25.10) - open/close 17488 (time 1787849749.64 total 820.55 last 22.40) - open/close 17733 (time 1787849760.60 total 831.51 last 22.35) - open/close 17971 (time 1787849770.61 total 841.52 last 23.79) - open/close 18207 (time 1787849781.45 total 852.36 last 21.76) - open/close 18447 (time 1787849792.13 total 863.04 last 22.47) - open/close 18686 (time 1787849802.77 total 873.68 last 22.48) - open/close 19234 (time 1787849812.79 total 883.70 last 54.69) - open/close 20000 (time 1787849821.82 total 892.73 last 84.82) - open/close 20831 (time 1787849831.82 total 902.73 last 83.07) - open/close 21644 (time 1787849841.84 total 912.75 last 81.18) - open/close 22395 (time 1787849851.84 total 922.75 last 75.08) - open/close 23210 (time 1787849861.84 total 932.75 last 81.46) - open/close 23920 (time 1787849871.87 total 942.78 last 70.85) - open/close 24757 (time 1787849881.87 total 952.78 last 83.68) - open/close 25471 (time 1787849891.87 total 962.78 last 71.36) - open/close 26214 (time 1787849901.89 total 972.80 last 74.15) - open/close 26961 (time 1787849911.90 total 982.81 last 74.63) - open/close 27705 (time 1787849921.91 total 992.82 last 74.33) - open/close 28405 (time 1787849931.94 total 1002.85 last 69.81) - open/close 29200 (time 1787849941.95 total 1012.86 last 79.45) - open/close 29977 (time 1787849951.98 total 1022.89 last 77.46) - open/close 30780 (time 1787849961.98 total 1032.89 last 80.26) - open/close 31539 (time 1787849971.99 total 1042.90 last 75.86) - open/close 32354 (time 1787849982.00 total 1052.91 last 81.43) - open/close 33105 (time 1787849992.01 total 1062.92 last 74.99) - open/close 33864 (time 1787850002.02 total 1072.93 last 75.86) - open/close 34598 (time 1787850012.02 total 1082.93 last 73.36) - open/close 35398 (time 1787850022.03 total 1092.94 last 79.94) - open/close 36192 (time 1787850032.05 total 1102.96 last 79.24) - open/close 36949 (time 1787850042.06 total 1112.97 last 75.64) - open/close 37612 (time 1787850052.06 total 1122.97 last 66.27) - open/close 38407 (time 1787850062.06 total 1132.97 last 79.48) - open/close 39119 (time 1787850072.07 total 1142.98 last 71.12) - open/close 39832 (time 1787850082.08 total 1152.99 last 71.26) - open/close 40000 (time 1787850084.39 total 1155.30 last 72.61) - open/close 40771 (time 1787850094.39 total 1165.30 last 77.10) - open/close 41506 (time 1787850104.41 total 1175.32 last 73.36) - open/close 42297 (time 1787850114.41 total 1185.32 last 79.09) - open/close 43080 (time 1787850124.42 total 1195.33 last 78.28) - open/close 43776 (time 1787850134.44 total 1205.35 last 69.42) - open/close 44526 (time 1787850144.46 total 1215.37 last 74.91) - open/close 45116 (time 1787850154.46 total 1225.37 last 58.97) - open/close 45803 (time 1787850164.47 total 1235.38 last 68.65) - open/close 46525 (time 1787850174.49 total 1245.40 last 72.05) - open/close 47288 (time 1787850184.50 total 1255.41 last 76.21) - open/close 48066 (time 1787850194.50 total 1265.41 last 77.77) - open/close 48764 (time 1787850204.51 total 1275.42 last 69.79) - open/close 49523 (time 1787850214.51 total 1285.42 last 75.90) total: 50000 open/close in 1291.30 seconds: 38.72 ops/second - unlinked 0 (time 1787850224 ; total 0 ; last 0) - unlinked 10000 (time 1787850367 ; total 143 ; last 143) - unlinked 20000 (time 1787850504 ; total 280 ; last 137) - unlinked 30000 (time 1787850643 ; total 419 ; last 139) - unlinked 40000 (time 1787850785 ; total 561 ; last 142) total: 50000 unlinks in 706 seconds: 70.821533 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg336-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server unloading modules via unload_modules_local on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing unload_modules_local oleg336-server: modules unloaded. pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2