== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:25:38 (1787606738) start mds service on oleg654-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg654-server' oleg654-server: oleg654-server.virtnet: executing load_modules_local oleg654-server: Loading modules from /home/green/git/lustre-release/lustre oleg654-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg654-server: detected 4 online CPUs by sysfs oleg654-server: MODOPTS_LIBCFS= oleg654-server: Force libcfs to create 2 CPU partitions oleg654-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg654-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg654-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg654-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg654-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg654-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg654-server: oleg654-server.virtnet: executing set_default_debug -1 all pdsh@oleg654-client: oleg654-server: ssh exited with exit code 1 Started lustre-OST0000 oleg654-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg654-client.virtnet: -o user_xattr,flock 192.168.206.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 308 (time 1787606830.79 total 10.51 last 29.31) - open/close 610 (time 1787606841.82 total 21.54 last 27.37) - open/close 885 (time 1787606851.84 total 31.56 last 27.46) - open/close 1145 (time 1787606861.96 total 41.68 last 25.68) - open/close 1394 (time 1787606871.96 total 51.68 last 24.89) - open/close 1605 (time 1787606881.97 total 61.69 last 21.08) - open/close 1905 (time 1787606893.00 total 72.72 last 27.20) - open/close 2166 (time 1787606903.68 total 83.40 last 24.43) - open/close 2383 (time 1787606913.69 total 93.41 last 21.69) - open/close 2593 (time 1787606923.70 total 103.42 last 20.97) - open/close 2842 (time 1787606934.22 total 113.95 last 23.66) - open/close 3116 (time 1787606944.59 total 124.31 last 26.43) - open/close 3402 (time 1787606955.27 total 134.99 last 26.77) - open/close 3642 (time 1787606965.76 total 145.48 last 22.89) - open/close 3996 (time 1787606975.76 total 155.48 last 35.38) - open/close 4252 (time 1787606986.25 total 165.97 last 24.41) - open/close 4511 (time 1787606996.26 total 175.98 last 25.88) - open/close 4773 (time 1787607006.29 total 186.01 last 26.12) - open/close 5019 (time 1787607016.29 total 196.01 last 24.59) - open/close 5293 (time 1787607026.30 total 206.02 last 27.39) - open/close 5485 (time 1787607036.69 total 216.41 last 18.47) - open/close 5792 (time 1787607047.25 total 226.97 last 29.08) - open/close 6043 (time 1787607057.25 total 236.97 last 25.09) - open/close 6288 (time 1787607067.27 total 246.99 last 24.46) - open/close 6543 (time 1787607077.27 total 257.00 last 25.48) - open/close 6821 (time 1787607087.50 total 267.23 last 27.17) - open/close 7084 (time 1787607097.67 total 277.39 last 25.87) - open/close 7359 (time 1787607107.67 total 287.40 last 27.49) - open/close 7584 (time 1787607118.20 total 297.93 last 21.37) - open/close 7821 (time 1787607129.02 total 308.74 last 21.91) - open/close 8074 (time 1787607139.03 total 318.75 last 25.28) - open/close 8324 (time 1787607149.44 total 329.16 last 24.01) - open/close 8423 (time 1787607159.45 total 339.17 last 9.89) - open/close 8569 (time 1787607169.46 total 349.19 last 14.57) - open/close 8804 (time 1787607179.47 total 359.19 last 23.49) - open/close 9030 (time 1787607190.28 total 370.00 last 20.90) - open/close 9275 (time 1787607200.28 total 380.00 last 24.50) - open/close 9521 (time 1787607211.20 total 390.92 last 22.54) - open/close 9757 (time 1787607221.20 total 400.92 last 23.59) - open/close 9960 (time 1787607231.27 total 410.99 last 20.16) - open/close 10174 (time 1787607241.28 total 421.00 last 21.38) - open/close 10458 (time 1787607252.03 total 431.75 last 26.42) - open/close 10721 (time 1787607262.03 total 441.75 last 26.29) - open/close 10961 (time 1787607272.04 total 451.76 last 23.98) - open/close 11193 (time 1787607282.77 total 462.49 last 21.63) - open/close 11418 (time 1787607292.80 total 472.52 last 22.43) - open/close 11546 (time 1787607303.38 total 483.10 last 12.10) - open/close 11822 (time 1787607314.05 total 493.77 last 25.86) - open/close 12127 (time 1787607325.00 total 504.72 last 27.87) - open/close 12412 (time 1787607335.01 total 514.73 last 28.46) - open/close 12682 (time 1787607345.19 total 524.92 last 26.52) - open/close 12964 (time 1787607355.78 total 535.50 last 26.63) - open/close 13188 (time 1787607366.03 total 545.75 last 21.86) - open/close 13346 (time 1787607376.34 total 556.06 last 15.32) - open/close 13568 (time 1787607386.35 total 566.07 last 22.18) - open/close 13830 (time 1787607397.32 total 577.04 last 23.88) - open/close 14099 (time 1787607407.44 total 587.16 last 26.58) - open/close 14355 (time 1787607418.05 total 597.77 last 24.14) - open/close 14653 (time 1787607428.49 total 608.21 last 28.54) - open/close 14888 (time 1787607438.50 total 618.23 last 23.46) - open/close 15127 (time 1787607449.29 total 629.01 last 22.17) - open/close 15380 (time 1787607459.29 total 639.01 last 25.28) - open/close 15635 (time 1787607469.30 total 649.02 last 25.47) - open/close 15879 (time 1787607479.58 total 659.30 last 23.75) - open/close 16138 (time 1787607490.33 total 670.05 last 24.09) - open/close 16394 (time 1787607500.81 total 680.53 last 24.43) - open/close 16657 (time 1787607510.82 total 690.55 last 26.26) - open/close 16910 (time 1787607521.62 total 701.34 last 23.44) - open/close 17139 (time 1787607531.62 total 711.35 last 22.89) - open/close 17353 (time 1787607542.02 total 721.74 last 20.58) - open/close 17566 (time 1787607553.09 total 732.81 last 19.24) - open/close 17802 (time 1787607563.81 total 743.53 last 22.02) - open/close 18120 (time 1787607574.66 total 754.39 last 29.30) - open/close 18375 (time 1787607585.41 total 765.13 last 23.73) - open/close 18651 (time 1787607595.44 total 775.16 last 27.53) - open/close 18902 (time 1787607605.45 total 785.17 last 25.07) - open/close 19147 (time 1787607615.46 total 795.19 last 24.46) - open/close 19428 (time 1787607626.44 total 806.16 last 25.61) - open/close 19719 (time 1787607636.76 total 816.48 last 28.19) - open/close 20000 (time 1787607646.41 total 826.13 last 29.12) - open/close 20270 (time 1787607656.42 total 836.14 last 26.98) - open/close 20508 (time 1787607666.42 total 846.15 last 23.78) - open/close 21016 (time 1787607676.43 total 856.15 last 50.78) - open/close 21939 (time 1787607686.44 total 866.16 last 92.22) - open/close 22858 (time 1787607696.44 total 876.16 last 91.88) - open/close 23787 (time 1787607706.44 total 886.17 last 92.85) - open/close 24745 (time 1787607716.45 total 896.17 last 95.79) - open/close 25619 (time 1787607726.46 total 906.18 last 87.30) - open/close 26515 (time 1787607736.46 total 916.18 last 89.59) - open/close 27425 (time 1787607746.48 total 926.20 last 90.84) - open/close 28333 (time 1787607756.48 total 936.20 last 90.76) - open/close 29256 (time 1787607766.48 total 946.20 last 92.29) - open/close 30000 (time 1787607774.94 total 954.66 last 87.96) - open/close 30791 (time 1787607784.96 total 964.68 last 78.94) - open/close 31711 (time 1787607794.96 total 974.69 last 91.95) - open/close 32603 (time 1787607804.98 total 984.70 last 89.10) - open/close 33503 (time 1787607814.98 total 994.70 last 89.96) - open/close 34415 (time 1787607824.98 total 1004.71 last 91.17) - open/close 35357 (time 1787607834.99 total 1014.71 last 94.16) - open/close 36180 (time 1787607845.00 total 1024.73 last 82.17) - open/close 36902 (time 1787607855.00 total 1034.73 last 72.19) - open/close 37680 (time 1787607865.00 total 1044.73 last 77.80) - open/close 38471 (time 1787607875.01 total 1054.74 last 79.03) - open/close 39355 (time 1787607885.01 total 1064.74 last 88.40) - open/close 40000 (time 1787607892.28 total 1072.01 last 88.71) - open/close 40911 (time 1787607902.29 total 1082.01 last 91.04) - open/close 41828 (time 1787607912.30 total 1092.02 last 91.66) - open/close 42701 (time 1787607922.30 total 1102.02 last 87.28) - open/close 43598 (time 1787607932.31 total 1112.03 last 89.63) - open/close 44475 (time 1787607942.31 total 1122.03 last 87.69) - open/close 45300 (time 1787607952.31 total 1132.03 last 82.50) - open/close 46179 (time 1787607962.31 total 1142.03 last 87.89) - open/close 47089 (time 1787607972.32 total 1152.04 last 90.91) - open/close 47929 (time 1787607982.32 total 1162.04 last 83.96) - open/close 48781 (time 1787607992.32 total 1172.05 last 85.19) - open/close 49690 (time 1787608002.34 total 1182.06 last 90.80) total: 50000 open/close in 1185.52 seconds: 42.18 ops/second - unlinked 0 (time 1787608009 ; total 0 ; last 0) - unlinked 10000 (time 1787608127 ; total 118 ; last 118) - unlinked 20000 (time 1787608247 ; total 238 ; last 120) - unlinked 30000 (time 1787608319 ; total 310 ; last 72) - unlinked 40000 (time 1787608389 ; total 380 ; last 70) total: 50000 unlinks in 451 seconds: 110.864746 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg654-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg654-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg654-server stop mds service on oleg654-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg654-server stop mds service on oleg654-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg654-server unloading modules via unload_modules_local on: 'oleg654-server' oleg654-server: oleg654-server.virtnet: executing unload_modules_local oleg654-server: modules unloaded. pdsh@oleg654-client: oleg654-client: ssh exited with exit code 2 pdsh@oleg654-client: oleg654-server: ssh exited with exit code 2 pdsh@oleg654-client: oleg654-client: ssh exited with exit code 2 pdsh@oleg654-client: oleg654-server: ssh exited with exit code 2