== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:49:15 (1789480155) start mds service on oleg136-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing load_modules_local oleg136-server: Loading modules from /home/green/git/lustre-release/lustre oleg136-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg136-server: detected 4 online CPUs by sysfs oleg136-server: MODOPTS_LIBCFS= oleg136-server: Force libcfs to create 2 CPU partitions oleg136-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg136-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg136-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg136-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg136-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug -1 all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 147 (time 1789480260.56 total 10.17 last 14.45) - open/close 365 (time 1789480270.57 total 20.18 last 21.79) - open/close 546 (time 1789480280.63 total 30.24 last 17.99) - open/close 673 (time 1789480290.70 total 40.30 last 12.62) - open/close 1008 (time 1789480301.64 total 51.24 last 30.62) - open/close 1173 (time 1789480311.64 total 61.25 last 16.49) - open/close 1277 (time 1789480321.66 total 71.26 last 10.38) - open/close 1436 (time 1789480331.68 total 81.28 last 15.87) - open/close 1661 (time 1789480341.68 total 91.29 last 22.49) - open/close 1879 (time 1789480351.68 total 101.29 last 21.79) - open/close 2022 (time 1789480361.71 total 111.31 last 14.27) - open/close 2208 (time 1789480371.71 total 121.32 last 18.59) - open/close 2268 (time 1789480381.72 total 131.32 last 6.00) - open/close 2475 (time 1789480392.52 total 142.13 last 19.16) - open/close 2597 (time 1789480402.52 total 152.13 last 12.19) - open/close 2704 (time 1789480412.53 total 162.13 last 10.70) - open/close 2827 (time 1789480423.44 total 173.04 last 11.27) - open/close 2991 (time 1789480434.30 total 183.91 last 15.09) - open/close 3143 (time 1789480445.17 total 194.78 last 13.99) - open/close 3322 (time 1789480455.42 total 205.02 last 17.47) - open/close 3430 (time 1789480466.44 total 216.05 last 9.80) - open/close 3654 (time 1789480476.45 total 226.06 last 22.37) - open/close 3875 (time 1789480487.17 total 236.78 last 20.62) - open/close 4035 (time 1789480497.18 total 246.79 last 15.99) - open/close 4271 (time 1789480508.16 total 257.76 last 21.50) - open/close 4387 (time 1789480518.16 total 267.76 last 11.60) - open/close 4568 (time 1789480528.65 total 278.26 last 17.25) - open/close 4727 (time 1789480539.57 total 289.18 last 14.56) - open/close 4861 (time 1789480550.30 total 299.91 last 12.49) - open/close 5039 (time 1789480560.50 total 310.11 last 17.45) - open/close 5272 (time 1789480570.50 total 320.11 last 23.30) - open/close 5493 (time 1789480581.13 total 330.74 last 20.79) - open/close 5724 (time 1789480594.23 total 343.83 last 17.63) - open/close 5903 (time 1789480604.23 total 353.84 last 17.89) - open/close 6116 (time 1789480614.66 total 364.27 last 20.43) - open/close 6336 (time 1789480624.68 total 374.28 last 21.96) - open/close 6566 (time 1789480634.71 total 384.32 last 22.92) - open/close 6808 (time 1789480645.68 total 395.29 last 22.06) - open/close 7031 (time 1789480656.72 total 406.33 last 20.19) - open/close 7212 (time 1789480666.73 total 416.34 last 18.09) - open/close 7390 (time 1789480677.44 total 427.04 last 16.63) - open/close 7657 (time 1789480687.44 total 437.05 last 26.69) - open/close 7880 (time 1789480697.45 total 447.06 last 22.27) - open/close 8098 (time 1789480707.46 total 457.07 last 21.78) - open/close 8322 (time 1789480717.47 total 467.08 last 22.38) - open/close 8535 (time 1789480727.49 total 477.09 last 21.26) - open/close 8744 (time 1789480737.60 total 487.21 last 20.67) - open/close 8934 (time 1789480748.47 total 498.08 last 17.47) - open/close 9048 (time 1789480758.72 total 508.32 last 11.13) - open/close 9224 (time 1789480769.21 total 518.81 last 16.78) - open/close 9486 (time 1789480779.21 total 528.82 last 26.18) - open/close 9739 (time 1789480789.70 total 539.31 last 24.12) - open/close 9975 (time 1789480799.70 total 549.31 last 23.59) - open/close 10223 (time 1789480810.50 total 560.10 last 22.98) - open/close 10463 (time 1789480821.30 total 570.90 last 22.22) - open/close 10697 (time 1789480831.30 total 580.90 last 23.40) - open/close 10956 (time 1789480842.04 total 591.65 last 24.10) - open/close 11282 (time 1789480852.05 total 601.66 last 32.57) - open/close 11508 (time 1789480862.05 total 611.66 last 22.60) - open/close 11686 (time 1789480872.33 total 621.94 last 17.32) - open/close 11924 (time 1789480883.01 total 632.61 last 22.30) - open/close 12150 (time 1789480893.69 total 643.30 last 21.16) - open/close 12416 (time 1789480904.66 total 654.26 last 24.25) - open/close 12700 (time 1789480915.38 total 664.99 last 26.48) - open/close 12921 (time 1789480925.39 total 675.00 last 22.07) - open/close 13114 (time 1789480935.41 total 685.02 last 19.26) - open/close 13272 (time 1789480945.46 total 695.07 last 15.72) - open/close 13532 (time 1789480955.47 total 705.08 last 25.98) - open/close 13674 (time 1789480966.42 total 716.03 last 12.97) - open/close 13924 (time 1789480976.42 total 726.03 last 24.99) - open/close 14135 (time 1789480986.43 total 736.04 last 21.09) - open/close 14310 (time 1789480996.44 total 746.05 last 17.47) - open/close 14539 (time 1789481007.06 total 756.67 last 21.57) - open/close 14773 (time 1789481018.12 total 767.73 last 21.15) - open/close 14916 (time 1789481028.13 total 777.74 last 14.28) - open/close 15045 (time 1789481038.25 total 787.85 last 12.76) - open/close 15234 (time 1789481049.17 total 798.78 last 17.30) - open/close 15459 (time 1789481059.39 total 809.00 last 22.02) - open/close 15678 (time 1789481069.81 total 819.42 last 21.02) - open/close 15888 (time 1789481079.82 total 829.43 last 20.98) - open/close 16137 (time 1789481090.49 total 840.10 last 23.34) - open/close 16369 (time 1789481100.73 total 850.34 last 22.66) - open/close 16552 (time 1789481111.29 total 860.90 last 17.32) - open/close 16770 (time 1789481122.27 total 871.88 last 19.86) - open/close 16982 (time 1789481132.27 total 881.88 last 21.19) - open/close 17259 (time 1789481142.29 total 891.89 last 27.66) - open/close 17480 (time 1789481152.29 total 901.90 last 22.08) - open/close 17731 (time 1789481162.29 total 911.90 last 25.10) - open/close 17914 (time 1789481173.11 total 922.72 last 16.92) - open/close 18182 (time 1789481183.48 total 933.09 last 25.83) - open/close 18433 (time 1789481193.49 total 943.10 last 25.08) - open/close 18793 (time 1789481203.50 total 953.11 last 35.97) - open/close 19495 (time 1789481213.50 total 963.11 last 70.15) - open/close 20000 (time 1789481220.11 total 969.72 last 76.41) - open/close 20750 (time 1789481230.12 total 979.73 last 74.93) - open/close 21543 (time 1789481240.13 total 989.74 last 79.26) - open/close 22294 (time 1789481250.13 total 999.74 last 75.08) - open/close 23112 (time 1789481260.14 total 1009.75 last 81.74) - open/close 23932 (time 1789481270.15 total 1019.76 last 81.92) - open/close 24671 (time 1789481280.17 total 1029.78 last 73.71) - open/close 25412 (time 1789481290.18 total 1039.79 last 74.07) - open/close 26292 (time 1789481300.18 total 1049.79 last 87.94) - open/close 27042 (time 1789481310.19 total 1059.80 last 74.93) - open/close 27908 (time 1789481320.20 total 1069.80 last 86.59) - open/close 28718 (time 1789481330.20 total 1079.81 last 80.97) - open/close 29500 (time 1789481340.20 total 1089.81 last 78.19) - open/close 30000 (time 1789481346.14 total 1095.75 last 84.20) - open/close 30845 (time 1789481356.14 total 1105.75 last 84.47) - open/close 31582 (time 1789481366.15 total 1115.76 last 73.63) - open/close 32422 (time 1789481376.15 total 1125.76 last 83.97) - open/close 33232 (time 1789481386.16 total 1135.77 last 80.98) - open/close 34097 (time 1789481396.16 total 1145.77 last 86.48) - open/close 34880 (time 1789481406.17 total 1155.78 last 78.24) - open/close 35614 (time 1789481416.17 total 1165.78 last 73.37) - open/close 36333 (time 1789481426.18 total 1175.79 last 71.82) - open/close 37028 (time 1789481436.20 total 1185.80 last 69.40) - open/close 37908 (time 1789481446.21 total 1195.81 last 87.92) - open/close 38840 (time 1789481456.22 total 1205.82 last 93.09) - open/close 39673 (time 1789481466.22 total 1215.83 last 83.24) - open/close 40000 (time 1789481470.15 total 1219.76 last 83.28) - open/close 40769 (time 1789481480.15 total 1229.76 last 76.88) - open/close 41658 (time 1789481490.17 total 1239.77 last 88.78) - open/close 42477 (time 1789481500.17 total 1249.78 last 81.86) - open/close 43316 (time 1789481510.18 total 1259.79 last 83.80) - open/close 44105 (time 1789481520.18 total 1269.79 last 78.89) - open/close 44931 (time 1789481530.20 total 1279.81 last 82.48) - open/close 45855 (time 1789481540.20 total 1289.81 last 92.39) - open/close 46693 (time 1789481550.20 total 1299.81 last 83.77) - open/close 47435 (time 1789481560.21 total 1309.82 last 74.14) - open/close 48264 (time 1789481570.22 total 1319.83 last 82.84) - open/close 49109 (time 1789481580.23 total 1329.83 last 84.44) - open/close 49971 (time 1789481590.23 total 1339.84 last 86.14) total: 50000 open/close in 1340.14 seconds: 37.31 ops/second - unlinked 0 (time 1789481593 ; total 0 ; last 0) - unlinked 10000 (time 1789481724 ; total 131 ; last 131) - unlinked 20000 (time 1789481901 ; total 308 ; last 177) - unlinked 30000 (time 1789482054 ; total 461 ; last 153) - unlinked 40000 (time 1789482212 ; total 619 ; last 158) total: 50000 unlinks in 774 seconds: 64.599480 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg136-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server stop mds service on oleg136-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server unloading modules via unload_modules_local on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing unload_modules_local oleg136-server: modules unloaded. pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-client: ssh exited with exit code 2 pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2