== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:29:34 (1788308974) start mds service on oleg225-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing load_modules_local oleg225-server: Loading modules from /home/green/git/lustre-release/lustre oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg225-server: detected 4 online CPUs by sysfs oleg225-server: MODOPTS_LIBCFS= oleg225-server: Force libcfs to create 2 CPU partitions oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg225-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 218 (time 1788309068.02 total 10.03 last 21.73) - open/close 459 (time 1788309078.03 total 20.04 last 24.08) - open/close 668 (time 1788309088.13 total 30.14 last 20.70) - open/close 890 (time 1788309098.56 total 40.57 last 21.28) - open/close 1063 (time 1788309108.61 total 50.62 last 17.21) - open/close 1242 (time 1788309118.61 total 60.62 last 17.89) - open/close 1470 (time 1788309129.21 total 71.22 last 21.51) - open/close 1683 (time 1788309139.23 total 81.24 last 21.25) - open/close 1880 (time 1788309149.27 total 91.28 last 19.63) - open/close 2038 (time 1788309160.07 total 102.08 last 14.63) - open/close 2158 (time 1788309171.01 total 113.02 last 10.97) - open/close 2371 (time 1788309181.75 total 123.76 last 19.82) - open/close 2627 (time 1788309191.76 total 133.77 last 25.57) - open/close 2848 (time 1788309202.18 total 144.19 last 21.22) - open/close 3051 (time 1788309212.64 total 154.65 last 19.40) - open/close 3274 (time 1788309223.37 total 165.38 last 20.80) - open/close 3434 (time 1788309233.53 total 175.54 last 15.74) - open/close 3638 (time 1788309243.53 total 185.54 last 20.39) - open/close 3873 (time 1788309254.52 total 196.53 last 21.39) - open/close 4027 (time 1788309264.56 total 206.57 last 15.33) - open/close 4158 (time 1788309274.97 total 216.98 last 12.58) - open/close 4393 (time 1788309285.07 total 227.08 last 23.28) - open/close 4633 (time 1788309296.00 total 238.01 last 21.97) - open/close 4848 (time 1788309306.45 total 248.46 last 20.58) - open/close 5073 (time 1788309317.18 total 259.19 last 20.96) - open/close 5233 (time 1788309327.80 total 269.81 last 15.07) - open/close 5352 (time 1788309337.81 total 279.82 last 11.88) - open/close 5508 (time 1788309348.29 total 290.30 last 14.89) - open/close 5580 (time 1788309358.29 total 300.30 last 7.20) - open/close 5779 (time 1788309368.97 total 310.98 last 18.64) - open/close 5911 (time 1788309379.39 total 321.41 last 12.66) - open/close 6024 (time 1788309389.43 total 331.44 last 11.26) - open/close 6155 (time 1788309399.45 total 341.46 last 13.08) - open/close 6376 (time 1788309409.48 total 351.49 last 22.03) - open/close 6590 (time 1788309419.65 total 361.67 last 21.04) - open/close 6788 (time 1788309429.67 total 371.68 last 19.78) - open/close 6924 (time 1788309442.68 total 384.69 last 10.45) - open/close 7142 (time 1788309452.69 total 394.70 last 21.79) - open/close 7310 (time 1788309462.70 total 404.71 last 16.78) - open/close 7536 (time 1788309473.41 total 415.43 last 21.09) - open/close 7742 (time 1788309483.43 total 425.44 last 20.57) - open/close 7990 (time 1788309494.45 total 436.46 last 22.50) - open/close 8118 (time 1788309506.05 total 448.06 last 11.03) - open/close 8333 (time 1788309516.54 total 458.55 last 20.50) - open/close 8572 (time 1788309527.50 total 469.51 last 21.80) - open/close 8848 (time 1788309537.87 total 479.88 last 26.61) - open/close 9063 (time 1788309548.53 total 490.54 last 20.17) - open/close 9300 (time 1788309559.16 total 501.17 last 22.30) - open/close 9549 (time 1788309570.03 total 512.04 last 22.91) - open/close 9640 (time 1788309580.23 total 522.24 last 8.92) - open/close 9889 (time 1788309590.26 total 532.27 last 24.83) - open/close 10000 (time 1788309596.88 total 538.89 last 16.77) - open/close 10044 (time 1788309611.20 total 553.21 last 3.07) - open/close 10178 (time 1788309621.21 total 563.22 last 13.38) - open/close 10387 (time 1788309632.13 total 574.14 last 19.13) - open/close 10672 (time 1788309642.64 total 584.65 last 27.13) - open/close 10925 (time 1788309652.65 total 594.66 last 25.26) - open/close 11153 (time 1788309663.45 total 605.46 last 21.13) - open/close 11375 (time 1788309674.05 total 616.06 last 20.94) - open/close 11497 (time 1788309684.66 total 626.67 last 11.50) - open/close 11740 (time 1788309695.55 total 637.56 last 22.31) - open/close 11972 (time 1788309706.35 total 648.36 last 21.49) - open/close 12154 (time 1788309716.40 total 658.41 last 18.11) - open/close 12280 (time 1788309727.19 total 669.20 last 11.68) - open/close 12503 (time 1788309737.20 total 679.21 last 22.28) - open/close 12650 (time 1788309747.58 total 689.59 last 14.16) - open/close 12747 (time 1788309758.40 total 700.41 last 8.96) - open/close 13028 (time 1788309768.91 total 710.92 last 26.75) - open/close 13240 (time 1788309778.92 total 720.93 last 21.17) - open/close 13372 (time 1788309789.20 total 731.21 last 12.84) - open/close 13647 (time 1788309799.89 total 741.90 last 25.74) - open/close 13885 (time 1788309809.98 total 751.99 last 23.58) - open/close 14079 (time 1788309819.99 total 762.00 last 19.38) - open/close 14315 (time 1788309830.53 total 772.54 last 22.39) - open/close 14546 (time 1788309840.82 total 782.84 last 22.44) - open/close 14792 (time 1788309850.83 total 792.84 last 24.59) - open/close 14911 (time 1788309860.84 total 802.85 last 11.89) - open/close 15146 (time 1788309871.87 total 813.88 last 21.30) - open/close 15428 (time 1788309882.42 total 824.43 last 26.73) - open/close 15688 (time 1788309892.45 total 834.46 last 25.93) - open/close 15779 (time 1788309902.53 total 844.54 last 9.02) - open/close 15855 (time 1788309913.53 total 855.54 last 6.91) - open/close 16029 (time 1788309923.57 total 865.58 last 17.34) - open/close 16195 (time 1788309935.13 total 877.14 last 14.35) - open/close 16362 (time 1788309945.14 total 887.15 last 16.68) - open/close 16415 (time 1788309955.76 total 897.77 last 4.99) - open/close 16471 (time 1788309966.79 total 908.80 last 5.08) - open/close 16648 (time 1788309976.79 total 918.80 last 17.69) - open/close 16887 (time 1788309986.82 total 928.83 last 23.84) - open/close 17084 (time 1788309996.84 total 938.85 last 19.66) - open/close 17216 (time 1788310007.37 total 949.38 last 12.54) - open/close 17434 (time 1788310017.44 total 959.45 last 21.65) - open/close 17633 (time 1788310029.49 total 971.50 last 16.51) - open/close 17731 (time 1788310039.50 total 981.52 last 9.79) - open/close 17885 (time 1788310049.60 total 991.61 last 15.25) - open/close 18070 (time 1788310059.64 total 1001.65 last 18.43) - open/close 18267 (time 1788310070.72 total 1012.73 last 17.78) - open/close 19049 (time 1788310080.72 total 1022.73 last 78.18) - open/close 19801 (time 1788310090.72 total 1032.73 last 75.18) - open/close 20000 (time 1788310093.22 total 1035.23 last 79.67) - open/close 20826 (time 1788310103.23 total 1045.24 last 82.51) - open/close 21513 (time 1788310113.23 total 1055.24 last 68.68) - open/close 22161 (time 1788310123.23 total 1065.24 last 64.80) - open/close 22887 (time 1788310133.26 total 1075.27 last 72.43) - open/close 23634 (time 1788310143.26 total 1085.27 last 74.67) - open/close 24267 (time 1788310153.26 total 1095.27 last 63.30) - open/close 24960 (time 1788310163.26 total 1105.27 last 69.28) - open/close 25665 (time 1788310173.27 total 1115.28 last 70.43) - open/close 26413 (time 1788310183.28 total 1125.29 last 74.78) - open/close 27163 (time 1788310193.28 total 1135.29 last 74.94) - open/close 27901 (time 1788310203.29 total 1145.30 last 73.76) - open/close 28534 (time 1788310213.29 total 1155.30 last 63.30) - open/close 29257 (time 1788310223.30 total 1165.31 last 72.25) - open/close 29915 (time 1788310233.30 total 1175.31 last 65.79) - open/close 30643 (time 1788310243.31 total 1185.32 last 72.73) - open/close 31348 (time 1788310253.31 total 1195.32 last 70.47) - open/close 32030 (time 1788310263.32 total 1205.33 last 68.14) - open/close 32827 (time 1788310273.33 total 1215.34 last 79.60) - open/close 33606 (time 1788310283.33 total 1225.34 last 77.89) - open/close 34416 (time 1788310293.34 total 1235.35 last 80.94) - open/close 35230 (time 1788310303.36 total 1245.37 last 81.21) - open/close 35901 (time 1788310313.38 total 1255.39 last 67.02) - open/close 36577 (time 1788310323.39 total 1265.40 last 67.51) - open/close 37286 (time 1788310333.40 total 1275.41 last 70.86) - open/close 37889 (time 1788310343.40 total 1285.41 last 60.29) - open/close 38563 (time 1788310353.41 total 1295.42 last 67.30) - open/close 39261 (time 1788310363.44 total 1305.45 last 69.62) - open/close 39756 (time 1788310373.44 total 1315.45 last 49.47) - open/close 40000 (time 1788310378.07 total 1320.08 last 52.79) - open/close 40444 (time 1788310388.08 total 1330.09 last 44.33) - open/close 40921 (time 1788310398.09 total 1340.10 last 47.66) - open/close 41554 (time 1788310408.09 total 1350.10 last 63.28) - open/close 42288 (time 1788310418.10 total 1360.11 last 73.36) - open/close 42909 (time 1788310428.10 total 1370.11 last 62.10) - open/close 43641 (time 1788310438.12 total 1380.13 last 73.04) - open/close 44315 (time 1788310448.13 total 1390.14 last 67.32) - open/close 45010 (time 1788310458.13 total 1400.14 last 69.50) - open/close 45635 (time 1788310468.14 total 1410.16 last 62.42) - open/close 46279 (time 1788310478.16 total 1420.17 last 64.31) - open/close 46974 (time 1788310488.17 total 1430.18 last 69.41) - open/close 47640 (time 1788310498.18 total 1440.19 last 66.52) - open/close 48274 (time 1788310508.19 total 1450.20 last 63.37) - open/close 48937 (time 1788310518.20 total 1460.21 last 66.24) - open/close 49508 (time 1788310528.20 total 1470.21 last 57.07) total: 50000 open/close in 1479.05 seconds: 33.81 ops/second - unlinked 0 (time 1788310541 ; total 0 ; last 0) - unlinked 10000 (time 1788310700 ; total 159 ; last 159) - unlinked 20000 (time 1788310869 ; total 328 ; last 169) - unlinked 30000 (time 1788311030 ; total 489 ; last 161) - unlinked 40000 (time 1788311198 ; total 657 ; last 168) total: 50000 unlinks in 824 seconds: 60.679611 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server unloading modules via unload_modules_local on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing unload_modules_local oleg225-server: modules unloaded. pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2