== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:51:19 (1787118679) start mds service on oleg358-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing load_modules_local oleg358-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg358-server: Loading modules from /home/green/git/lustre-release/lustre oleg358-server: detected 4 online CPUs by sysfs oleg358-server: MODOPTS_LIBCFS= oleg358-server: Force libcfs to create 2 CPU partitions oleg358-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg358-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg358-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg358-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg358-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg358-server: oleg358-server.virtnet: executing set_default_debug -1 all pdsh@oleg358-client: oleg358-server: ssh exited with exit code 1 Started lustre-OST0000 oleg358-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg358-client.virtnet: -o user_xattr,flock 192.168.203.158@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 79 (time 1787118768.95 total 10.00 last 7.90) - open/close 183 (time 1787118779.72 total 20.77 last 9.66) - open/close 397 (time 1787118790.05 total 31.10 last 20.71) - open/close 568 (time 1787118803.11 total 44.17 last 13.09) - open/close 804 (time 1787118813.13 total 54.18 last 23.57) - open/close 852 (time 1787118823.60 total 64.66 last 4.58) - open/close 949 (time 1787118834.42 total 75.47 last 8.97) - open/close 1119 (time 1787118845.23 total 86.28 last 15.73) - open/close 1179 (time 1787118855.99 total 97.04 last 5.57) - open/close 1332 (time 1787118866.92 total 107.97 last 14.00) - open/close 1565 (time 1787118876.92 total 117.98 last 23.30) - open/close 1804 (time 1787118886.92 total 127.98 last 23.90) - open/close 1965 (time 1787118897.41 total 138.46 last 15.36) - open/close 2143 (time 1787118907.51 total 148.56 last 17.63) - open/close 2362 (time 1787118920.42 total 161.48 last 16.95) - open/close 2474 (time 1787118931.33 total 172.38 last 10.27) - open/close 2615 (time 1787118941.33 total 182.38 last 14.09) - open/close 2754 (time 1787118951.48 total 192.53 last 13.70) - open/close 2902 (time 1787118961.84 total 202.89 last 14.29) - open/close 3144 (time 1787118971.84 total 212.89 last 24.20) - open/close 3355 (time 1787118984.00 total 225.05 last 17.35) - open/close 3565 (time 1787118996.41 total 237.46 last 16.92) - open/close 3738 (time 1787119009.44 total 250.49 last 13.28) - open/close 3811 (time 1787119019.46 total 260.51 last 7.29) - open/close 3904 (time 1787119030.06 total 271.11 last 8.78) - open/close 4074 (time 1787119040.08 total 281.13 last 16.97) - open/close 4168 (time 1787119050.75 total 291.80 last 8.81) - open/close 4238 (time 1787119060.92 total 301.97 last 6.88) - open/close 4379 (time 1787119071.92 total 312.97 last 12.82) - open/close 4467 (time 1787119082.88 total 323.94 last 8.02) - open/close 4511 (time 1787119093.35 total 334.40 last 4.21) - open/close 4595 (time 1787119103.65 total 344.70 last 8.15) - open/close 4752 (time 1787119113.66 total 354.71 last 15.69) - open/close 4922 (time 1787119123.66 total 364.71 last 17.00) - open/close 5070 (time 1787119133.67 total 374.72 last 14.79) - open/close 5263 (time 1787119143.78 total 384.83 last 19.08) - open/close 5441 (time 1787119154.62 total 395.67 last 16.42) - open/close 5616 (time 1787119164.64 total 405.69 last 17.47) - open/close 5818 (time 1787119175.08 total 416.13 last 19.35) - open/close 6060 (time 1787119186.10 total 427.15 last 21.96) - open/close 6230 (time 1787119196.12 total 437.17 last 16.97) - open/close 6452 (time 1787119206.12 total 447.18 last 22.18) - open/close 6508 (time 1787119216.68 total 457.73 last 5.31) - open/close 6646 (time 1787119227.15 total 468.20 last 13.18) - open/close 6751 (time 1787119239.73 total 480.78 last 8.35) - open/close 6933 (time 1787119249.89 total 490.95 last 17.91) - open/close 7081 (time 1787119260.46 total 501.51 last 14.00) - open/close 7248 (time 1787119270.47 total 511.53 last 16.68) - open/close 7386 (time 1787119283.11 total 524.16 last 10.92) - open/close 7608 (time 1787119293.81 total 534.86 last 20.75) - open/close 7780 (time 1787119305.03 total 546.08 last 15.33) - open/close 7978 (time 1787119315.43 total 556.48 last 19.05) - open/close 8177 (time 1787119326.07 total 567.12 last 18.71) - open/close 8411 (time 1787119336.60 total 577.65 last 22.21) - open/close 8552 (time 1787119347.72 total 588.77 last 12.69) - open/close 8643 (time 1787119358.37 total 599.43 last 8.54) - open/close 8824 (time 1787119368.70 total 609.75 last 17.53) - open/close 8928 (time 1787119380.42 total 621.47 last 8.87) - open/close 9159 (time 1787119390.42 total 631.47 last 23.09) - open/close 9373 (time 1787119401.34 total 642.40 last 19.59) - open/close 9565 (time 1787119411.70 total 652.75 last 18.55) - open/close 9812 (time 1787119421.75 total 662.80 last 24.57) - open/close 10000 (time 1787119429.99 total 671.04 last 22.81) - open/close 10142 (time 1787119442.83 total 683.88 last 11.06) - open/close 10372 (time 1787119452.84 total 693.89 last 22.98) - open/close 10541 (time 1787119462.92 total 703.97 last 16.77) - open/close 10761 (time 1787119475.75 total 716.80 last 17.15) - open/close 10937 (time 1787119486.56 total 727.61 last 16.28) - open/close 11156 (time 1787119497.31 total 738.36 last 20.37) - open/close 11376 (time 1787119508.26 total 749.31 last 20.09) - open/close 11502 (time 1787119518.29 total 759.35 last 12.56) - open/close 11658 (time 1787119528.31 total 769.36 last 15.58) - open/close 11821 (time 1787119540.08 total 781.13 last 13.85) - open/close 11976 (time 1787119551.85 total 792.90 last 13.17) - open/close 12154 (time 1787119562.60 total 803.66 last 16.56) - open/close 12228 (time 1787119572.84 total 813.89 last 7.23) - open/close 12352 (time 1787119583.08 total 824.14 last 12.10) - open/close 12460 (time 1787119593.21 total 834.26 last 10.67) - open/close 12578 (time 1787119603.22 total 844.27 last 11.79) - open/close 12804 (time 1787119613.22 total 854.27 last 22.60) - open/close 12906 (time 1787119623.78 total 864.83 last 9.66) - open/close 13089 (time 1787119636.79 total 877.84 last 14.06) - open/close 13238 (time 1787119649.55 total 890.60 last 11.68) - open/close 13407 (time 1787119659.56 total 900.61 last 16.89) - open/close 13531 (time 1787119669.56 total 910.61 last 12.39) - open/close 13739 (time 1787119679.57 total 920.62 last 20.78) - open/close 13932 (time 1787119690.11 total 931.16 last 18.31) - open/close 14092 (time 1787119700.47 total 941.53 last 15.43) - open/close 14293 (time 1787119711.22 total 952.27 last 18.71) - open/close 14580 (time 1787119721.23 total 962.28 last 28.67) - open/close 14840 (time 1787119731.25 total 972.30 last 25.94) - open/close 15051 (time 1787119741.35 total 982.40 last 20.89) - open/close 15293 (time 1787119752.12 total 993.17 last 22.47) - open/close 15514 (time 1787119762.87 total 1003.92 last 20.56) - open/close 15734 (time 1787119772.89 total 1013.94 last 21.97) - open/close 15963 (time 1787119785.74 total 1026.79 last 17.82) - open/close 16233 (time 1787119795.74 total 1036.79 last 26.99) - open/close 16891 (time 1787119805.78 total 1046.83 last 65.52) - open/close 17603 (time 1787119815.79 total 1056.84 last 71.14) - open/close 18285 (time 1787119825.80 total 1066.86 last 68.11) - open/close 19008 (time 1787119835.82 total 1076.87 last 72.17) - open/close 19691 (time 1787119845.83 total 1086.88 last 68.27) - open/close 20000 (time 1787119850.37 total 1091.43 last 67.95) - open/close 20637 (time 1787119860.38 total 1101.43 last 63.68) - open/close 21285 (time 1787119870.38 total 1111.43 last 64.78) - open/close 21893 (time 1787119880.39 total 1121.44 last 60.76) - open/close 22517 (time 1787119890.40 total 1131.45 last 62.34) - open/close 23165 (time 1787119900.41 total 1141.46 last 64.73) - open/close 23732 (time 1787119910.41 total 1151.46 last 56.68) - open/close 24364 (time 1787119920.43 total 1161.49 last 63.06) - open/close 24980 (time 1787119930.44 total 1171.49 last 61.57) - open/close 25607 (time 1787119940.45 total 1181.50 last 62.61) - open/close 26248 (time 1787119950.46 total 1191.51 last 64.04) - open/close 26911 (time 1787119960.47 total 1201.52 last 66.24) - open/close 27573 (time 1787119970.48 total 1211.53 last 66.14) - open/close 28172 (time 1787119980.48 total 1221.54 last 59.88) - open/close 28746 (time 1787119990.49 total 1231.54 last 57.35) - open/close 29363 (time 1787120000.50 total 1241.55 last 61.65) - open/close 30000 (time 1787120010.35 total 1251.40 last 64.67) - open/close 30655 (time 1787120020.37 total 1261.42 last 65.41) - open/close 31219 (time 1787120030.38 total 1271.43 last 56.34) - open/close 31802 (time 1787120040.38 total 1281.43 last 58.26) - open/close 32364 (time 1787120050.40 total 1291.45 last 56.12) - open/close 32941 (time 1787120060.40 total 1301.45 last 57.67) - open/close 33483 (time 1787120070.40 total 1311.45 last 54.20) - open/close 33999 (time 1787120080.41 total 1321.46 last 51.57) - open/close 34630 (time 1787120090.42 total 1331.47 last 63.04) - open/close 35318 (time 1787120100.42 total 1341.48 last 68.76) - open/close 35984 (time 1787120110.44 total 1351.49 last 66.48) - open/close 36588 (time 1787120120.45 total 1361.50 last 60.36) - open/close 37203 (time 1787120130.46 total 1371.51 last 61.42) - open/close 37763 (time 1787120140.46 total 1381.51 last 56.00) - open/close 38360 (time 1787120150.46 total 1391.51 last 59.70) - open/close 39022 (time 1787120160.48 total 1401.53 last 66.10) - open/close 39698 (time 1787120170.48 total 1411.54 last 67.56) - open/close 40000 (time 1787120175.78 total 1416.84 last 56.98) - open/close 40626 (time 1787120185.79 total 1426.84 last 62.57) - open/close 41243 (time 1787120195.79 total 1436.84 last 61.69) - open/close 41849 (time 1787120205.79 total 1446.85 last 60.57) - open/close 42509 (time 1787120215.80 total 1456.85 last 66.00) - open/close 43166 (time 1787120225.82 total 1466.87 last 65.54) - open/close 43866 (time 1787120235.82 total 1476.88 last 69.97) - open/close 44545 (time 1787120245.83 total 1486.89 last 67.83) - open/close 45169 (time 1787120255.85 total 1496.90 last 62.32) - open/close 45833 (time 1787120265.86 total 1506.91 last 66.33) - open/close 46486 (time 1787120275.86 total 1516.92 last 65.26) - open/close 47203 (time 1787120285.87 total 1526.92 last 71.66) - open/close 47867 (time 1787120295.87 total 1536.92 last 66.37) - open/close 48479 (time 1787120305.88 total 1546.93 last 61.17) - open/close 49089 (time 1787120315.88 total 1556.93 last 60.99) - open/close 49645 (time 1787120325.89 total 1566.94 last 55.57) total: 50000 open/close in 1572.35 seconds: 31.80 ops/second - unlinked 0 (time 1787120335 ; total 0 ; last 0) - unlinked 10000 (time 1787120499 ; total 164 ; last 164) - unlinked 20000 (time 1787120664 ; total 329 ; last 165) - unlinked 30000 (time 1787120830 ; total 495 ; last 166) - unlinked 40000 (time 1787120992 ; total 657 ; last 162) total: 50000 unlinks in 826 seconds: 60.532688 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg358-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg358-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg358-server stop mds service on oleg358-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg358-server unloading modules via unload_modules_local on: 'oleg358-server' oleg358-server: oleg358-server.virtnet: executing unload_modules_local oleg358-server: modules unloaded. pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-client: ssh exited with exit code 2 pdsh@oleg358-client: oleg358-server: ssh exited with exit code 2