== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:24:59 (1780338299) start mds service on oleg131-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg131-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg131-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 153 (time 1780338386.18 total 10.75 last 14.23) - open/close 329 (time 1780338397.04 total 21.61 last 16.20) - open/close 507 (time 1780338407.05 total 31.62 last 17.79) - open/close 612 (time 1780338417.05 total 41.63 last 10.50) - open/close 768 (time 1780338427.06 total 51.63 last 15.59) - open/close 983 (time 1780338437.57 total 62.14 last 20.45) - open/close 1065 (time 1780338447.75 total 72.33 last 8.05) - open/close 1136 (time 1780338457.76 total 82.33 last 7.10) - open/close 1377 (time 1780338468.82 total 93.39 last 21.80) - open/close 1658 (time 1780338479.22 total 103.80 last 26.99) - open/close 1848 (time 1780338490.12 total 114.69 last 17.44) - open/close 2099 (time 1780338502.34 total 126.91 last 20.54) - open/close 2222 (time 1780338513.34 total 137.91 last 11.18) - open/close 2449 (time 1780338523.35 total 147.92 last 22.67) - open/close 2632 (time 1780338533.60 total 158.17 last 17.86) - open/close 2793 (time 1780338543.73 total 168.30 last 15.88) - open/close 2894 (time 1780338555.34 total 179.91 last 8.70) - open/close 3039 (time 1780338565.59 total 190.17 last 14.14) - open/close 3117 (time 1780338575.60 total 200.17 last 7.79) - open/close 3283 (time 1780338585.60 total 210.18 last 16.59) - open/close 3394 (time 1780338595.61 total 220.18 last 11.09) - open/close 3462 (time 1780338608.32 total 232.89 last 5.35) - open/close 3655 (time 1780338618.71 total 243.28 last 18.58) - open/close 3771 (time 1780338628.72 total 253.29 last 11.59) - open/close 3992 (time 1780338639.03 total 263.60 last 21.44) - open/close 4189 (time 1780338649.38 total 273.95 last 19.04) - open/close 4335 (time 1780338659.43 total 284.00 last 14.52) - open/close 4458 (time 1780338669.45 total 294.02 last 12.28) - open/close 4647 (time 1780338679.46 total 304.03 last 18.88) - open/close 4831 (time 1780338690.17 total 314.74 last 17.19) - open/close 4949 (time 1780338700.21 total 324.78 last 11.75) - open/close 5139 (time 1780338710.65 total 335.22 last 18.20) - open/close 5354 (time 1780338720.84 total 345.41 last 21.10) - open/close 5462 (time 1780338731.28 total 355.85 last 10.35) - open/close 5555 (time 1780338741.96 total 366.53 last 8.70) - open/close 5633 (time 1780338752.40 total 376.97 last 7.48) - open/close 5845 (time 1780338762.44 total 387.01 last 21.12) - open/close 6000 (time 1780338772.45 total 397.02 last 15.48) - open/close 6147 (time 1780338782.93 total 407.50 last 14.02) - open/close 6278 (time 1780338795.32 total 419.89 last 10.57) - open/close 6464 (time 1780338806.29 total 430.86 last 16.96) - open/close 6607 (time 1780338816.29 total 440.86 last 14.30) - open/close 6736 (time 1780338826.59 total 451.16 last 12.53) - open/close 6875 (time 1780338836.69 total 461.26 last 13.76) - open/close 6972 (time 1780338847.48 total 472.05 last 8.99) - open/close 7086 (time 1780338860.51 total 485.08 last 8.75) - open/close 7257 (time 1780338870.51 total 495.08 last 17.09) - open/close 7422 (time 1780338880.58 total 505.15 last 16.38) - open/close 7524 (time 1780338890.61 total 515.18 last 10.18) - open/close 7664 (time 1780338900.63 total 525.20 last 13.97) - open/close 7822 (time 1780338912.38 total 536.95 last 13.44) - open/close 7945 (time 1780338923.90 total 548.47 last 10.68) - open/close 8075 (time 1780338934.66 total 559.23 last 12.08) - open/close 8184 (time 1780338945.87 total 570.44 last 9.73) - open/close 8269 (time 1780338956.49 total 581.06 last 8.00) - open/close 8450 (time 1780338966.66 total 591.23 last 17.80) - open/close 8521 (time 1780338977.69 total 602.26 last 6.44) - open/close 8610 (time 1780338988.61 total 613.18 last 8.15) - open/close 8722 (time 1780338999.39 total 623.96 last 10.39) - open/close 8916 (time 1780339009.93 total 634.50 last 18.40) - open/close 8996 (time 1780339019.95 total 644.52 last 7.99) - open/close 9148 (time 1780339030.44 total 655.01 last 14.49) - open/close 9288 (time 1780339040.45 total 665.02 last 13.98) - open/close 9437 (time 1780339051.32 total 675.89 last 13.71) - open/close 9507 (time 1780339061.60 total 686.17 last 6.81) - open/close 9694 (time 1780339072.53 total 697.10 last 17.11) - open/close 9916 (time 1780339082.54 total 707.11 last 22.18) - open/close 10000 (time 1780339086.66 total 711.23 last 20.41) - open/close 10218 (time 1780339098.50 total 723.07 last 18.40) - open/close 10434 (time 1780339108.67 total 733.24 last 21.24) - open/close 10488 (time 1780339119.17 total 743.74 last 5.14) - open/close 10608 (time 1780339129.18 total 753.75 last 11.99) - open/close 10817 (time 1780339139.86 total 764.43 last 19.57) - open/close 10951 (time 1780339149.87 total 774.45 last 13.38) - open/close 11081 (time 1780339160.03 total 784.60 last 12.80) - open/close 11234 (time 1780339172.92 total 797.49 last 11.87) - open/close 11434 (time 1780339183.36 total 807.93 last 19.16) - open/close 11642 (time 1780339193.37 total 817.94 last 20.79) - open/close 11882 (time 1780339203.37 total 827.94 last 24.00) - open/close 12054 (time 1780339213.38 total 837.95 last 17.17) - open/close 12175 (time 1780339223.39 total 847.96 last 12.10) - open/close 12297 (time 1780339234.44 total 859.01 last 11.04) - open/close 12417 (time 1780339244.46 total 869.03 last 11.98) - open/close 12685 (time 1780339254.46 total 879.03 last 26.80) - open/close 12851 (time 1780339264.47 total 889.04 last 16.58) - open/close 13070 (time 1780339274.50 total 899.07 last 21.85) - open/close 13187 (time 1780339284.62 total 909.19 last 11.55) - open/close 13351 (time 1780339294.62 total 919.20 last 16.40) - open/close 13529 (time 1780339305.02 total 929.59 last 17.12) - open/close 13694 (time 1780339315.39 total 939.96 last 15.91) - open/close 13821 (time 1780339326.21 total 950.78 last 11.74) - open/close 13941 (time 1780339336.59 total 961.16 last 11.56) - open/close 14049 (time 1780339347.10 total 971.67 last 10.28) - open/close 14163 (time 1780339357.11 total 981.68 last 11.38) - open/close 14264 (time 1780339367.13 total 991.70 last 10.08) - open/close 14376 (time 1780339377.14 total 1001.71 last 11.19) - open/close 14520 (time 1780339387.80 total 1012.37 last 13.50) - open/close 14617 (time 1780339398.41 total 1022.98 last 9.15) - open/close 14859 (time 1780339408.88 total 1033.45 last 23.10) - open/close 14997 (time 1780339418.89 total 1043.46 last 13.79) - open/close 15179 (time 1780339428.90 total 1053.47 last 18.18) - open/close 15276 (time 1780339439.58 total 1064.15 last 9.09) - open/close 15474 (time 1780339450.56 total 1075.13 last 18.02) - open/close 15681 (time 1780339461.25 total 1085.82 last 19.37) - open/close 15976 (time 1780339471.27 total 1095.84 last 29.44) - open/close 16624 (time 1780339481.27 total 1105.85 last 64.76) - open/close 17221 (time 1780339491.27 total 1115.85 last 59.70) - open/close 17756 (time 1780339501.28 total 1125.85 last 53.47) - open/close 18369 (time 1780339511.28 total 1135.86 last 61.27) - open/close 18993 (time 1780339521.29 total 1145.86 last 62.36) - open/close 19668 (time 1780339531.30 total 1155.87 last 67.47) - open/close 20000 (time 1780339536.54 total 1161.11 last 63.35) - open/close 20542 (time 1780339546.55 total 1171.13 last 54.10) - open/close 21184 (time 1780339556.56 total 1181.13 last 64.15) - open/close 21784 (time 1780339566.57 total 1191.14 last 59.96) - open/close 22460 (time 1780339576.58 total 1201.15 last 67.55) - open/close 23101 (time 1780339586.58 total 1211.16 last 64.06) - open/close 23720 (time 1780339596.59 total 1221.16 last 61.85) - open/close 24335 (time 1780339606.60 total 1231.17 last 61.45) - open/close 24909 (time 1780339616.63 total 1241.20 last 57.26) - open/close 25521 (time 1780339626.67 total 1251.24 last 60.94) - open/close 26163 (time 1780339636.68 total 1261.25 last 64.13) - open/close 26783 (time 1780339646.71 total 1271.28 last 61.79) - open/close 27384 (time 1780339656.73 total 1281.30 last 59.99) - open/close 27988 (time 1780339666.73 total 1291.30 last 60.39) - open/close 28659 (time 1780339676.74 total 1301.31 last 67.03) - open/close 29166 (time 1780339686.74 total 1311.32 last 50.69) - open/close 29722 (time 1780339696.75 total 1321.32 last 55.59) - open/close 30000 (time 1780339701.67 total 1326.24 last 56.43) - open/close 30561 (time 1780339711.67 total 1336.24 last 56.10) - open/close 31190 (time 1780339721.67 total 1346.24 last 62.90) - open/close 31840 (time 1780339731.68 total 1356.25 last 64.93) - open/close 32457 (time 1780339741.70 total 1366.27 last 61.61) - open/close 33048 (time 1780339751.70 total 1376.27 last 59.08) - open/close 33701 (time 1780339761.71 total 1386.28 last 65.27) - open/close 34325 (time 1780339771.72 total 1396.29 last 62.33) - open/close 34897 (time 1780339781.72 total 1406.30 last 57.16) - open/close 35529 (time 1780339791.72 total 1416.30 last 63.20) - open/close 36223 (time 1780339801.73 total 1426.30 last 69.36) - open/close 36783 (time 1780339811.76 total 1436.33 last 55.86) - open/close 37337 (time 1780339821.78 total 1446.36 last 55.24) - open/close 37939 (time 1780339831.80 total 1456.37 last 60.12) - open/close 38523 (time 1780339841.83 total 1466.40 last 58.20) - open/close 39182 (time 1780339851.86 total 1476.43 last 65.70) - open/close 39841 (time 1780339861.88 total 1486.45 last 65.80) - open/close 40000 (time 1780339864.42 total 1489.00 last 62.43) - open/close 40640 (time 1780339874.43 total 1499.00 last 63.98) - open/close 41223 (time 1780339884.43 total 1509.00 last 58.30) - open/close 41737 (time 1780339894.44 total 1519.01 last 51.32) - open/close 42262 (time 1780339904.47 total 1529.04 last 52.36) - open/close 42852 (time 1780339914.48 total 1539.05 last 58.92) - open/close 43424 (time 1780339924.48 total 1549.06 last 57.20) - open/close 44066 (time 1780339934.50 total 1559.07 last 64.10) - open/close 44634 (time 1780339944.51 total 1569.08 last 56.75) - open/close 45175 (time 1780339954.53 total 1579.10 last 54.00) - open/close 45716 (time 1780339964.57 total 1589.14 last 53.90) - open/close 46264 (time 1780339974.57 total 1599.14 last 54.80) - open/close 46723 (time 1780339984.58 total 1609.15 last 45.85) - open/close 47376 (time 1780339994.59 total 1619.16 last 65.23) - open/close 48011 (time 1780340004.60 total 1629.17 last 63.43) - open/close 48539 (time 1780340014.61 total 1639.18 last 52.76) - open/close 49194 (time 1780340024.62 total 1649.19 last 65.42) - open/close 49764 (time 1780340034.63 total 1659.20 last 56.95) total: 50000 open/close in 1662.52 seconds: 30.07 ops/second - unlinked 0 (time 1780340043 ; total 0 ; last 0) - unlinked 10000 (time 1780340225 ; total 182 ; last 182) - unlinked 20000 (time 1780340392 ; total 349 ; last 167) - unlinked 30000 (time 1780340572 ; total 529 ; last 180) - unlinked 40000 (time 1780340737 ; total 694 ; last 165) total: 50000 unlinks in 863 seconds: 57.937428 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server unloading modules via unload_modules_local on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local oleg131-server: modules unloaded. pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2