== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:55:41 (1788447341) start mds service on oleg260-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions oleg260-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg260-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg260-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg260-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 95 (time 1788447437.71 total 10.49 last 9.06) - open/close 238 (time 1788447448.40 total 21.19 last 13.37) - open/close 358 (time 1788447458.42 total 31.20 last 11.98) - open/close 586 (time 1788447468.43 total 41.21 last 22.78) - open/close 710 (time 1788447478.44 total 51.22 last 12.39) - open/close 898 (time 1788447488.46 total 61.24 last 18.76) - open/close 1013 (time 1788447499.45 total 72.23 last 10.46) - open/close 1258 (time 1788447509.47 total 82.25 last 24.47) - open/close 1451 (time 1788447520.45 total 93.23 last 17.58) - open/close 1571 (time 1788447530.45 total 103.23 last 11.99) - open/close 1586 (time 1788447540.86 total 113.64 last 1.44) - open/close 1693 (time 1788447551.21 total 124.00 last 10.33) - open/close 1845 (time 1788447561.23 total 134.02 last 15.17) - open/close 1890 (time 1788447571.97 total 144.75 last 4.19) - open/close 2082 (time 1788447582.76 total 155.54 last 17.79) - open/close 2300 (time 1788447592.76 total 165.55 last 21.79) - open/close 2416 (time 1788447602.77 total 175.56 last 11.59) - open/close 2553 (time 1788447613.55 total 186.33 last 12.72) - open/close 2705 (time 1788447623.57 total 196.35 last 15.17) - open/close 2829 (time 1788447634.17 total 206.96 last 11.69) - open/close 2952 (time 1788447644.55 total 217.33 last 11.85) - open/close 3061 (time 1788447654.56 total 227.35 last 10.89) - open/close 3196 (time 1788447665.27 total 238.05 last 12.61) - open/close 3318 (time 1788447676.33 total 249.12 last 11.03) - open/close 3423 (time 1788447687.16 total 259.94 last 9.70) - open/close 3577 (time 1788447697.19 total 269.98 last 15.34) - open/close 3751 (time 1788447709.37 total 282.16 last 14.29) - open/close 3933 (time 1788447719.92 total 292.71 last 17.25) - open/close 4033 (time 1788447730.68 total 303.46 last 9.30) - open/close 4178 (time 1788447740.70 total 313.48 last 14.47) - open/close 4387 (time 1788447751.09 total 323.87 last 20.11) - open/close 4428 (time 1788447762.41 total 335.20 last 3.62) - open/close 4592 (time 1788447772.42 total 345.20 last 16.39) - open/close 4810 (time 1788447782.51 total 355.29 last 21.61) - open/close 4960 (time 1788447792.89 total 365.67 last 14.46) - open/close 5229 (time 1788447802.89 total 375.67 last 26.89) - open/close 5428 (time 1788447813.27 total 386.05 last 19.17) - open/close 5569 (time 1788447823.66 total 396.44 last 13.57) - open/close 5746 (time 1788447833.67 total 406.45 last 17.68) - open/close 5843 (time 1788447843.71 total 416.49 last 9.66) - open/close 6005 (time 1788447853.76 total 426.54 last 16.12) - open/close 6205 (time 1788447864.68 total 437.46 last 18.30) - open/close 6362 (time 1788447874.70 total 447.49 last 15.67) - open/close 6564 (time 1788447884.85 total 457.64 last 19.90) - open/close 6641 (time 1788447895.44 total 468.22 last 7.27) - open/close 6743 (time 1788447907.22 total 480.00 last 8.66) - open/close 6824 (time 1788447917.97 total 490.75 last 7.53) - open/close 6958 (time 1788447929.01 total 501.79 last 12.14) - open/close 7026 (time 1788447939.47 total 512.25 last 6.50) - open/close 7117 (time 1788447950.52 total 523.30 last 8.24) - open/close 7212 (time 1788447961.30 total 534.08 last 8.81) - open/close 7377 (time 1788447972.46 total 545.24 last 14.79) - open/close 7530 (time 1788447983.46 total 556.24 last 13.90) - open/close 7777 (time 1788447993.74 total 566.52 last 24.04) - open/close 7889 (time 1788448006.64 total 579.43 last 8.68) - open/close 7988 (time 1788448017.34 total 590.12 last 9.26) - open/close 8150 (time 1788448027.36 total 600.14 last 16.16) - open/close 8315 (time 1788448037.87 total 610.66 last 15.69) - open/close 8483 (time 1788448047.98 total 620.76 last 16.62) - open/close 8777 (time 1788448057.98 total 630.76 last 29.39) - open/close 8994 (time 1788448068.01 total 640.79 last 21.65) - open/close 9077 (time 1788448079.09 total 651.87 last 7.49) - open/close 9185 (time 1788448089.09 total 661.87 last 10.80) - open/close 9362 (time 1788448099.20 total 671.98 last 17.50) - open/close 9585 (time 1788448109.21 total 681.99 last 22.29) - open/close 9757 (time 1788448119.24 total 692.02 last 17.14) - open/close 9896 (time 1788448129.60 total 702.38 last 13.42) - open/close 10000 (time 1788448135.47 total 708.25 last 17.70) - open/close 10165 (time 1788448145.47 total 718.26 last 16.50) - open/close 10435 (time 1788448156.48 total 729.26 last 24.53) - open/close 10560 (time 1788448167.37 total 740.15 last 11.48) - open/close 10703 (time 1788448177.37 total 750.15 last 14.30) - open/close 10777 (time 1788448188.25 total 761.03 last 6.80) - open/close 10965 (time 1788448198.34 total 771.13 last 18.62) - open/close 11226 (time 1788448208.35 total 781.13 last 26.09) - open/close 11410 (time 1788448218.35 total 791.13 last 18.40) - open/close 11566 (time 1788448229.10 total 801.88 last 14.51) - open/close 11716 (time 1788448239.40 total 812.18 last 14.56) - open/close 11879 (time 1788448250.22 total 823.01 last 15.06) - open/close 12001 (time 1788448260.81 total 833.59 last 11.53) - open/close 12091 (time 1788448270.82 total 843.60 last 8.99) - open/close 12257 (time 1788448281.67 total 854.45 last 15.30) - open/close 12387 (time 1788448292.04 total 864.82 last 12.53) - open/close 12509 (time 1788448302.06 total 874.84 last 12.18) - open/close 12589 (time 1788448312.56 total 885.34 last 7.62) - open/close 12720 (time 1788448322.58 total 895.36 last 13.07) - open/close 12843 (time 1788448332.60 total 905.38 last 12.28) - open/close 12957 (time 1788448345.68 total 918.46 last 8.72) - open/close 13110 (time 1788448356.24 total 929.02 last 14.49) - open/close 13308 (time 1788448366.64 total 939.42 last 19.04) - open/close 13389 (time 1788448377.24 total 950.02 last 7.64) - open/close 13598 (time 1788448387.24 total 960.02 last 20.89) - open/close 13736 (time 1788448397.75 total 970.54 last 13.12) - open/close 13896 (time 1788448408.63 total 981.41 last 14.72) - open/close 14094 (time 1788448419.50 total 992.28 last 18.21) - open/close 14175 (time 1788448429.93 total 1002.71 last 7.77) - open/close 14296 (time 1788448439.94 total 1012.73 last 12.08) - open/close 14398 (time 1788448449.96 total 1022.74 last 10.19) - open/close 14561 (time 1788448459.98 total 1032.76 last 16.27) - open/close 14668 (time 1788448470.39 total 1043.17 last 10.28) - open/close 14766 (time 1788448480.74 total 1053.53 last 9.46) - open/close 14973 (time 1788448491.34 total 1064.12 last 19.54) - open/close 15195 (time 1788448502.13 total 1074.91 last 20.56) - open/close 15375 (time 1788448513.08 total 1085.86 last 16.44) - open/close 15585 (time 1788448523.09 total 1095.87 last 20.98) - open/close 15758 (time 1788448533.12 total 1105.90 last 17.26) - open/close 15900 (time 1788448543.28 total 1116.06 last 13.98) - open/close 16200 (time 1788448553.29 total 1126.07 last 29.97) - open/close 16857 (time 1788448563.30 total 1136.09 last 65.58) - open/close 17378 (time 1788448573.31 total 1146.09 last 52.08) - open/close 18054 (time 1788448583.31 total 1156.09 last 67.58) - open/close 18744 (time 1788448593.32 total 1166.10 last 68.93) - open/close 19365 (time 1788448603.33 total 1176.11 last 62.06) - open/close 20000 (time 1788448611.76 total 1184.54 last 75.33) - open/close 20637 (time 1788448621.76 total 1194.54 last 63.67) - open/close 21305 (time 1788448631.79 total 1204.57 last 66.64) - open/close 21833 (time 1788448641.81 total 1214.59 last 52.69) - open/close 22363 (time 1788448651.81 total 1224.59 last 53.00) - open/close 23111 (time 1788448661.81 total 1234.60 last 74.77) - open/close 23703 (time 1788448671.83 total 1244.61 last 59.12) - open/close 24305 (time 1788448681.83 total 1254.61 last 60.20) - open/close 24964 (time 1788448691.84 total 1264.62 last 65.83) - open/close 25666 (time 1788448701.85 total 1274.63 last 70.13) - open/close 26372 (time 1788448711.85 total 1284.63 last 70.58) - open/close 26999 (time 1788448721.86 total 1294.64 last 62.64) - open/close 27664 (time 1788448731.89 total 1304.67 last 66.28) - open/close 28348 (time 1788448741.89 total 1314.68 last 68.38) - open/close 29091 (time 1788448751.90 total 1324.68 last 74.26) - open/close 29742 (time 1788448761.91 total 1334.69 last 65.02) - open/close 30000 (time 1788448765.43 total 1338.22 last 73.23) - open/close 30677 (time 1788448775.44 total 1348.22 last 67.67) - open/close 31361 (time 1788448785.44 total 1358.22 last 68.38) - open/close 32085 (time 1788448795.45 total 1368.23 last 72.34) - open/close 32732 (time 1788448805.45 total 1378.24 last 64.67) - open/close 33487 (time 1788448815.47 total 1388.25 last 75.38) - open/close 34192 (time 1788448825.50 total 1398.28 last 70.31) - open/close 34879 (time 1788448835.51 total 1408.29 last 68.61) - open/close 35600 (time 1788448845.53 total 1418.31 last 71.96) - open/close 36290 (time 1788448855.53 total 1428.31 last 68.98) - open/close 36995 (time 1788448865.54 total 1438.32 last 70.42) - open/close 37571 (time 1788448875.56 total 1448.34 last 57.50) - open/close 38161 (time 1788448885.57 total 1458.35 last 58.97) - open/close 38684 (time 1788448895.57 total 1468.35 last 52.26) - open/close 39362 (time 1788448905.58 total 1478.36 last 67.78) - open/close 40000 (time 1788448914.28 total 1487.06 last 73.29) - open/close 40736 (time 1788448924.28 total 1497.06 last 73.59) - open/close 41354 (time 1788448934.29 total 1507.07 last 61.78) - open/close 41943 (time 1788448944.29 total 1517.07 last 58.88) - open/close 42647 (time 1788448954.30 total 1527.08 last 70.34) - open/close 43288 (time 1788448964.31 total 1537.09 last 64.05) - open/close 44043 (time 1788448974.31 total 1547.10 last 75.43) - open/close 44778 (time 1788448984.33 total 1557.11 last 73.38) - open/close 45473 (time 1788448994.34 total 1567.12 last 69.47) - open/close 46184 (time 1788449004.35 total 1577.13 last 71.01) - open/close 46872 (time 1788449014.35 total 1587.13 last 68.79) - open/close 47546 (time 1788449024.36 total 1597.14 last 67.33) - open/close 48277 (time 1788449034.38 total 1607.16 last 73.00) - open/close 48936 (time 1788449044.38 total 1617.16 last 65.87) - open/close 49613 (time 1788449054.39 total 1627.17 last 67.65) total: 50000 open/close in 1633.10 seconds: 30.62 ops/second - unlinked 0 (time 1788449064 ; total 0 ; last 0) - unlinked 10000 (time 1788449228 ; total 164 ; last 164) - unlinked 20000 (time 1788449394 ; total 330 ; last 166) - unlinked 30000 (time 1788449566 ; total 502 ; last 172) - unlinked 40000 (time 1788449736 ; total 672 ; last 170) total: 50000 unlinks in 836 seconds: 59.808613 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server unloading modules via unload_modules_local on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing unload_modules_local oleg260-server: modules unloaded. pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2