== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:03:25 (1787850205) start mds service on oleg449-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing load_modules_local oleg449-server: Loading modules from /home/green/git/lustre-release/lustre oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg449-server: detected 4 online CPUs by sysfs oleg449-server: MODOPTS_LIBCFS= oleg449-server: Force libcfs to create 2 CPU partitions oleg449-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg449-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg449-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg449-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-OST0000 oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg449-client.virtnet: -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 167 (time 1787850303.94 total 10.77 last 15.50) - open/close 261 (time 1787850314.63 total 21.47 last 8.79) - open/close 364 (time 1787850327.04 total 33.87 last 8.31) - open/close 400 (time 1787850337.46 total 44.30 last 3.45) - open/close 600 (time 1787850347.49 total 54.32 last 19.96) - open/close 786 (time 1787850357.50 total 64.34 last 18.57) - open/close 1030 (time 1787850367.54 total 74.37 last 24.32) - open/close 1166 (time 1787850377.75 total 84.59 last 13.31) - open/close 1413 (time 1787850388.36 total 95.20 last 23.29) - open/close 1625 (time 1787850398.38 total 105.22 last 21.16) - open/close 1779 (time 1787850408.95 total 115.79 last 14.57) - open/close 1966 (time 1787850419.16 total 126.00 last 18.31) - open/close 2152 (time 1787850429.17 total 136.00 last 18.59) - open/close 2428 (time 1787850439.17 total 146.01 last 27.59) - open/close 2506 (time 1787850449.17 total 156.01 last 7.80) - open/close 2637 (time 1787850462.21 total 169.04 last 10.05) - open/close 2782 (time 1787850472.33 total 179.16 last 14.33) - open/close 2967 (time 1787850482.33 total 189.17 last 18.49) - open/close 3198 (time 1787850494.59 total 201.43 last 18.85) - open/close 3401 (time 1787850504.91 total 211.75 last 19.67) - open/close 3618 (time 1787850515.53 total 222.36 last 20.44) - open/close 3783 (time 1787850526.35 total 233.18 last 15.25) - open/close 3879 (time 1787850536.64 total 243.48 last 9.32) - open/close 3992 (time 1787850547.20 total 254.04 last 10.70) - open/close 4206 (time 1787850557.38 total 264.21 last 21.04) - open/close 4452 (time 1787850568.39 total 275.23 last 22.33) - open/close 4616 (time 1787850578.83 total 285.67 last 15.71) - open/close 4873 (time 1787850588.87 total 295.70 last 25.60) - open/close 5136 (time 1787850599.49 total 306.33 last 24.76) - open/close 5366 (time 1787850609.49 total 316.33 last 23.00) - open/close 5560 (time 1787850619.52 total 326.36 last 19.34) - open/close 5813 (time 1787850629.53 total 336.36 last 25.29) - open/close 5933 (time 1787850640.53 total 347.37 last 10.91) - open/close 6157 (time 1787850650.53 total 357.37 last 22.39) - open/close 6438 (time 1787850663.31 total 370.14 last 22.00) - open/close 6538 (time 1787850673.79 total 380.62 last 9.54) - open/close 6704 (time 1787850684.75 total 391.59 last 15.14) - open/close 6830 (time 1787850695.36 total 402.20 last 11.88) - open/close 6921 (time 1787850706.08 total 412.92 last 8.49) - open/close 7044 (time 1787850716.36 total 423.19 last 11.97) - open/close 7186 (time 1787850726.97 total 433.80 last 13.38) - open/close 7327 (time 1787850737.48 total 444.31 last 13.42) - open/close 7493 (time 1787850747.53 total 454.36 last 16.52) - open/close 7598 (time 1787850757.64 total 464.47 last 10.38) - open/close 7684 (time 1787850768.27 total 475.11 last 8.09) - open/close 7872 (time 1787850778.86 total 485.70 last 17.76) - open/close 8003 (time 1787850794.39 total 501.22 last 8.44) - open/close 8174 (time 1787850804.41 total 511.25 last 17.05) - open/close 8308 (time 1787850815.16 total 522.00 last 12.47) - open/close 8420 (time 1787850828.36 total 535.20 last 8.48) - open/close 8547 (time 1787850838.37 total 545.21 last 12.69) - open/close 8672 (time 1787850848.37 total 555.21 last 12.50) - open/close 8753 (time 1787850858.43 total 565.26 last 8.06) - open/close 8847 (time 1787850868.43 total 575.27 last 9.39) - open/close 8986 (time 1787850878.79 total 585.62 last 13.43) - open/close 9112 (time 1787850889.05 total 595.88 last 12.28) - open/close 9269 (time 1787850899.06 total 605.90 last 15.68) - open/close 9487 (time 1787850909.65 total 616.49 last 20.58) - open/close 9680 (time 1787850920.13 total 626.96 last 18.43) - open/close 9794 (time 1787850933.12 total 639.95 last 8.78) - open/close 9873 (time 1787850943.34 total 650.17 last 7.73) - open/close 9985 (time 1787850955.00 total 661.84 last 9.60) - open/close 10152 (time 1787850965.92 total 672.75 last 15.30) - open/close 10394 (time 1787850975.93 total 682.77 last 24.17) - open/close 10698 (time 1787850985.94 total 692.78 last 30.37) - open/close 10908 (time 1787850996.51 total 703.34 last 19.87) - open/close 11116 (time 1787851007.49 total 714.32 last 18.95) - open/close 11246 (time 1787851017.92 total 724.75 last 12.46) - open/close 11414 (time 1787851028.25 total 735.09 last 16.26) - open/close 11488 (time 1787851038.65 total 745.48 last 7.12) - open/close 11696 (time 1787851049.64 total 756.48 last 18.92) - open/close 11878 (time 1787851060.59 total 767.42 last 16.63) - open/close 12064 (time 1787851070.60 total 777.43 last 18.58) - open/close 12296 (time 1787851080.61 total 787.45 last 23.17) - open/close 12436 (time 1787851090.64 total 797.47 last 13.97) - open/close 12505 (time 1787851100.83 total 807.67 last 6.77) - open/close 12667 (time 1787851113.44 total 820.27 last 12.85) - open/close 12783 (time 1787851123.45 total 830.28 last 11.59) - open/close 12959 (time 1787851134.49 total 841.32 last 15.94) - open/close 13039 (time 1787851145.24 total 852.07 last 7.44) - open/close 13183 (time 1787851155.25 total 862.08 last 14.38) - open/close 13239 (time 1787851165.37 total 872.21 last 5.53) - open/close 13346 (time 1787851176.13 total 882.97 last 9.95) - open/close 13430 (time 1787851187.01 total 893.84 last 7.72) - open/close 13553 (time 1787851197.90 total 904.73 last 11.30) - open/close 13685 (time 1787851210.45 total 917.29 last 10.52) - open/close 13806 (time 1787851220.69 total 927.53 last 11.81) - open/close 13925 (time 1787851230.72 total 937.56 last 11.87) - open/close 14088 (time 1787851240.72 total 947.56 last 16.29) - open/close 14245 (time 1787851250.74 total 957.57 last 15.68) - open/close 14403 (time 1787851261.13 total 967.97 last 15.20) - open/close 14588 (time 1787851271.14 total 977.97 last 18.49) - open/close 14760 (time 1787851281.17 total 988.00 last 17.15) - open/close 14888 (time 1787851291.20 total 998.03 last 12.76) - open/close 14981 (time 1787851301.38 total 1008.21 last 9.14) - open/close 15097 (time 1787851312.44 total 1019.28 last 10.49) - open/close 15217 (time 1787851322.45 total 1029.28 last 11.99) - open/close 15283 (time 1787851332.46 total 1039.29 last 6.59) - open/close 15397 (time 1787851342.46 total 1049.30 last 11.40) - open/close 15503 (time 1787851353.55 total 1060.39 last 9.56) - open/close 15603 (time 1787851364.43 total 1071.27 last 9.19) - open/close 15720 (time 1787851374.59 total 1081.43 last 11.52) - open/close 15825 (time 1787851387.74 total 1094.58 last 7.98) - open/close 16401 (time 1787851397.75 total 1104.59 last 57.57) - open/close 16866 (time 1787851407.75 total 1114.59 last 46.48) - open/close 17365 (time 1787851417.77 total 1124.60 last 49.84) - open/close 17910 (time 1787851427.77 total 1134.61 last 54.47) - open/close 18419 (time 1787851437.80 total 1144.63 last 50.77) - open/close 18897 (time 1787851447.81 total 1154.64 last 47.75) - open/close 19381 (time 1787851457.82 total 1164.65 last 48.36) - open/close 19967 (time 1787851467.83 total 1174.66 last 58.54) - open/close 20542 (time 1787851477.83 total 1184.67 last 57.46) - open/close 21106 (time 1787851487.85 total 1194.68 last 56.33) - open/close 21708 (time 1787851497.87 total 1204.70 last 60.07) - open/close 22246 (time 1787851507.88 total 1214.71 last 53.75) - open/close 22859 (time 1787851517.90 total 1224.73 last 61.19) - open/close 23453 (time 1787851527.92 total 1234.76 last 59.24) - open/close 24048 (time 1787851537.93 total 1244.77 last 59.44) - open/close 24661 (time 1787851547.93 total 1254.77 last 61.30) - open/close 25236 (time 1787851557.95 total 1264.78 last 57.41) - open/close 25809 (time 1787851567.95 total 1274.79 last 57.29) - open/close 26384 (time 1787851577.95 total 1284.79 last 57.48) - open/close 26938 (time 1787851587.97 total 1294.80 last 55.34) - open/close 27485 (time 1787851597.97 total 1304.80 last 54.69) - open/close 28061 (time 1787851607.97 total 1314.80 last 57.60) - open/close 28596 (time 1787851617.98 total 1324.82 last 53.42) - open/close 29125 (time 1787851627.99 total 1334.82 last 52.87) - open/close 29590 (time 1787851638.00 total 1344.83 last 46.47) - open/close 29989 (time 1787851648.01 total 1354.84 last 39.85) - open/close 30530 (time 1787851658.03 total 1364.86 last 54.00) - open/close 31199 (time 1787851668.05 total 1374.89 last 66.74) - open/close 31821 (time 1787851678.06 total 1384.89 last 62.18) - open/close 32353 (time 1787851688.06 total 1394.89 last 53.19) - open/close 32775 (time 1787851698.06 total 1404.89 last 42.19) - open/close 33281 (time 1787851708.06 total 1414.90 last 50.59) - open/close 33784 (time 1787851718.06 total 1424.90 last 50.28) - open/close 34365 (time 1787851728.08 total 1434.91 last 58.03) - open/close 34859 (time 1787851738.10 total 1444.93 last 49.30) - open/close 35375 (time 1787851748.10 total 1454.94 last 51.57) - open/close 35873 (time 1787851758.12 total 1464.95 last 49.73) - open/close 36428 (time 1787851768.13 total 1474.96 last 55.44) - open/close 36940 (time 1787851778.15 total 1484.99 last 51.08) - open/close 37517 (time 1787851788.15 total 1494.99 last 57.68) - open/close 38113 (time 1787851798.18 total 1505.02 last 59.45) - open/close 38713 (time 1787851808.19 total 1515.02 last 59.95) - open/close 39330 (time 1787851818.19 total 1525.02 last 61.70) - open/close 39883 (time 1787851828.20 total 1535.04 last 55.23) - open/close 40418 (time 1787851838.21 total 1545.05 last 53.44) - open/close 40964 (time 1787851848.23 total 1555.07 last 54.51) - open/close 41500 (time 1787851858.24 total 1565.07 last 53.56) - open/close 41989 (time 1787851868.25 total 1575.09 last 48.82) - open/close 42517 (time 1787851878.28 total 1585.11 last 52.67) - open/close 43086 (time 1787851888.28 total 1595.12 last 56.87) - open/close 43659 (time 1787851898.29 total 1605.13 last 57.24) - open/close 44231 (time 1787851908.30 total 1615.13 last 57.18) - open/close 44813 (time 1787851918.30 total 1625.14 last 58.17) - open/close 45345 (time 1787851928.31 total 1635.14 last 53.17) - open/close 45811 (time 1787851938.31 total 1645.15 last 46.56) - open/close 46221 (time 1787851948.31 total 1655.15 last 41.00) - open/close 46587 (time 1787851958.33 total 1665.17 last 36.53) - open/close 47107 (time 1787851968.35 total 1675.19 last 51.91) - open/close 47605 (time 1787851978.35 total 1685.19 last 49.78) - open/close 48150 (time 1787851988.39 total 1695.23 last 54.29) - open/close 48637 (time 1787851998.41 total 1705.24 last 48.63) - open/close 49119 (time 1787852008.43 total 1715.27 last 48.07) - open/close 49610 (time 1787852018.45 total 1725.28 last 49.04) total: 50000 open/close in 1732.98 seconds: 28.85 ops/second - unlinked 0 (time 1787852031 ; total 0 ; last 0) - unlinked 10000 (time 1787852240 ; total 209 ; last 209) - unlinked 20000 (time 1787852455 ; total 424 ; last 215) - unlinked 30000 (time 1787852660 ; total 629 ; last 205) - unlinked 40000 (time 1787852877 ; total 846 ; last 217) total: 50000 unlinks in 1047 seconds: 47.755493 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg449-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg449-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg449-server stop mds service on oleg449-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg449-server unloading modules via unload_modules_local on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing unload_modules_local oleg449-server: modules unloaded. pdsh@oleg449-client: oleg449-client: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-client: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2