== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:55:56 (1787752556) start mds service on oleg105-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing load_modules_local oleg105-server: Loading modules from /home/green/git/lustre-release/lustre oleg105-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg105-server: detected 4 online CPUs by sysfs oleg105-server: MODOPTS_LIBCFS= oleg105-server: Force libcfs to create 2 CPU partitions oleg105-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg105-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg105-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg105-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg105-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-OST0000 oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 164 (time 1787752684.56 total 10.35 last 15.85) - open/close 294 (time 1787752694.57 total 20.36 last 12.99) - open/close 479 (time 1787752705.55 total 31.34 last 16.84) - open/close 635 (time 1787752717.87 total 43.66 last 12.66) - open/close 825 (time 1787752728.74 total 54.53 last 17.48) - open/close 1048 (time 1787752738.76 total 64.55 last 22.26) - open/close 1138 (time 1787752748.78 total 74.57 last 8.98) - open/close 1412 (time 1787752758.79 total 84.58 last 27.38) - open/close 1599 (time 1787752768.79 total 94.59 last 18.69) - open/close 1825 (time 1787752778.80 total 104.59 last 22.60) - open/close 1914 (time 1787752789.28 total 115.07 last 8.49) - open/close 2051 (time 1787752799.28 total 125.07 last 13.70) - open/close 2195 (time 1787752809.30 total 135.09 last 14.38) - open/close 2444 (time 1787752820.08 total 145.87 last 23.10) - open/close 2509 (time 1787752831.05 total 156.84 last 5.92) - open/close 2672 (time 1787752841.50 total 167.29 last 15.61) - open/close 2879 (time 1787752853.98 total 179.77 last 16.59) - open/close 3123 (time 1787752863.98 total 189.77 last 24.39) - open/close 3331 (time 1787752873.99 total 199.78 last 20.77) - open/close 3526 (time 1787752884.31 total 210.10 last 18.90) - open/close 3684 (time 1787752894.89 total 220.68 last 14.94) - open/close 3908 (time 1787752905.13 total 230.92 last 21.86) - open/close 4021 (time 1787752915.14 total 240.93 last 11.29) - open/close 4272 (time 1787752925.59 total 251.38 last 24.01) - open/close 4417 (time 1787752936.09 total 261.88 last 13.82) - open/close 4576 (time 1787752946.10 total 271.89 last 15.87) - open/close 4756 (time 1787752957.15 total 282.94 last 16.30) - open/close 4941 (time 1787752967.16 total 292.95 last 18.49) - open/close 5049 (time 1787752977.81 total 303.60 last 10.14) - open/close 5268 (time 1787752987.83 total 313.62 last 21.86) - open/close 5356 (time 1787752998.48 total 324.27 last 8.26) - open/close 5455 (time 1787753009.03 total 334.83 last 9.38) - open/close 5610 (time 1787753019.04 total 344.83 last 15.49) - open/close 5742 (time 1787753029.06 total 354.85 last 13.18) - open/close 5906 (time 1787753039.07 total 364.86 last 16.38) - open/close 6003 (time 1787753049.90 total 375.69 last 8.96) - open/close 6116 (time 1787753059.93 total 385.72 last 11.27) - open/close 6229 (time 1787753070.74 total 396.53 last 10.45) - open/close 6513 (time 1787753080.76 total 406.55 last 28.36) - open/close 6703 (time 1787753090.76 total 416.55 last 18.99) - open/close 6854 (time 1787753101.41 total 427.20 last 14.19) - open/close 6972 (time 1787753111.42 total 437.21 last 11.79) - open/close 7197 (time 1787753122.22 total 448.01 last 20.83) - open/close 7333 (time 1787753132.23 total 458.02 last 13.59) - open/close 7534 (time 1787753142.34 total 468.13 last 19.87) - open/close 7671 (time 1787753152.35 total 478.14 last 13.70) - open/close 7775 (time 1787753162.37 total 488.16 last 10.38) - open/close 7990 (time 1787753173.21 total 499.00 last 19.83) - open/close 8182 (time 1787753183.88 total 509.67 last 17.99) - open/close 8328 (time 1787753194.22 total 520.01 last 14.12) - open/close 8498 (time 1787753204.23 total 530.02 last 16.97) - open/close 8654 (time 1787753214.36 total 540.15 last 15.41) - open/close 8746 (time 1787753225.36 total 551.15 last 8.36) - open/close 8849 (time 1787753235.38 total 561.17 last 10.28) - open/close 8977 (time 1787753245.39 total 571.18 last 12.79) - open/close 9146 (time 1787753255.41 total 581.20 last 16.87) - open/close 9277 (time 1787753265.75 total 591.54 last 12.68) - open/close 9433 (time 1787753276.13 total 601.92 last 15.03) - open/close 9555 (time 1787753286.29 total 612.08 last 12.00) - open/close 9681 (time 1787753296.33 total 622.12 last 12.55) - open/close 9799 (time 1787753306.80 total 632.59 last 11.28) - open/close 9908 (time 1787753317.74 total 643.53 last 9.96) - open/close 10000 (time 1787753321.66 total 647.45 last 23.50) - open/close 10223 (time 1787753331.66 total 657.45 last 22.29) - open/close 10384 (time 1787753341.66 total 667.45 last 16.10) - open/close 10581 (time 1787753352.66 total 678.45 last 17.91) - open/close 10696 (time 1787753363.10 total 688.89 last 11.02) - open/close 10865 (time 1787753373.78 total 699.57 last 15.82) - open/close 11014 (time 1787753384.21 total 710.00 last 14.29) - open/close 11179 (time 1787753394.85 total 720.64 last 15.50) - open/close 11434 (time 1787753405.52 total 731.31 last 23.91) - open/close 11603 (time 1787753415.95 total 741.74 last 16.20) - open/close 11738 (time 1787753426.13 total 751.92 last 13.26) - open/close 11856 (time 1787753436.18 total 761.97 last 11.73) - open/close 12093 (time 1787753446.82 total 772.61 last 22.27) - open/close 12222 (time 1787753457.82 total 783.61 last 11.73) - open/close 12402 (time 1787753468.89 total 794.68 last 16.25) - open/close 12538 (time 1787753479.84 total 805.63 last 12.43) - open/close 12604 (time 1787753491.48 total 817.27 last 5.67) - open/close 12767 (time 1787753502.25 total 828.04 last 15.14) - open/close 12832 (time 1787753512.27 total 838.06 last 6.48) - open/close 12915 (time 1787753522.90 total 848.69 last 7.81) - open/close 13017 (time 1787753535.33 total 861.12 last 8.21) - open/close 13147 (time 1787753545.49 total 871.28 last 12.80) - open/close 13301 (time 1787753555.62 total 881.41 last 15.21) - open/close 13476 (time 1787753566.04 total 891.83 last 16.78) - open/close 13628 (time 1787753576.55 total 902.34 last 14.47) - open/close 13753 (time 1787753586.86 total 912.65 last 12.12) - open/close 13851 (time 1787753596.93 total 922.72 last 9.73) - open/close 13932 (time 1787753607.64 total 933.43 last 7.57) - open/close 14057 (time 1787753617.64 total 943.43 last 12.49) - open/close 14150 (time 1787753627.95 total 953.74 last 9.03) - open/close 14255 (time 1787753637.98 total 963.77 last 10.46) - open/close 14377 (time 1787753648.00 total 973.79 last 12.18) - open/close 14471 (time 1787753658.83 total 984.62 last 8.68) - open/close 14566 (time 1787753668.85 total 994.64 last 9.48) - open/close 14662 (time 1787753679.37 total 1005.17 last 9.12) - open/close 14787 (time 1787753689.40 total 1015.19 last 12.47) - open/close 14904 (time 1787753699.41 total 1025.20 last 11.68) - open/close 15009 (time 1787753710.50 total 1036.29 last 9.47) - open/close 15140 (time 1787753720.86 total 1046.65 last 12.65) - open/close 15301 (time 1787753731.07 total 1056.86 last 15.77) - open/close 15371 (time 1787753741.75 total 1067.54 last 6.55) - open/close 15485 (time 1787753751.76 total 1077.55 last 11.40) - open/close 15599 (time 1787753761.98 total 1087.77 last 11.15) - open/close 15753 (time 1787753773.07 total 1098.86 last 13.89) - open/close 15960 (time 1787753783.68 total 1109.47 last 19.51) - open/close 16107 (time 1787753794.62 total 1120.41 last 13.44) - open/close 16249 (time 1787753805.08 total 1130.87 last 13.57) - open/close 16355 (time 1787753815.65 total 1141.44 last 10.03) - open/close 16471 (time 1787753825.67 total 1151.46 last 11.58) - open/close 16583 (time 1787753835.68 total 1161.47 last 11.19) - open/close 16728 (time 1787753845.69 total 1171.48 last 14.48) - open/close 16872 (time 1787753855.71 total 1181.50 last 14.38) - open/close 17006 (time 1787753865.71 total 1191.50 last 13.40) - open/close 17173 (time 1787753876.12 total 1201.91 last 16.05) - open/close 17320 (time 1787753886.13 total 1211.92 last 14.68) - open/close 17477 (time 1787753896.15 total 1221.94 last 15.67) - open/close 17639 (time 1787753906.78 total 1232.57 last 15.24) - open/close 17789 (time 1787753917.02 total 1242.81 last 14.65) - open/close 18178 (time 1787753927.02 total 1252.81 last 38.90) - open/close 18889 (time 1787753937.02 total 1262.81 last 71.07) - open/close 19710 (time 1787753947.03 total 1272.82 last 82.03) - open/close 20000 (time 1787753950.35 total 1276.14 last 87.44) - open/close 20706 (time 1787753960.36 total 1286.15 last 70.49) - open/close 21464 (time 1787753970.36 total 1296.15 last 75.77) - open/close 22356 (time 1787753980.37 total 1306.16 last 89.16) - open/close 23178 (time 1787753990.37 total 1316.16 last 82.19) - open/close 24317 (time 1787754000.38 total 1326.17 last 113.83) - open/close 25231 (time 1787754010.38 total 1336.17 last 91.36) - open/close 26163 (time 1787754020.38 total 1346.17 last 93.20) - open/close 27132 (time 1787754030.38 total 1356.17 last 96.87) - open/close 28056 (time 1787754040.38 total 1366.18 last 92.39) - open/close 29011 (time 1787754050.39 total 1376.18 last 95.47) - open/close 30000 (time 1787754058.91 total 1384.70 last 116.00) - open/close 31005 (time 1787754068.92 total 1394.71 last 100.49) - open/close 31918 (time 1787754078.92 total 1404.71 last 91.23) - open/close 32656 (time 1787754088.93 total 1414.72 last 73.74) - open/close 33722 (time 1787754098.93 total 1424.72 last 106.57) - open/close 34752 (time 1787754108.95 total 1434.74 last 102.88) - open/close 35783 (time 1787754118.95 total 1444.74 last 103.03) - open/close 36568 (time 1787754128.95 total 1454.74 last 78.49) - open/close 37278 (time 1787754138.96 total 1464.75 last 70.95) - open/close 37715 (time 1787754148.98 total 1474.77 last 43.60) - open/close 38202 (time 1787754159.01 total 1484.80 last 48.57) - open/close 38945 (time 1787754169.02 total 1494.81 last 74.24) - open/close 39866 (time 1787754179.02 total 1504.81 last 92.08) - open/close 40659 (time 1787754189.03 total 1514.82 last 79.20) - open/close 41499 (time 1787754199.04 total 1524.83 last 83.95) - open/close 42285 (time 1787754209.05 total 1534.84 last 78.51) - open/close 43051 (time 1787754219.05 total 1544.84 last 76.59) - open/close 43988 (time 1787754229.06 total 1554.85 last 93.65) - open/close 44579 (time 1787754239.07 total 1564.86 last 59.02) - open/close 45313 (time 1787754249.07 total 1574.86 last 73.36) - open/close 45924 (time 1787754259.08 total 1584.87 last 61.04) - open/close 46495 (time 1787754269.09 total 1594.88 last 57.08) - open/close 47323 (time 1787754279.09 total 1604.88 last 82.80) - open/close 48317 (time 1787754289.09 total 1614.88 last 99.34) - open/close 49453 (time 1787754299.10 total 1624.89 last 113.49) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1626.40 seconds: 30.52 ops/second - unlinked 0 (time 1787754304 ; total 0 ; last 0) - unlinked 10000 (time 1787754416 ; total 112 ; last 112) - unlinked 20000 (time 1787754504 ; total 200 ; last 88) - unlinked 30000 (time 1787754569 ; total 265 ; last 65) - unlinked 40000 (time 1787754653 ; total 349 ; last 84) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 411 seconds: 120.756691 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg105-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg105-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg105-server unloading modules via unload_modules_local on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing unload_modules_local oleg105-server: modules unloaded. pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2