== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:16:25 (1786079785) start mds service on oleg406-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing load_modules_local oleg406-server: Loading modules from /home/green/git/lustre-release/lustre oleg406-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg406-server: detected 4 online CPUs by sysfs oleg406-server: MODOPTS_LIBCFS= oleg406-server: Force libcfs to create 2 CPU partitions oleg406-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg406-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg406-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg406-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg406-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 99 (time 1786079920.01 total 10.06 last 9.84) - open/close 217 (time 1786079930.58 total 20.64 last 11.16) - open/close 293 (time 1786079940.70 total 30.75 last 7.51) - open/close 351 (time 1786079950.80 total 40.86 last 5.74) - open/close 502 (time 1786079961.43 total 51.49 last 14.20) - open/close 611 (time 1786079971.78 total 61.84 last 10.53) - open/close 763 (time 1786079981.79 total 71.85 last 15.19) - open/close 945 (time 1786079991.80 total 81.86 last 18.18) - open/close 1147 (time 1786080004.16 total 94.22 last 16.34) - open/close 1231 (time 1786080014.37 total 104.43 last 8.23) - open/close 1351 (time 1786080025.38 total 115.43 last 10.91) - open/close 1457 (time 1786080036.64 total 126.69 last 9.41) - open/close 1554 (time 1786080046.67 total 136.72 last 9.67) - open/close 1722 (time 1786080056.99 total 147.05 last 16.27) - open/close 1858 (time 1786080067.73 total 157.79 last 12.66) - open/close 1948 (time 1786080078.82 total 168.87 last 8.12) - open/close 2052 (time 1786080090.13 total 180.19 last 9.19) - open/close 2138 (time 1786080100.85 total 190.91 last 8.02) - open/close 2259 (time 1786080110.86 total 200.92 last 12.09) - open/close 2464 (time 1786080122.88 total 212.93 last 17.06) - open/close 2656 (time 1786080133.44 total 223.49 last 18.19) - open/close 2812 (time 1786080144.54 total 234.60 last 14.05) - open/close 2981 (time 1786080154.90 total 244.96 last 16.31) - open/close 3063 (time 1786080165.92 total 255.97 last 7.44) - open/close 3161 (time 1786080176.91 total 266.97 last 8.91) - open/close 3269 (time 1786080186.98 total 277.03 last 10.73) - open/close 3406 (time 1786080197.98 total 288.04 last 12.45) - open/close 3548 (time 1786080209.56 total 299.61 last 12.26) - open/close 3688 (time 1786080220.00 total 310.05 last 13.41) - open/close 3839 (time 1786080230.23 total 320.28 last 14.76) - open/close 4018 (time 1786080240.74 total 330.80 last 17.03) - open/close 4208 (time 1786080251.68 total 341.73 last 17.38) - open/close 4312 (time 1786080261.70 total 351.75 last 10.38) - open/close 4375 (time 1786080271.72 total 361.77 last 6.29) - open/close 4565 (time 1786080282.73 total 372.78 last 17.26) - open/close 4714 (time 1786080293.11 total 383.17 last 14.35) - open/close 4820 (time 1786080303.12 total 393.18 last 10.59) - open/close 4890 (time 1786080313.31 total 403.37 last 6.87) - open/close 5022 (time 1786080324.39 total 414.44 last 11.92) - open/close 5121 (time 1786080334.40 total 424.45 last 9.89) - open/close 5227 (time 1786080345.07 total 435.12 last 9.94) - open/close 5307 (time 1786080356.08 total 446.13 last 7.26) - open/close 5429 (time 1786080367.08 total 457.13 last 11.09) - open/close 5518 (time 1786080377.11 total 467.17 last 8.87) - open/close 5623 (time 1786080387.30 total 477.35 last 10.31) - open/close 5730 (time 1786080397.31 total 487.37 last 10.69) - open/close 5827 (time 1786080408.99 total 499.05 last 8.30) - open/close 5933 (time 1786080419.02 total 509.08 last 10.57) - open/close 6027 (time 1786080429.04 total 519.10 last 9.38) - open/close 6138 (time 1786080442.17 total 532.23 last 8.46) - open/close 6246 (time 1786080452.18 total 542.23 last 10.79) - open/close 6300 (time 1786080464.64 total 554.69 last 4.33) - open/close 6468 (time 1786080474.64 total 564.69 last 16.80) - open/close 6626 (time 1786080484.67 total 574.72 last 15.76) - open/close 6736 (time 1786080495.03 total 585.09 last 10.61) - open/close 6904 (time 1786080505.64 total 595.70 last 15.83) - open/close 7037 (time 1786080516.38 total 606.43 last 12.39) - open/close 7181 (time 1786080526.39 total 616.45 last 14.38) - open/close 7284 (time 1786080537.37 total 627.43 last 9.38) - open/close 7427 (time 1786080547.40 total 637.46 last 14.26) - open/close 7624 (time 1786080558.26 total 648.31 last 18.14) - open/close 7764 (time 1786080569.28 total 659.33 last 12.70) - open/close 7951 (time 1786080579.29 total 669.35 last 18.68) - open/close 8122 (time 1786080589.96 total 680.02 last 16.02) - open/close 8252 (time 1786080600.42 total 690.48 last 12.43) - open/close 8321 (time 1786080610.44 total 700.49 last 6.89) - open/close 8483 (time 1786080620.45 total 710.51 last 16.17) - open/close 8606 (time 1786080631.03 total 721.09 last 11.63) - open/close 8689 (time 1786080641.29 total 731.34 last 8.09) - open/close 8814 (time 1786080651.94 total 741.99 last 11.73) - open/close 8958 (time 1786080661.95 total 752.01 last 14.38) - open/close 9097 (time 1786080676.21 total 766.26 last 9.75) - open/close 9208 (time 1786080689.24 total 779.30 last 8.52) - open/close 9310 (time 1786080699.24 total 789.30 last 10.20) - open/close 9478 (time 1786080709.25 total 799.30 last 16.79) - open/close 9599 (time 1786080719.67 total 809.72 last 11.62) - open/close 9700 (time 1786080730.07 total 820.13 last 9.71) - open/close 9820 (time 1786080741.01 total 831.07 last 10.97) - open/close 9907 (time 1786080751.53 total 841.58 last 8.28) - open/close 10000 (time 1786080754.63 total 844.69 last 29.94) - open/close 10122 (time 1786080765.15 total 855.20 last 11.60) - open/close 10192 (time 1786080775.38 total 865.44 last 6.84) - open/close 10289 (time 1786080785.95 total 876.01 last 9.18) - open/close 10408 (time 1786080796.84 total 886.89 last 10.93) - open/close 10549 (time 1786080807.63 total 897.69 last 13.06) - open/close 10679 (time 1786080818.29 total 908.34 last 12.20) - open/close 10780 (time 1786080828.59 total 918.64 last 9.80) - open/close 10918 (time 1786080838.63 total 928.69 last 13.74) - open/close 11073 (time 1786080849.38 total 939.44 last 14.42) - open/close 11152 (time 1786080860.06 total 950.11 last 7.40) - open/close 11267 (time 1786080870.17 total 960.22 last 11.37) - open/close 11356 (time 1786080880.79 total 970.84 last 8.38) - open/close 11443 (time 1786080890.82 total 980.87 last 8.68) - open/close 11556 (time 1786080901.63 total 991.68 last 10.45) - open/close 11665 (time 1786080911.64 total 1001.69 last 10.89) - open/close 11745 (time 1786080922.23 total 1012.28 last 7.55) - open/close 11851 (time 1786080933.09 total 1023.15 last 9.76) - open/close 11978 (time 1786080943.11 total 1033.17 last 12.67) - open/close 12201 (time 1786080953.13 total 1043.19 last 22.26) - open/close 12272 (time 1786080963.80 total 1053.86 last 6.65) - open/close 12398 (time 1786080974.24 total 1064.30 last 12.07) - open/close 12550 (time 1786080985.26 total 1075.31 last 13.80) - open/close 12662 (time 1786080996.22 total 1086.27 last 10.22) - open/close 12858 (time 1786081006.23 total 1096.28 last 19.57) - open/close 12929 (time 1786081016.27 total 1106.33 last 7.07) - open/close 13108 (time 1786081027.11 total 1117.16 last 16.52) - open/close 13271 (time 1786081037.72 total 1127.77 last 15.36) - open/close 13332 (time 1786081047.74 total 1137.79 last 6.09) - open/close 13458 (time 1786081058.25 total 1148.31 last 11.98) - open/close 13621 (time 1786081068.37 total 1158.43 last 16.11) - open/close 13785 (time 1786081079.14 total 1169.20 last 15.23) - open/close 13877 (time 1786081090.02 total 1180.08 last 8.46) - open/close 14026 (time 1786081100.16 total 1190.22 last 14.70) - open/close 14105 (time 1786081110.47 total 1200.52 last 7.66) - open/close 14228 (time 1786081120.68 total 1210.73 last 12.05) - open/close 14425 (time 1786081131.46 total 1221.51 last 18.28) - open/close 14715 (time 1786081141.47 total 1231.52 last 28.97) - open/close 15404 (time 1786081151.47 total 1241.52 last 68.89) - open/close 16161 (time 1786081161.48 total 1251.53 last 75.66) - open/close 16851 (time 1786081171.49 total 1261.54 last 68.90) - open/close 17489 (time 1786081181.50 total 1271.55 last 63.76) - open/close 18208 (time 1786081191.50 total 1281.56 last 71.85) - open/close 18945 (time 1786081201.51 total 1291.56 last 73.67) - open/close 19591 (time 1786081211.51 total 1301.56 last 64.58) - open/close 20000 (time 1786081217.36 total 1307.41 last 69.92) - open/close 20742 (time 1786081227.38 total 1317.43 last 74.06) - open/close 21444 (time 1786081237.39 total 1327.45 last 70.09) - open/close 22275 (time 1786081247.39 total 1337.45 last 83.09) - open/close 23048 (time 1786081257.40 total 1347.45 last 77.26) - open/close 23823 (time 1786081267.40 total 1357.46 last 77.46) - open/close 24584 (time 1786081277.42 total 1367.47 last 75.99) - open/close 25283 (time 1786081287.43 total 1377.49 last 69.82) - open/close 26034 (time 1786081297.44 total 1387.50 last 75.00) - open/close 26753 (time 1786081307.44 total 1397.50 last 71.90) - open/close 27546 (time 1786081317.45 total 1407.51 last 79.23) - open/close 28369 (time 1786081327.46 total 1417.52 last 82.22) - open/close 29162 (time 1786081337.48 total 1427.54 last 79.16) - open/close 29994 (time 1786081347.49 total 1437.55 last 83.10) - open/close 30740 (time 1786081357.51 total 1447.57 last 74.46) - open/close 31362 (time 1786081367.53 total 1457.59 last 62.08) - open/close 32077 (time 1786081377.54 total 1467.59 last 71.46) - open/close 32759 (time 1786081387.54 total 1477.59 last 68.20) - open/close 33483 (time 1786081397.54 total 1487.59 last 72.39) - open/close 34283 (time 1786081407.55 total 1497.60 last 79.91) - open/close 35080 (time 1786081417.55 total 1507.61 last 79.67) - open/close 35884 (time 1786081427.56 total 1517.61 last 80.36) - open/close 36546 (time 1786081437.56 total 1527.62 last 66.17) - open/close 37293 (time 1786081447.57 total 1537.63 last 74.64) - open/close 38041 (time 1786081457.58 total 1547.63 last 74.74) - open/close 38807 (time 1786081467.59 total 1557.64 last 76.52) - open/close 39480 (time 1786081477.60 total 1567.65 last 67.24) - open/close 40000 (time 1786081485.76 total 1575.82 last 63.68) - open/close 40673 (time 1786081495.77 total 1585.82 last 67.26) - open/close 41420 (time 1786081505.79 total 1595.84 last 74.57) - open/close 42111 (time 1786081515.79 total 1605.85 last 69.06) - open/close 42824 (time 1786081525.81 total 1615.87 last 71.17) - open/close 43454 (time 1786081535.82 total 1625.88 last 62.92) - open/close 44108 (time 1786081545.84 total 1635.89 last 65.32) - open/close 44819 (time 1786081555.85 total 1645.91 last 70.98) - open/close 45590 (time 1786081565.86 total 1655.92 last 77.04) - open/close 46302 (time 1786081575.89 total 1665.95 last 70.99) - open/close 46899 (time 1786081585.89 total 1675.95 last 59.69) - open/close 47598 (time 1786081595.90 total 1685.95 last 69.89) - open/close 48393 (time 1786081605.91 total 1695.96 last 79.41) - open/close 49100 (time 1786081615.93 total 1705.98 last 70.56) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1713.01 seconds: 28.97 ops/second - unlinked 0 (time 1786081628 ; total 0 ; last 0) - unlinked 10000 (time 1786081786 ; total 158 ; last 158) - unlinked 20000 (time 1786081942 ; total 314 ; last 156) - unlinked 30000 (time 1786082101 ; total 473 ; last 159) - unlinked 40000 (time 1786082257 ; total 629 ; last 156) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 767 seconds: 64.707954 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg406-server unloading modules via unload_modules_local on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing unload_modules_local oleg406-server: modules unloaded. pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2