== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:54:17 (1787860457) start mds service on oleg408-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions oleg408-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg408-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg408-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 109 (time 1787860550.23 total 11.65 last 9.35) - open/close 290 (time 1787860561.36 total 22.79 last 16.26) - open/close 431 (time 1787860572.07 total 33.49 last 13.17) - open/close 519 (time 1787860582.14 total 43.56 last 8.74) - open/close 672 (time 1787860592.93 total 54.36 last 14.17) - open/close 791 (time 1787860603.15 total 64.58 last 11.65) - open/close 848 (time 1787860613.99 total 75.41 last 5.26) - open/close 935 (time 1787860624.40 total 85.83 last 8.35) - open/close 1023 (time 1787860634.78 total 96.20 last 8.48) - open/close 1121 (time 1787860644.79 total 106.22 last 9.78) - open/close 1330 (time 1787860655.00 total 116.43 last 20.48) - open/close 1565 (time 1787860665.43 total 126.86 last 22.52) - open/close 1766 (time 1787860675.85 total 137.28 last 19.29) - open/close 1916 (time 1787860686.24 total 147.67 last 14.44) - open/close 2108 (time 1787860696.52 total 157.95 last 18.67) - open/close 2232 (time 1787860706.81 total 168.23 last 12.06) - open/close 2474 (time 1787860716.82 total 178.24 last 24.18) - open/close 2697 (time 1787860727.79 total 189.22 last 20.32) - open/close 2760 (time 1787860738.78 total 200.21 last 5.73) - open/close 2955 (time 1787860749.27 total 210.70 last 18.58) - open/close 3069 (time 1787860764.13 total 225.56 last 7.67) - open/close 3227 (time 1787860774.15 total 235.57 last 15.78) - open/close 3354 (time 1787860784.68 total 246.11 last 12.05) - open/close 3508 (time 1787860794.70 total 256.12 last 15.38) - open/close 3646 (time 1787860806.76 total 268.19 last 11.44) - open/close 3841 (time 1787860816.76 total 278.19 last 19.49) - open/close 4033 (time 1787860826.77 total 288.19 last 19.20) - open/close 4229 (time 1787860837.30 total 298.72 last 18.61) - open/close 4309 (time 1787860849.95 total 311.37 last 6.32) - open/close 4458 (time 1787860861.79 total 323.22 last 12.58) - open/close 4623 (time 1787860871.81 total 333.23 last 16.47) - open/close 4743 (time 1787860881.85 total 343.28 last 11.95) - open/close 4875 (time 1787860892.45 total 353.88 last 12.45) - open/close 5020 (time 1787860903.52 total 364.95 last 13.10) - open/close 5187 (time 1787860914.20 total 375.62 last 15.64) - open/close 5315 (time 1787860924.44 total 385.87 last 12.49) - open/close 5420 (time 1787860934.45 total 395.88 last 10.49) - open/close 5541 (time 1787860944.59 total 406.02 last 11.93) - open/close 5626 (time 1787860955.48 total 416.91 last 7.81) - open/close 5755 (time 1787860967.51 total 428.94 last 10.72) - open/close 6007 (time 1787860978.12 total 439.55 last 23.75) - open/close 6230 (time 1787860988.13 total 449.55 last 22.29) - open/close 6379 (time 1787860998.75 total 460.18 last 14.02) - open/close 6578 (time 1787861008.77 total 470.20 last 19.86) - open/close 6764 (time 1787861018.90 total 480.33 last 18.37) - open/close 6940 (time 1787861028.91 total 490.33 last 17.59) - open/close 7107 (time 1787861038.91 total 500.34 last 16.68) - open/close 7241 (time 1787861048.94 total 510.37 last 13.36) - open/close 7404 (time 1787861059.21 total 520.64 last 15.87) - open/close 7588 (time 1787861070.23 total 531.65 last 16.71) - open/close 7756 (time 1787861080.26 total 541.69 last 16.74) - open/close 7862 (time 1787861090.27 total 551.70 last 10.59) - open/close 8042 (time 1787861100.36 total 561.79 last 17.84) - open/close 8201 (time 1787861110.37 total 571.80 last 15.89) - open/close 8320 (time 1787861121.51 total 582.93 last 10.69) - open/close 8483 (time 1787861131.51 total 592.93 last 16.30) - open/close 8591 (time 1787861143.01 total 604.44 last 9.39) - open/close 8805 (time 1787861153.02 total 614.44 last 21.39) - open/close 9025 (time 1787861163.02 total 624.45 last 21.99) - open/close 9291 (time 1787861173.03 total 634.45 last 26.59) - open/close 9505 (time 1787861183.03 total 644.45 last 21.40) - open/close 9685 (time 1787861197.88 total 659.30 last 12.12) - open/close 9840 (time 1787861208.28 total 669.71 last 14.90) - open/close 10000 (time 1787861216.75 total 678.18 last 18.90) - open/close 10019 (time 1787861227.55 total 688.98 last 1.76) - open/close 10065 (time 1787861238.00 total 699.43 last 4.40) - open/close 10252 (time 1787861248.01 total 709.43 last 18.70) - open/close 10471 (time 1787861258.28 total 719.71 last 21.31) - open/close 10720 (time 1787861268.30 total 729.73 last 24.86) - open/close 10859 (time 1787861279.25 total 740.68 last 12.70) - open/close 10987 (time 1787861289.30 total 750.73 last 12.73) - open/close 11074 (time 1787861299.55 total 760.97 last 8.49) - open/close 11200 (time 1787861309.56 total 770.99 last 12.58) - open/close 11386 (time 1787861319.57 total 781.00 last 18.58) - open/close 11622 (time 1787861329.58 total 791.00 last 23.59) - open/close 11816 (time 1787861340.51 total 801.94 last 17.75) - open/close 11943 (time 1787861350.74 total 812.16 last 12.42) - open/close 11974 (time 1787861361.12 total 822.55 last 2.98) - open/close 12096 (time 1787861371.13 total 832.56 last 12.19) - open/close 12211 (time 1787861382.03 total 843.46 last 10.56) - open/close 12388 (time 1787861392.04 total 853.47 last 17.68) - open/close 12555 (time 1787861402.05 total 863.48 last 16.69) - open/close 12725 (time 1787861413.01 total 874.43 last 15.52) - open/close 12925 (time 1787861423.50 total 884.93 last 19.05) - open/close 13065 (time 1787861433.69 total 895.12 last 13.74) - open/close 13239 (time 1787861444.20 total 905.63 last 16.55) - open/close 13338 (time 1787861454.25 total 915.68 last 9.85) - open/close 13597 (time 1787861465.01 total 926.44 last 24.07) - open/close 13781 (time 1787861475.03 total 936.46 last 18.36) - open/close 13977 (time 1787861487.45 total 948.88 last 15.78) - open/close 14091 (time 1787861499.28 total 960.71 last 9.64) - open/close 14356 (time 1787861511.80 total 973.23 last 21.17) - open/close 14581 (time 1787861522.39 total 983.82 last 21.25) - open/close 14670 (time 1787861532.65 total 994.08 last 8.68) - open/close 14805 (time 1787861543.35 total 1004.78 last 12.62) - open/close 14910 (time 1787861553.43 total 1014.86 last 10.41) - open/close 15045 (time 1787861563.44 total 1024.87 last 13.49) - open/close 15111 (time 1787861573.45 total 1034.87 last 6.60) - open/close 15278 (time 1787861583.46 total 1044.89 last 16.68) - open/close 15423 (time 1787861593.76 total 1055.19 last 14.07) - open/close 15652 (time 1787861603.92 total 1065.35 last 22.54) - open/close 15835 (time 1787861614.29 total 1075.72 last 17.65) - open/close 16191 (time 1787861624.31 total 1085.73 last 35.54) - open/close 16771 (time 1787861634.32 total 1095.75 last 57.91) - open/close 17341 (time 1787861644.33 total 1105.75 last 56.98) - open/close 17896 (time 1787861654.35 total 1115.78 last 55.37) - open/close 18450 (time 1787861664.38 total 1125.81 last 55.23) - open/close 18974 (time 1787861674.39 total 1135.82 last 52.36) - open/close 19535 (time 1787861684.40 total 1145.83 last 56.04) - open/close 20000 (time 1787861692.34 total 1153.76 last 58.59) - open/close 20562 (time 1787861702.34 total 1163.77 last 56.15) - open/close 21191 (time 1787861712.36 total 1173.78 last 62.83) - open/close 21808 (time 1787861722.36 total 1183.79 last 61.68) - open/close 22480 (time 1787861732.37 total 1193.80 last 67.12) - open/close 23017 (time 1787861742.38 total 1203.80 last 53.67) - open/close 23601 (time 1787861752.38 total 1213.81 last 58.38) - open/close 24101 (time 1787861762.39 total 1223.82 last 49.93) - open/close 24684 (time 1787861772.42 total 1233.85 last 58.13) - open/close 25295 (time 1787861782.43 total 1243.86 last 61.04) - open/close 25839 (time 1787861792.44 total 1253.87 last 54.35) - open/close 26477 (time 1787861802.45 total 1263.87 last 63.78) - open/close 27065 (time 1787861812.45 total 1273.88 last 58.77) - open/close 27669 (time 1787861822.45 total 1283.88 last 60.40) - open/close 28236 (time 1787861832.47 total 1293.90 last 56.59) - open/close 28822 (time 1787861842.48 total 1303.91 last 58.52) - open/close 29413 (time 1787861852.49 total 1313.91 last 59.08) - open/close 29908 (time 1787861862.51 total 1323.94 last 49.39) - open/close 30375 (time 1787861872.53 total 1333.95 last 46.62) - open/close 30940 (time 1787861882.57 total 1344.00 last 56.25) - open/close 31548 (time 1787861892.59 total 1354.01 last 60.69) - open/close 32075 (time 1787861902.60 total 1364.02 last 52.66) - open/close 32660 (time 1787861912.61 total 1374.04 last 58.42) - open/close 33298 (time 1787861922.62 total 1384.05 last 63.72) - open/close 33897 (time 1787861932.63 total 1394.05 last 59.87) - open/close 34507 (time 1787861942.63 total 1404.06 last 60.98) - open/close 35051 (time 1787861952.66 total 1414.09 last 54.22) - open/close 35475 (time 1787861962.68 total 1424.11 last 42.32) - open/close 35896 (time 1787861972.69 total 1434.11 last 42.08) - open/close 36347 (time 1787861982.69 total 1444.12 last 45.09) - open/close 36924 (time 1787861992.70 total 1454.13 last 57.64) - open/close 37479 (time 1787862002.71 total 1464.13 last 55.47) - open/close 38080 (time 1787862012.72 total 1474.14 last 60.02) - open/close 38674 (time 1787862022.73 total 1484.16 last 59.31) - open/close 39205 (time 1787862032.76 total 1494.19 last 52.96) - open/close 39626 (time 1787862042.78 total 1504.20 last 42.03) - open/close 40000 (time 1787862050.30 total 1511.73 last 49.70) - open/close 40426 (time 1787862060.34 total 1521.77 last 42.44) - open/close 40963 (time 1787862070.34 total 1531.77 last 53.70) - open/close 41437 (time 1787862080.35 total 1541.78 last 47.36) - open/close 41923 (time 1787862090.36 total 1551.78 last 48.57) - open/close 42286 (time 1787862100.37 total 1561.79 last 36.26) - open/close 42714 (time 1787862110.39 total 1571.82 last 42.70) - open/close 43194 (time 1787862120.40 total 1581.82 last 47.96) - open/close 43737 (time 1787862130.41 total 1591.83 last 54.25) - open/close 44337 (time 1787862140.42 total 1601.84 last 59.95) - open/close 44825 (time 1787862150.43 total 1611.86 last 48.71) - open/close 45324 (time 1787862160.44 total 1621.87 last 49.86) - open/close 45771 (time 1787862170.47 total 1631.89 last 44.59) - open/close 46247 (time 1787862180.47 total 1641.90 last 47.58) - open/close 46758 (time 1787862190.49 total 1651.91 last 51.03) - open/close 47355 (time 1787862200.50 total 1661.93 last 59.62) - open/close 47937 (time 1787862210.52 total 1671.94 last 58.10) - open/close 48597 (time 1787862220.54 total 1681.96 last 65.86) - open/close 49163 (time 1787862230.55 total 1691.97 last 56.55) - open/close 49739 (time 1787862240.57 total 1702.00 last 57.47) total: 50000 open/close in 1706.83 seconds: 29.29 ops/second - unlinked 0 (time 1787862250 ; total 0 ; last 0) - unlinked 10000 (time 1787862442 ; total 192 ; last 192) - unlinked 20000 (time 1787862618 ; total 368 ; last 176) - unlinked 30000 (time 1787862794 ; total 544 ; last 176) - unlinked 40000 (time 1787862967 ; total 717 ; last 173) total: 50000 unlinks in 895 seconds: 55.865921 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: modules unloaded. pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2