== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:49:33 (1785080973) start mds service on oleg423-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing load_modules_local oleg423-server: Loading modules from /home/green/git/lustre-release/lustre oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg423-server: detected 4 online CPUs by sysfs oleg423-server: MODOPTS_LIBCFS= oleg423-server: Force libcfs to create 2 CPU partitions oleg423-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg423-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg423-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg423-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 239 (time 1785081061.18 total 10.92 last 21.88) - open/close 435 (time 1785081071.20 total 20.94 last 19.57) - open/close 675 (time 1785081081.20 total 30.94 last 23.99) - open/close 910 (time 1785081091.21 total 40.95 last 23.49) - open/close 1181 (time 1785081101.86 total 51.60 last 25.44) - open/close 1449 (time 1785081111.88 total 61.62 last 26.74) - open/close 1706 (time 1785081122.46 total 72.20 last 24.29) - open/close 1956 (time 1785081133.17 total 82.91 last 23.34) - open/close 2219 (time 1785081143.19 total 92.93 last 26.26) - open/close 2508 (time 1785081153.20 total 102.94 last 28.87) - open/close 2673 (time 1785081163.39 total 113.12 last 16.19) - open/close 2808 (time 1785081174.45 total 124.19 last 12.20) - open/close 3004 (time 1785081185.29 total 135.03 last 18.08) - open/close 3266 (time 1785081196.28 total 146.02 last 23.84) - open/close 3507 (time 1785081206.76 total 156.50 last 22.99) - open/close 3714 (time 1785081217.77 total 167.51 last 18.80) - open/close 4010 (time 1785081227.78 total 177.52 last 29.58) - open/close 4243 (time 1785081237.78 total 187.52 last 23.28) - open/close 4463 (time 1785081247.79 total 197.53 last 21.98) - open/close 4749 (time 1785081257.81 total 207.55 last 28.55) - open/close 4990 (time 1785081267.81 total 217.55 last 24.09) - open/close 5281 (time 1785081278.69 total 228.43 last 26.76) - open/close 5528 (time 1785081290.92 total 240.66 last 20.19) - open/close 5797 (time 1785081301.62 total 251.36 last 25.13) - open/close 6064 (time 1785081311.63 total 261.37 last 26.68) - open/close 6294 (time 1785081324.40 total 274.14 last 18.01) - open/close 6432 (time 1785081335.27 total 285.01 last 12.70) - open/close 6679 (time 1785081346.02 total 295.76 last 22.98) - open/close 6810 (time 1785081356.02 total 305.76 last 13.09) - open/close 6993 (time 1785081366.46 total 316.20 last 17.54) - open/close 7241 (time 1785081376.47 total 326.21 last 24.77) - open/close 7459 (time 1785081386.86 total 336.60 last 21.00) - open/close 7765 (time 1785081396.87 total 346.61 last 30.57) - open/close 8025 (time 1785081407.68 total 357.42 last 24.05) - open/close 8259 (time 1785081417.70 total 367.44 last 23.35) - open/close 8416 (time 1785081427.72 total 377.46 last 15.67) - open/close 8628 (time 1785081437.73 total 387.47 last 21.17) - open/close 8873 (time 1785081447.75 total 397.49 last 24.47) - open/close 9081 (time 1785081457.76 total 407.50 last 20.77) - open/close 9318 (time 1785081468.28 total 418.02 last 22.54) - open/close 9556 (time 1785081478.45 total 428.19 last 23.38) - open/close 9817 (time 1785081488.46 total 438.20 last 26.09) - open/close 10000 (time 1785081497.91 total 447.65 last 19.37) - open/close 10239 (time 1785081507.94 total 457.68 last 23.81) - open/close 10444 (time 1785081518.45 total 468.19 last 19.52) - open/close 10712 (time 1785081528.46 total 478.20 last 26.77) - open/close 10985 (time 1785081538.47 total 488.21 last 27.27) - open/close 11230 (time 1785081549.03 total 498.77 last 23.19) - open/close 11464 (time 1785081560.69 total 510.43 last 20.07) - open/close 11582 (time 1785081571.05 total 520.79 last 11.39) - open/close 11882 (time 1785081581.05 total 530.79 last 29.99) - open/close 12152 (time 1785081591.06 total 540.80 last 26.98) - open/close 12385 (time 1785081601.95 total 551.69 last 21.39) - open/close 12656 (time 1785081612.71 total 562.45 last 25.18) - open/close 12861 (time 1785081623.14 total 572.88 last 19.66) - open/close 13116 (time 1785081633.81 total 583.55 last 23.90) - open/close 13389 (time 1785081644.57 total 594.31 last 25.36) - open/close 13591 (time 1785081654.60 total 604.34 last 20.15) - open/close 13796 (time 1785081664.75 total 614.49 last 20.18) - open/close 13935 (time 1785081674.92 total 624.66 last 13.67) - open/close 14018 (time 1785081687.66 total 637.40 last 6.52) - open/close 14225 (time 1785081698.61 total 648.35 last 18.90) - open/close 14484 (time 1785081708.62 total 658.36 last 25.88) - open/close 14746 (time 1785081718.62 total 668.36 last 26.20) - open/close 14961 (time 1785081728.63 total 678.37 last 21.48) - open/close 15225 (time 1785081739.39 total 689.13 last 24.54) - open/close 15492 (time 1785081749.39 total 699.13 last 26.68) - open/close 15667 (time 1785081759.41 total 709.15 last 17.47) - open/close 15895 (time 1785081769.57 total 719.31 last 22.44) - open/close 16151 (time 1785081779.89 total 729.63 last 24.80) - open/close 16407 (time 1785081790.12 total 739.86 last 25.03) - open/close 16675 (time 1785081804.92 total 754.66 last 18.10) - open/close 16829 (time 1785081815.01 total 764.75 last 15.26) - open/close 17075 (time 1785081825.40 total 775.14 last 23.69) - open/close 17347 (time 1785081835.41 total 785.15 last 27.16) - open/close 17465 (time 1785081846.00 total 795.74 last 11.14) - open/close 17643 (time 1785081857.00 total 806.74 last 16.19) - open/close 17828 (time 1785081867.30 total 817.04 last 17.96) - open/close 18088 (time 1785081877.31 total 827.05 last 25.98) - open/close 18362 (time 1785081887.33 total 837.07 last 27.34) - open/close 18612 (time 1785081897.77 total 847.51 last 23.94) - open/close 18865 (time 1785081907.79 total 857.53 last 25.26) - open/close 18999 (time 1785081918.64 total 868.37 last 12.36) - open/close 19234 (time 1785081929.19 total 878.93 last 22.26) - open/close 19492 (time 1785081939.20 total 888.94 last 25.78) - open/close 19722 (time 1785081949.21 total 898.95 last 22.97) - open/close 20000 (time 1785081960.01 total 909.75 last 25.74) - open/close 20131 (time 1785081970.02 total 919.76 last 13.09) - open/close 20311 (time 1785081980.54 total 930.28 last 17.11) - open/close 20620 (time 1785081991.54 total 941.28 last 28.11) - open/close 21355 (time 1785082001.55 total 951.29 last 73.41) - open/close 22192 (time 1785082011.55 total 961.29 last 83.68) - open/close 22986 (time 1785082021.57 total 971.31 last 79.28) - open/close 23711 (time 1785082031.57 total 981.31 last 72.49) - open/close 24481 (time 1785082041.58 total 991.32 last 76.92) - open/close 25291 (time 1785082051.60 total 1001.34 last 80.85) - open/close 25909 (time 1785082061.61 total 1011.35 last 61.71) - open/close 26674 (time 1785082071.62 total 1021.36 last 76.46) - open/close 27410 (time 1785082081.62 total 1031.36 last 73.56) - open/close 28286 (time 1785082091.63 total 1041.37 last 87.51) - open/close 29099 (time 1785082101.64 total 1051.38 last 81.21) - open/close 29886 (time 1785082111.67 total 1061.41 last 78.50) - open/close 30720 (time 1785082121.68 total 1071.42 last 83.31) - open/close 31520 (time 1785082131.70 total 1081.44 last 79.81) - open/close 32284 (time 1785082141.71 total 1091.45 last 76.32) - open/close 33105 (time 1785082151.72 total 1101.46 last 82.06) - open/close 33937 (time 1785082161.73 total 1111.47 last 83.13) - open/close 34744 (time 1785082171.74 total 1121.48 last 80.61) - open/close 35532 (time 1785082181.74 total 1131.48 last 78.80) - open/close 36275 (time 1785082191.74 total 1141.48 last 74.27) - open/close 37047 (time 1785082201.76 total 1151.50 last 77.02) - open/close 37763 (time 1785082211.78 total 1161.52 last 71.51) - open/close 38585 (time 1785082221.78 total 1171.52 last 82.15) - open/close 39396 (time 1785082231.79 total 1181.52 last 81.09) - open/close 40000 (time 1785082239.26 total 1189.00 last 80.83) - open/close 40780 (time 1785082249.28 total 1199.02 last 77.85) - open/close 41540 (time 1785082259.28 total 1209.02 last 75.95) - open/close 42383 (time 1785082269.29 total 1219.03 last 84.26) - open/close 43205 (time 1785082279.30 total 1229.04 last 82.15) - open/close 43979 (time 1785082289.30 total 1239.04 last 77.34) - open/close 44770 (time 1785082299.31 total 1249.05 last 79.04) - open/close 45550 (time 1785082309.32 total 1259.06 last 77.94) - open/close 46288 (time 1785082319.33 total 1269.07 last 73.70) - open/close 47067 (time 1785082329.34 total 1279.07 last 77.88) - open/close 47848 (time 1785082339.35 total 1289.09 last 77.95) - open/close 48591 (time 1785082349.36 total 1299.10 last 74.28) - open/close 49370 (time 1785082359.36 total 1309.10 last 77.87) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49729) error: No space left on device total: 49729 open/close in 1313.52 seconds: 37.86 ops/second - unlinked 0 (time 1785082367 ; total 0 ; last 0) - unlinked 10000 (time 1785082499 ; total 132 ; last 132) - unlinked 20000 (time 1785082633 ; total 266 ; last 134) - unlinked 30000 (time 1785082760 ; total 393 ; last 127) - unlinked 40000 (time 1785082902 ; total 535 ; last 142) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49729) error: No such file or directory total: 49729 unlinks in 657 seconds: 75.691017 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg423-server unloading modules via unload_modules_local on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing unload_modules_local oleg423-server: modules unloaded. pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2