== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:30:16 (1785774616) start mds service on oleg630-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing load_modules_local oleg630-server: Loading modules from /home/green/git/lustre-release/lustre oleg630-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg630-server: detected 4 online CPUs by sysfs oleg630-server: MODOPTS_LIBCFS= oleg630-server: Force libcfs to create 2 CPU partitions oleg630-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg630-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg630-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg630-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-OST0000 oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 213 (time 1785774690.05 total 10.00 last 21.30) - open/close 442 (time 1785774700.05 total 20.00 last 22.89) - open/close 527 (time 1785774710.06 total 30.01 last 8.49) - open/close 632 (time 1785774720.70 total 40.65 last 9.87) - open/close 820 (time 1785774731.02 total 50.97 last 18.22) - open/close 1067 (time 1785774741.45 total 61.41 last 23.66) - open/close 1394 (time 1785774751.45 total 71.41 last 32.70) - open/close 1606 (time 1785774761.52 total 81.47 last 21.06) - open/close 1852 (time 1785774772.17 total 92.12 last 23.11) - open/close 2123 (time 1785774782.17 total 102.12 last 27.09) - open/close 2351 (time 1785774792.95 total 112.90 last 21.15) - open/close 2578 (time 1785774803.05 total 123.01 last 22.47) - open/close 2803 (time 1785774813.05 total 133.01 last 22.50) - open/close 3053 (time 1785774823.49 total 143.45 last 23.95) - open/close 3271 (time 1785774834.06 total 154.01 last 20.63) - open/close 3525 (time 1785774844.79 total 164.74 last 23.67) - open/close 3735 (time 1785774855.54 total 175.50 last 19.53) - open/close 3950 (time 1785774866.55 total 186.50 last 19.53) - open/close 4125 (time 1785774876.56 total 196.51 last 17.48) - open/close 4327 (time 1785774887.49 total 207.45 last 18.48) - open/close 4531 (time 1785774897.85 total 217.80 last 19.70) - open/close 4765 (time 1785774908.78 total 228.73 last 21.40) - open/close 5035 (time 1785774918.79 total 238.75 last 26.96) - open/close 5225 (time 1785774928.81 total 248.77 last 18.97) - open/close 5400 (time 1785774939.83 total 259.79 last 15.88) - open/close 5621 (time 1785774950.60 total 270.55 last 20.52) - open/close 5876 (time 1785774961.59 total 281.55 last 23.20) - open/close 6103 (time 1785774971.67 total 291.62 last 22.53) - open/close 6358 (time 1785774982.52 total 302.47 last 23.50) - open/close 6645 (time 1785774993.40 total 313.35 last 26.39) - open/close 6910 (time 1785775003.96 total 323.92 last 25.07) - open/close 7140 (time 1785775014.22 total 334.17 last 22.43) - open/close 7363 (time 1785775024.23 total 344.18 last 22.28) - open/close 7588 (time 1785775035.13 total 355.08 last 20.64) - open/close 7841 (time 1785775045.24 total 365.20 last 25.02) - open/close 8006 (time 1785775055.26 total 375.22 last 16.46) - open/close 8214 (time 1785775065.85 total 385.80 last 19.65) - open/close 8453 (time 1785775075.85 total 395.80 last 23.90) - open/close 8601 (time 1785775086.92 total 406.88 last 13.37) - open/close 8765 (time 1785775096.92 total 416.88 last 16.40) - open/close 8977 (time 1785775106.93 total 426.88 last 21.19) - open/close 9179 (time 1785775119.43 total 439.39 last 16.15) - open/close 9378 (time 1785775130.30 total 450.25 last 18.32) - open/close 9605 (time 1785775141.11 total 461.06 last 20.99) - open/close 9871 (time 1785775151.75 total 471.71 last 25.00) - open/close 10000 (time 1785775157.24 total 477.19 last 23.52) - open/close 10283 (time 1785775168.00 total 487.95 last 26.30) - open/close 10488 (time 1785775178.56 total 498.52 last 19.40) - open/close 10762 (time 1785775188.58 total 508.54 last 27.35) - open/close 10952 (time 1785775198.61 total 518.56 last 18.95) - open/close 11105 (time 1785775208.70 total 528.66 last 15.16) - open/close 11333 (time 1785775218.71 total 538.66 last 22.79) - open/close 11561 (time 1785775228.72 total 548.67 last 22.78) - open/close 11757 (time 1785775238.74 total 558.69 last 19.56) - open/close 11901 (time 1785775249.40 total 569.36 last 13.50) - open/close 12143 (time 1785775260.18 total 580.13 last 22.45) - open/close 12382 (time 1785775270.18 total 590.13 last 23.90) - open/close 12608 (time 1785775280.31 total 600.26 last 22.32) - open/close 12835 (time 1785775290.99 total 610.95 last 21.24) - open/close 13027 (time 1785775301.02 total 620.97 last 19.16) - open/close 13203 (time 1785775311.03 total 630.99 last 17.58) - open/close 13463 (time 1785775321.91 total 641.87 last 23.89) - open/close 13596 (time 1785775332.04 total 651.99 last 13.14) - open/close 13804 (time 1785775342.05 total 662.01 last 20.77) - open/close 13931 (time 1785775352.10 total 672.05 last 12.65) - open/close 14131 (time 1785775362.36 total 682.31 last 19.49) - open/close 14275 (time 1785775372.40 total 692.36 last 14.34) - open/close 14471 (time 1785775382.98 total 702.94 last 18.53) - open/close 14685 (time 1785775393.01 total 712.96 last 21.34) - open/close 14921 (time 1785775403.01 total 722.96 last 23.60) - open/close 15137 (time 1785775413.01 total 732.97 last 21.59) - open/close 15387 (time 1785775423.41 total 743.37 last 24.04) - open/close 15615 (time 1785775433.42 total 753.38 last 22.78) - open/close 15834 (time 1785775443.43 total 763.38 last 21.89) - open/close 16054 (time 1785775453.43 total 773.39 last 21.99) - open/close 16319 (time 1785775464.38 total 784.33 last 24.21) - open/close 16554 (time 1785775474.43 total 794.39 last 23.37) - open/close 16778 (time 1785775484.44 total 804.40 last 22.38) - open/close 16988 (time 1785775495.16 total 815.11 last 19.61) - open/close 17202 (time 1785775505.16 total 825.11 last 21.40) - open/close 17381 (time 1785775515.19 total 835.14 last 17.85) - open/close 17617 (time 1785775525.63 total 845.58 last 22.60) - open/close 17921 (time 1785775536.68 total 856.64 last 27.50) - open/close 18157 (time 1785775546.69 total 866.65 last 23.57) - open/close 18342 (time 1785775557.24 total 877.20 last 17.53) - open/close 18569 (time 1785775567.25 total 887.20 last 22.69) - open/close 18794 (time 1785775578.29 total 898.24 last 20.38) - open/close 19042 (time 1785775588.29 total 908.25 last 24.79) - open/close 19577 (time 1785775598.31 total 918.27 last 53.40) - open/close 20000 (time 1785775605.07 total 925.03 last 62.58) - open/close 20668 (time 1785775615.07 total 935.03 last 66.80) - open/close 21339 (time 1785775625.09 total 945.04 last 67.02) - open/close 21936 (time 1785775635.09 total 955.04 last 59.69) - open/close 22629 (time 1785775645.10 total 965.05 last 69.21) - open/close 23200 (time 1785775655.11 total 975.06 last 57.06) - open/close 23920 (time 1785775665.11 total 985.07 last 71.95) - open/close 24556 (time 1785775675.12 total 995.07 last 63.59) - open/close 25198 (time 1785775685.12 total 1005.08 last 64.16) - open/close 25801 (time 1785775695.14 total 1015.09 last 60.22) - open/close 26513 (time 1785775705.14 total 1025.09 last 71.16) - open/close 27214 (time 1785775715.14 total 1035.10 last 70.07) - open/close 27874 (time 1785775725.16 total 1045.11 last 65.93) - open/close 28589 (time 1785775735.20 total 1055.15 last 71.22) - open/close 29225 (time 1785775745.20 total 1065.15 last 63.57) - open/close 29901 (time 1785775755.21 total 1075.16 last 67.55) - open/close 30594 (time 1785775765.23 total 1085.18 last 69.15) - open/close 31205 (time 1785775775.24 total 1095.19 last 61.06) - open/close 31879 (time 1785775785.25 total 1105.21 last 67.28) - open/close 32583 (time 1785775795.27 total 1115.23 last 70.25) - open/close 33195 (time 1785775805.28 total 1125.23 last 61.17) - open/close 33926 (time 1785775815.29 total 1135.24 last 73.04) - open/close 34684 (time 1785775825.29 total 1145.24 last 75.79) - open/close 35430 (time 1785775835.30 total 1155.25 last 74.54) - open/close 36188 (time 1785775845.30 total 1165.25 last 75.78) - open/close 36891 (time 1785775855.31 total 1175.27 last 70.22) - open/close 37551 (time 1785775865.32 total 1185.27 last 65.95) - open/close 38246 (time 1785775875.32 total 1195.28 last 69.47) - open/close 38941 (time 1785775885.33 total 1205.28 last 69.44) - open/close 39649 (time 1785775895.34 total 1215.29 last 70.74) - open/close 40000 (time 1785775899.92 total 1219.87 last 76.68) - open/close 40701 (time 1785775909.93 total 1229.88 last 70.02) - open/close 41455 (time 1785775919.95 total 1239.90 last 75.27) - open/close 42150 (time 1785775929.95 total 1249.91 last 69.46) - open/close 42782 (time 1785775939.97 total 1259.92 last 63.11) - open/close 43526 (time 1785775949.97 total 1269.92 last 74.38) - open/close 44236 (time 1785775959.98 total 1279.93 last 70.92) - open/close 44939 (time 1785775969.99 total 1289.94 last 70.23) - open/close 45620 (time 1785775980.02 total 1299.97 last 67.90) - open/close 46315 (time 1785775990.03 total 1309.98 last 69.46) - open/close 47051 (time 1785776000.04 total 1320.00 last 73.48) - open/close 47758 (time 1785776010.05 total 1330.00 last 70.65) - open/close 48542 (time 1785776020.05 total 1340.00 last 78.40) - open/close 49221 (time 1785776030.06 total 1350.01 last 67.84) - open/close 49918 (time 1785776040.06 total 1360.01 last 69.69) total: 50000 open/close in 1361.35 seconds: 36.73 ops/second - unlinked 0 (time 1785776045 ; total 0 ; last 0) - unlinked 10000 (time 1785776190 ; total 145 ; last 145) - unlinked 20000 (time 1785776347 ; total 302 ; last 157) - unlinked 30000 (time 1785776509 ; total 464 ; last 162) - unlinked 40000 (time 1785776667 ; total 622 ; last 158) total: 50000 unlinks in 771 seconds: 64.850845 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg630-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg630-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg630-server stop mds service on oleg630-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg630-server unloading modules via unload_modules_local on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing unload_modules_local oleg630-server: modules unloaded. pdsh@oleg630-client: oleg630-client: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-server: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-server: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-client: ssh exited with exit code 2