== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:12:17 (1786180337) start mds service on oleg236-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg236-server' oleg236-server: oleg236-server.virtnet: executing load_modules_local oleg236-server: Loading modules from /home/green/git/lustre-release/lustre oleg236-server: detected 4 online CPUs by sysfs oleg236-server: MODOPTS_LIBCFS= oleg236-server: Force libcfs to create 2 CPU partitions oleg236-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg236-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg236-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg236-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg236-server: oleg236-server.virtnet: executing set_default_debug -1 all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg236-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug -1 all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0000 oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 245 (time 1786180427.16 total 11.00 last 22.28) - open/close 460 (time 1786180437.18 total 21.01 last 21.46) - open/close 659 (time 1786180448.10 total 31.93 last 18.23) - open/close 872 (time 1786180460.28 total 44.12 last 17.48) - open/close 1079 (time 1786180471.19 total 55.02 last 18.98) - open/close 1263 (time 1786180484.06 total 67.90 last 14.29) - open/close 1415 (time 1786180494.68 total 78.51 last 14.32) - open/close 1590 (time 1786180505.11 total 88.94 last 16.77) - open/close 1777 (time 1786180515.60 total 99.43 last 17.83) - open/close 1924 (time 1786180525.73 total 109.56 last 14.51) - open/close 2144 (time 1786180538.58 total 122.42 last 17.11) - open/close 2348 (time 1786180549.52 total 133.36 last 18.65) - open/close 2565 (time 1786180560.71 total 144.55 last 19.39) - open/close 2843 (time 1786180571.03 total 154.87 last 26.93) - open/close 3088 (time 1786180581.20 total 165.03 last 24.10) - open/close 3403 (time 1786180591.78 total 175.61 last 29.77) - open/close 3597 (time 1786180601.87 total 185.71 last 19.22) - open/close 3707 (time 1786180612.11 total 195.94 last 10.75) - open/close 3797 (time 1786180622.12 total 205.95 last 8.99) - open/close 3912 (time 1786180634.04 total 217.87 last 9.65) - open/close 4061 (time 1786180645.07 total 228.90 last 13.51) - open/close 4309 (time 1786180655.85 total 239.68 last 23.01) - open/close 4541 (time 1786180666.38 total 250.22 last 22.02) - open/close 4765 (time 1786180677.05 total 260.89 last 20.99) - open/close 4918 (time 1786180687.18 total 271.02 last 15.10) - open/close 5081 (time 1786180697.28 total 281.11 last 16.15) - open/close 5331 (time 1786180707.30 total 291.14 last 24.94) - open/close 5368 (time 1786180717.79 total 301.62 last 3.53) - open/close 5516 (time 1786180727.81 total 311.65 last 14.77) - open/close 5635 (time 1786180737.82 total 321.65 last 11.89) - open/close 5759 (time 1786180748.90 total 332.73 last 11.19) - open/close 5969 (time 1786180761.63 total 345.46 last 16.50) - open/close 6194 (time 1786180771.63 total 355.47 last 22.49) - open/close 6421 (time 1786180782.18 total 366.01 last 21.53) - open/close 6682 (time 1786180792.42 total 376.26 last 25.47) - open/close 6883 (time 1786180803.12 total 386.95 last 18.80) - open/close 7086 (time 1786180814.12 total 397.95 last 18.46) - open/close 7262 (time 1786180824.85 total 408.68 last 16.40) - open/close 7451 (time 1786180836.18 total 420.02 last 16.68) - open/close 7671 (time 1786180847.13 total 430.96 last 20.10) - open/close 7863 (time 1786180858.08 total 441.92 last 17.53) - open/close 8098 (time 1786180868.49 total 452.32 last 22.58) - open/close 8322 (time 1786180878.54 total 462.37 last 22.29) - open/close 8532 (time 1786180888.57 total 472.40 last 20.94) - open/close 8713 (time 1786180898.59 total 482.43 last 18.05) - open/close 8924 (time 1786180909.15 total 492.98 last 20.00) - open/close 9003 (time 1786180919.16 total 503.00 last 7.88) - open/close 9018 (time 1786180929.87 total 513.70 last 1.40) - open/close 9180 (time 1786180942.09 total 525.92 last 13.25) - open/close 9389 (time 1786180952.10 total 535.94 last 20.88) - open/close 9627 (time 1786180962.11 total 545.95 last 23.78) - open/close 9793 (time 1786180973.18 total 557.01 last 15.00) - open/close 9862 (time 1786180983.54 total 567.37 last 6.66) - open/close 10000 (time 1786180989.76 total 573.60 last 22.16) - open/close 10576 (time 1786180999.80 total 583.63 last 57.39) - open/close 10691 (time 1786181009.81 total 593.64 last 11.49) - open/close 10797 (time 1786181020.75 total 604.59 last 9.68) - open/close 10994 (time 1786181030.77 total 614.61 last 19.66) - open/close 11133 (time 1786181041.11 total 624.95 last 13.44) - open/close 11319 (time 1786181051.13 total 634.96 last 18.57) - open/close 11573 (time 1786181061.14 total 644.97 last 25.38) - open/close 11800 (time 1786181073.70 total 657.53 last 18.07) - open/close 12022 (time 1786181084.12 total 667.96 last 21.30) - open/close 12191 (time 1786181094.69 total 678.52 last 16.00) - open/close 12412 (time 1786181104.71 total 688.55 last 22.05) - open/close 12617 (time 1786181115.03 total 698.87 last 19.86) - open/close 12760 (time 1786181125.46 total 709.29 last 13.71) - open/close 12979 (time 1786181137.43 total 721.26 last 18.29) - open/close 13129 (time 1786181147.45 total 731.29 last 14.97) - open/close 13261 (time 1786181158.81 total 742.64 last 11.63) - open/close 13435 (time 1786181169.36 total 753.19 last 16.49) - open/close 13627 (time 1786181179.85 total 763.68 last 18.30) - open/close 13793 (time 1786181190.55 total 774.38 last 15.52) - open/close 13995 (time 1786181200.56 total 784.39 last 20.18) - open/close 14183 (time 1786181210.57 total 794.40 last 18.78) - open/close 14414 (time 1786181220.82 total 804.66 last 22.53) - open/close 14611 (time 1786181231.11 total 814.95 last 19.14) - open/close 14836 (time 1786181241.89 total 825.72 last 20.89) - open/close 15103 (time 1786181252.53 total 836.36 last 25.08) - open/close 15333 (time 1786181263.45 total 847.29 last 21.06) - open/close 15566 (time 1786181273.47 total 857.31 last 23.25) - open/close 15823 (time 1786181283.51 total 867.35 last 25.60) - open/close 16002 (time 1786181293.90 total 877.73 last 17.23) - open/close 16162 (time 1786181304.93 total 888.77 last 14.50) - open/close 16300 (time 1786181315.06 total 898.89 last 13.63) - open/close 16452 (time 1786181325.66 total 909.50 last 14.33) - open/close 16696 (time 1786181335.67 total 919.50 last 24.40) - open/close 16943 (time 1786181345.71 total 929.54 last 24.61) - open/close 17243 (time 1786181355.72 total 939.55 last 29.97) - open/close 17515 (time 1786181365.73 total 949.56 last 27.17) - open/close 17743 (time 1786181376.11 total 959.94 last 21.97) - open/close 17987 (time 1786181386.79 total 970.62 last 22.84) - open/close 18227 (time 1786181397.49 total 981.32 last 22.43) - open/close 18971 (time 1786181407.49 total 991.33 last 74.38) - open/close 19653 (time 1786181417.50 total 1001.33 last 68.14) - open/close 20000 (time 1786181421.37 total 1005.21 last 89.62) - open/close 20879 (time 1786181431.37 total 1015.21 last 87.90) - open/close 21747 (time 1786181441.37 total 1025.21 last 86.79) - open/close 22526 (time 1786181451.39 total 1035.22 last 77.76) - open/close 23330 (time 1786181461.39 total 1045.23 last 80.38) - open/close 24026 (time 1786181471.40 total 1055.23 last 69.56) - open/close 24900 (time 1786181481.41 total 1065.24 last 87.34) - open/close 25714 (time 1786181491.41 total 1075.24 last 81.40) - open/close 26700 (time 1786181501.41 total 1085.24 last 98.56) - open/close 27522 (time 1786181511.41 total 1095.24 last 82.19) - open/close 28362 (time 1786181521.41 total 1105.25 last 83.99) - open/close 29135 (time 1786181531.46 total 1115.29 last 76.94) - open/close 29967 (time 1786181541.46 total 1125.30 last 83.15) - open/close 30760 (time 1786181551.48 total 1135.31 last 79.18) - open/close 31452 (time 1786181561.48 total 1145.32 last 69.17) - open/close 32181 (time 1786181571.49 total 1155.32 last 72.85) - open/close 33005 (time 1786181581.50 total 1165.33 last 82.37) - open/close 33821 (time 1786181591.50 total 1175.34 last 81.52) - open/close 34683 (time 1786181601.51 total 1185.34 last 86.17) - open/close 35425 (time 1786181611.51 total 1195.34 last 74.17) - open/close 36131 (time 1786181621.51 total 1205.35 last 70.58) - open/close 36864 (time 1786181631.52 total 1215.35 last 73.29) - open/close 37497 (time 1786181641.52 total 1225.35 last 63.27) - open/close 38280 (time 1786181651.52 total 1235.36 last 78.29) - open/close 39042 (time 1786181661.53 total 1245.36 last 76.14) - open/close 39867 (time 1786181671.54 total 1255.37 last 82.41) - open/close 40760 (time 1786181681.55 total 1265.38 last 89.24) - open/close 41549 (time 1786181691.55 total 1275.39 last 78.84) - open/close 42401 (time 1786181701.56 total 1285.39 last 85.16) - open/close 43229 (time 1786181711.57 total 1295.40 last 82.73) - open/close 44043 (time 1786181721.57 total 1305.40 last 81.38) - open/close 44828 (time 1786181731.59 total 1315.42 last 78.34) - open/close 45650 (time 1786181741.59 total 1325.43 last 82.17) - open/close 46419 (time 1786181751.60 total 1335.43 last 76.87) - open/close 47321 (time 1786181761.60 total 1345.43 last 90.17) - open/close 48150 (time 1786181771.60 total 1355.43 last 82.90) - open/close 48889 (time 1786181781.61 total 1365.44 last 73.83) - open/close 49574 (time 1786181791.62 total 1375.45 last 68.44) total: 50000 open/close in 1380.82 seconds: 36.21 ops/second - unlinked 0 (time 1786181802 ; total 0 ; last 0) - unlinked 10000 (time 1786181941 ; total 139 ; last 139) - unlinked 20000 (time 1786182082 ; total 280 ; last 141) - unlinked 30000 (time 1786182222 ; total 420 ; last 140) - unlinked 40000 (time 1786182357 ; total 555 ; last 135) total: 50000 unlinks in 691 seconds: 72.358902 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg236-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg236-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg236-server stop mds service on oleg236-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg236-server unloading modules via unload_modules_local on: 'oleg236-server' oleg236-server: oleg236-server.virtnet: executing unload_modules_local oleg236-server: modules unloaded. pdsh@oleg236-client: oleg236-client: ssh exited with exit code 2 pdsh@oleg236-client: oleg236-server: ssh exited with exit code 2 pdsh@oleg236-client: oleg236-client: ssh exited with exit code 2 pdsh@oleg236-client: oleg236-server: ssh exited with exit code 2