== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:46:47 (1786816007) start mds service on oleg646-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg646-server' oleg646-server: oleg646-server.virtnet: executing load_modules_local oleg646-server: Loading modules from /home/green/git/lustre-release/lustre oleg646-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg646-server: detected 4 online CPUs by sysfs oleg646-server: MODOPTS_LIBCFS= oleg646-server: Force libcfs to create 2 CPU partitions oleg646-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg646-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg646-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg646-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg646-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg646-server: oleg646-server.virtnet: executing set_default_debug -1 all pdsh@oleg646-client: oleg646-server: ssh exited with exit code 1 Started lustre-OST0000 oleg646-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg646-client.virtnet: -o user_xattr,flock 192.168.206.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 224 (time 1786816086.98 total 11.05 last 20.28) - open/close 415 (time 1786816097.00 total 21.06 last 19.08) - open/close 550 (time 1786816108.37 total 32.43 last 11.87) - open/close 779 (time 1786816118.38 total 42.44 last 22.88) - open/close 987 (time 1786816128.39 total 52.45 last 20.77) - open/close 1198 (time 1786816139.41 total 63.47 last 19.15) - open/close 1465 (time 1786816150.01 total 74.07 last 25.19) - open/close 1701 (time 1786816160.25 total 84.32 last 23.03) - open/close 1963 (time 1786816170.26 total 94.32 last 26.20) - open/close 1989 (time 1786816180.27 total 104.33 last 2.60) - open/close 2071 (time 1786816191.12 total 115.18 last 7.56) - open/close 2327 (time 1786816202.18 total 126.24 last 23.13) - open/close 2589 (time 1786816212.20 total 136.26 last 26.16) - open/close 2798 (time 1786816222.21 total 146.27 last 20.89) - open/close 3077 (time 1786816232.21 total 156.27 last 27.88) - open/close 3310 (time 1786816242.43 total 166.49 last 22.80) - open/close 3545 (time 1786816252.77 total 176.83 last 22.73) - open/close 3734 (time 1786816263.50 total 187.56 last 17.62) - open/close 4006 (time 1786816273.51 total 197.57 last 27.17) - open/close 4237 (time 1786816284.29 total 208.35 last 21.42) - open/close 4525 (time 1786816294.30 total 218.36 last 28.78) - open/close 4760 (time 1786816304.31 total 228.38 last 23.47) - open/close 5031 (time 1786816314.33 total 238.39 last 27.07) - open/close 5280 (time 1786816324.35 total 248.41 last 24.85) - open/close 5449 (time 1786816334.38 total 258.44 last 16.84) - open/close 5599 (time 1786816344.98 total 269.04 last 14.15) - open/close 5895 (time 1786816355.70 total 279.76 last 27.62) - open/close 6141 (time 1786816365.70 total 289.76 last 24.59) - open/close 6358 (time 1786816375.72 total 299.78 last 21.66) - open/close 6638 (time 1786816385.72 total 309.79 last 27.98) - open/close 6892 (time 1786816396.04 total 320.10 last 24.63) - open/close 7149 (time 1786816406.79 total 330.85 last 23.90) - open/close 7412 (time 1786816417.28 total 341.34 last 25.07) - open/close 7684 (time 1786816427.56 total 351.62 last 26.47) - open/close 7943 (time 1786816438.42 total 362.48 last 23.85) - open/close 8180 (time 1786816448.43 total 372.49 last 23.68) - open/close 8461 (time 1786816458.43 total 382.49 last 28.09) - open/close 8731 (time 1786816468.45 total 392.51 last 26.96) - open/close 8947 (time 1786816479.42 total 403.48 last 19.68) - open/close 9206 (time 1786816490.04 total 414.10 last 24.39) - open/close 9533 (time 1786816501.06 total 425.12 last 29.68) - open/close 9763 (time 1786816511.39 total 435.45 last 22.26) - open/close 10000 (time 1786816519.74 total 443.80 last 28.39) - open/close 10241 (time 1786816529.74 total 453.80 last 24.09) - open/close 10437 (time 1786816539.75 total 463.81 last 19.58) - open/close 10631 (time 1786816550.08 total 474.15 last 18.78) - open/close 10868 (time 1786816560.10 total 484.16 last 23.66) - open/close 11102 (time 1786816570.12 total 494.18 last 23.36) - open/close 11251 (time 1786816580.12 total 504.18 last 14.90) - open/close 11522 (time 1786816590.68 total 514.74 last 25.67) - open/close 11734 (time 1786816600.69 total 524.75 last 21.18) - open/close 11991 (time 1786816611.52 total 535.58 last 23.71) - open/close 12245 (time 1786816621.54 total 545.60 last 25.37) - open/close 12442 (time 1786816631.54 total 555.61 last 19.68) - open/close 12625 (time 1786816642.51 total 566.57 last 16.69) - open/close 12872 (time 1786816653.39 total 577.45 last 22.71) - open/close 13064 (time 1786816663.39 total 587.46 last 19.19) - open/close 13318 (time 1786816676.17 total 600.23 last 19.88) - open/close 13532 (time 1786816686.17 total 610.23 last 21.40) - open/close 13761 (time 1786816696.22 total 620.28 last 22.79) - open/close 14007 (time 1786816706.22 total 630.28 last 24.59) - open/close 14233 (time 1786816716.93 total 640.99 last 21.11) - open/close 14475 (time 1786816726.93 total 650.99 last 24.20) - open/close 14692 (time 1786816736.93 total 660.99 last 21.69) - open/close 14920 (time 1786816747.71 total 671.77 last 21.15) - open/close 15147 (time 1786816758.29 total 682.35 last 21.46) - open/close 15351 (time 1786816768.52 total 692.59 last 19.93) - open/close 15486 (time 1786816778.54 total 702.60 last 13.48) - open/close 15636 (time 1786816788.55 total 712.61 last 14.99) - open/close 15851 (time 1786816798.56 total 722.62 last 21.47) - open/close 16090 (time 1786816808.84 total 732.90 last 23.25) - open/close 16376 (time 1786816818.85 total 742.92 last 28.55) - open/close 16615 (time 1786816829.82 total 753.89 last 21.79) - open/close 16848 (time 1786816839.83 total 763.89 last 23.28) - open/close 17054 (time 1786816850.17 total 774.23 last 19.93) - open/close 17282 (time 1786816860.29 total 784.35 last 22.53) - open/close 17566 (time 1786816871.16 total 795.22 last 26.13) - open/close 17841 (time 1786816881.68 total 805.74 last 26.15) - open/close 18042 (time 1786816892.44 total 816.50 last 18.67) - open/close 18281 (time 1786816902.46 total 826.52 last 23.85) - open/close 18495 (time 1786816913.35 total 837.41 last 19.65) - open/close 18749 (time 1786816924.35 total 848.41 last 23.09) - open/close 19009 (time 1786816934.37 total 858.43 last 25.95) - open/close 19227 (time 1786816945.36 total 869.42 last 19.84) - open/close 19957 (time 1786816955.36 total 879.43 last 72.98) - open/close 20696 (time 1786816965.38 total 889.44 last 73.81) - open/close 21328 (time 1786816975.38 total 899.44 last 63.20) - open/close 22018 (time 1786816985.39 total 909.45 last 68.93) - open/close 22713 (time 1786816995.39 total 919.46 last 69.45) - open/close 23470 (time 1786817005.48 total 929.55 last 75.02) - open/close 24231 (time 1786817015.49 total 939.55 last 76.08) - open/close 25030 (time 1786817025.51 total 949.57 last 79.74) - open/close 25822 (time 1786817035.51 total 959.57 last 79.17) - open/close 26566 (time 1786817045.52 total 969.58 last 74.35) - open/close 27376 (time 1786817055.53 total 979.59 last 80.94) - open/close 28138 (time 1786817065.53 total 989.59 last 76.19) - open/close 28859 (time 1786817075.54 total 999.60 last 72.04) - open/close 29554 (time 1786817085.54 total 1009.60 last 69.50) - open/close 30000 (time 1786817091.46 total 1015.52 last 75.28) - open/close 30791 (time 1786817101.47 total 1025.54 last 78.99) - open/close 31545 (time 1786817111.49 total 1035.55 last 75.31) - open/close 32317 (time 1786817121.50 total 1045.56 last 77.13) - open/close 33055 (time 1786817131.51 total 1055.57 last 73.70) - open/close 33771 (time 1786817141.52 total 1065.58 last 71.53) - open/close 34563 (time 1786817151.53 total 1075.59 last 79.12) - open/close 35343 (time 1786817161.54 total 1085.61 last 77.88) - open/close 36108 (time 1786817171.55 total 1095.61 last 76.43) - open/close 36786 (time 1786817181.55 total 1105.61 last 67.79) - open/close 37511 (time 1786817191.56 total 1115.62 last 72.43) - open/close 38349 (time 1786817201.57 total 1125.63 last 83.76) - open/close 39125 (time 1786817211.57 total 1135.63 last 77.58) - open/close 39862 (time 1786817221.57 total 1145.63 last 73.69) - open/close 40618 (time 1786817231.58 total 1155.65 last 75.51) - open/close 41358 (time 1786817241.59 total 1165.65 last 73.98) - open/close 42162 (time 1786817251.60 total 1175.67 last 80.26) - open/close 42940 (time 1786817261.60 total 1185.67 last 77.80) - open/close 43746 (time 1786817271.62 total 1195.68 last 80.51) - open/close 44507 (time 1786817281.62 total 1205.68 last 76.08) - open/close 45302 (time 1786817291.62 total 1215.68 last 79.49) - open/close 46057 (time 1786817301.63 total 1225.69 last 75.46) - open/close 46791 (time 1786817311.63 total 1235.69 last 73.36) - open/close 47496 (time 1786817321.65 total 1245.72 last 70.34) - open/close 48161 (time 1786817331.66 total 1255.73 last 66.42) - open/close 48906 (time 1786817341.67 total 1265.74 last 74.43) - open/close 49601 (time 1786817351.69 total 1275.75 last 69.39) total: 50000 open/close in 1281.55 seconds: 39.02 ops/second - unlinked 0 (time 1786817361 ; total 0 ; last 0) - unlinked 10000 (time 1786817513 ; total 152 ; last 152) - unlinked 20000 (time 1786817669 ; total 308 ; last 156) - unlinked 30000 (time 1786817825 ; total 464 ; last 156) - unlinked 40000 (time 1786817970 ; total 609 ; last 145) total: 50000 unlinks in 756 seconds: 66.137566 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg646-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg646-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg646-server stop mds service on oleg646-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg646-server unloading modules via unload_modules_local on: 'oleg646-server' oleg646-server: oleg646-server.virtnet: executing unload_modules_local oleg646-server: modules unloaded. pdsh@oleg646-client: oleg646-client: ssh exited with exit code 2 pdsh@oleg646-client: oleg646-server: ssh exited with exit code 2 pdsh@oleg646-client: oleg646-client: ssh exited with exit code 2 pdsh@oleg646-client: oleg646-server: ssh exited with exit code 2