== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:33:40 (1786815220) start mds service on oleg103-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg103-server' oleg103-server: oleg103-server.virtnet: executing load_modules_local oleg103-server: Loading modules from /home/green/git/lustre-release/lustre oleg103-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg103-server: detected 4 online CPUs by sysfs oleg103-server: MODOPTS_LIBCFS= oleg103-server: Force libcfs to create 2 CPU partitions oleg103-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg103-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg103-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg103-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Started lustre-OST0000 oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg103-client.virtnet: -o user_xattr,flock 192.168.201.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 89 (time 1786815312.98 total 10.01 last 8.89) - open/close 154 (time 1786815323.40 total 20.44 last 6.23) - open/close 316 (time 1786815334.17 total 31.21 last 15.04) - open/close 522 (time 1786815344.22 total 41.26 last 20.51) - open/close 766 (time 1786815354.23 total 51.26 last 24.38) - open/close 914 (time 1786815364.53 total 61.57 last 14.36) - open/close 1215 (time 1786815374.54 total 71.58 last 30.07) - open/close 1437 (time 1786815384.56 total 81.60 last 22.17) - open/close 1658 (time 1786815395.57 total 92.61 last 20.06) - open/close 1812 (time 1786815405.67 total 102.71 last 15.26) - open/close 1913 (time 1786815416.54 total 113.57 last 9.29) - open/close 2085 (time 1786815426.57 total 123.61 last 17.14) - open/close 2160 (time 1786815439.02 total 136.06 last 6.02) - open/close 2270 (time 1786815449.03 total 146.07 last 10.99) - open/close 2459 (time 1786815459.18 total 156.22 last 18.62) - open/close 2571 (time 1786815469.20 total 166.24 last 11.18) - open/close 2704 (time 1786815479.65 total 176.69 last 12.72) - open/close 2862 (time 1786815490.60 total 187.64 last 14.44) - open/close 2956 (time 1786815501.26 total 198.30 last 8.82) - open/close 3122 (time 1786815511.28 total 208.31 last 16.57) - open/close 3264 (time 1786815522.09 total 219.13 last 13.13) - open/close 3438 (time 1786815532.10 total 229.13 last 17.39) - open/close 3500 (time 1786815543.01 total 240.05 last 5.68) - open/close 3578 (time 1786815554.56 total 251.60 last 6.76) - open/close 3702 (time 1786815564.57 total 261.60 last 12.39) - open/close 3832 (time 1786815574.57 total 271.61 last 12.99) - open/close 3995 (time 1786815585.01 total 282.04 last 15.62) - open/close 4156 (time 1786815595.02 total 292.06 last 16.08) - open/close 4197 (time 1786815608.62 total 305.66 last 3.01) - open/close 4344 (time 1786815619.19 total 316.22 last 13.91) - open/close 4506 (time 1786815629.19 total 326.23 last 16.19) - open/close 4754 (time 1786815639.20 total 336.24 last 24.78) - open/close 4894 (time 1786815649.32 total 346.36 last 13.83) - open/close 5038 (time 1786815659.49 total 356.53 last 14.16) - open/close 5200 (time 1786815669.50 total 366.54 last 16.19) - open/close 5441 (time 1786815679.51 total 376.55 last 24.08) - open/close 5591 (time 1786815689.83 total 386.87 last 14.53) - open/close 5786 (time 1786815699.85 total 396.88 last 19.47) - open/close 5993 (time 1786815710.21 total 407.25 last 19.97) - open/close 6207 (time 1786815720.23 total 417.27 last 21.36) - open/close 6263 (time 1786815730.26 total 427.30 last 5.58) - open/close 6403 (time 1786815740.73 total 437.77 last 13.37) - open/close 6619 (time 1786815750.74 total 447.78 last 21.58) - open/close 6801 (time 1786815760.77 total 457.81 last 18.15) - open/close 7014 (time 1786815770.77 total 467.81 last 21.28) - open/close 7247 (time 1786815780.78 total 477.82 last 23.29) - open/close 7322 (time 1786815790.79 total 487.83 last 7.49) - open/close 7470 (time 1786815801.12 total 498.16 last 14.32) - open/close 7716 (time 1786815811.13 total 508.16 last 24.60) - open/close 7879 (time 1786815822.17 total 519.21 last 14.76) - open/close 7977 (time 1786815832.43 total 529.46 last 9.56) - open/close 8147 (time 1786815843.37 total 540.40 last 15.54) - open/close 8404 (time 1786815854.32 total 551.35 last 23.47) - open/close 8593 (time 1786815864.34 total 561.37 last 18.86) - open/close 8800 (time 1786815874.47 total 571.51 last 20.42) - open/close 9040 (time 1786815885.31 total 582.35 last 22.14) - open/close 9230 (time 1786815895.32 total 592.36 last 18.98) - open/close 9383 (time 1786815905.35 total 602.39 last 15.26) - open/close 9553 (time 1786815916.01 total 613.05 last 15.94) - open/close 9760 (time 1786815926.77 total 623.81 last 19.24) - open/close 9964 (time 1786815937.32 total 634.36 last 19.35) - open/close 10202 (time 1786815947.66 total 644.70 last 23.02) - open/close 10350 (time 1786815957.99 total 655.03 last 14.33) - open/close 10519 (time 1786815967.99 total 665.03 last 16.90) - open/close 10604 (time 1786815978.18 total 675.22 last 8.34) - open/close 10729 (time 1786815988.21 total 685.24 last 12.46) - open/close 10919 (time 1786815999.02 total 696.06 last 17.57) - open/close 11072 (time 1786816009.03 total 706.06 last 15.29) - open/close 11261 (time 1786816019.52 total 716.56 last 18.01) - open/close 11387 (time 1786816030.45 total 727.49 last 11.53) - open/close 11498 (time 1786816040.46 total 737.50 last 11.09) - open/close 11573 (time 1786816050.60 total 747.64 last 7.40) - open/close 11692 (time 1786816060.79 total 757.83 last 11.68) - open/close 11926 (time 1786816070.79 total 767.83 last 23.38) - open/close 12141 (time 1786816080.81 total 777.84 last 21.47) - open/close 12359 (time 1786816091.57 total 788.61 last 20.25) - open/close 12542 (time 1786816101.87 total 798.91 last 17.76) - open/close 12778 (time 1786816111.88 total 808.91 last 23.59) - open/close 13018 (time 1786816121.90 total 818.94 last 23.94) - open/close 13237 (time 1786816132.54 total 829.58 last 20.59) - open/close 13433 (time 1786816142.55 total 839.59 last 19.57) - open/close 13650 (time 1786816153.52 total 850.55 last 19.79) - open/close 13901 (time 1786816163.69 total 860.73 last 24.67) - open/close 14076 (time 1786816174.64 total 871.68 last 15.98) - open/close 14290 (time 1786816185.21 total 882.25 last 20.25) - open/close 14450 (time 1786816195.51 total 892.55 last 15.53) - open/close 14675 (time 1786816205.52 total 902.56 last 22.49) - open/close 14898 (time 1786816215.52 total 912.56 last 22.29) - open/close 15176 (time 1786816225.53 total 922.56 last 27.79) - open/close 15426 (time 1786816235.54 total 932.57 last 24.97) - open/close 15701 (time 1786816246.40 total 943.44 last 25.32) - open/close 15943 (time 1786816256.41 total 953.45 last 24.18) - open/close 16089 (time 1786816266.60 total 963.64 last 14.32) - open/close 16211 (time 1786816277.37 total 974.41 last 11.33) - open/close 16337 (time 1786816287.54 total 984.57 last 12.39) - open/close 16557 (time 1786816298.41 total 995.45 last 20.23) - open/close 16870 (time 1786816309.47 total 1006.51 last 28.30) - open/close 17136 (time 1786816320.50 total 1017.53 last 24.13) - open/close 17328 (time 1786816330.50 total 1027.54 last 19.20) - open/close 17570 (time 1786816341.04 total 1038.07 last 22.96) - open/close 18395 (time 1786816351.04 total 1048.08 last 82.44) - open/close 19147 (time 1786816361.04 total 1058.08 last 75.20) - open/close 19832 (time 1786816371.04 total 1068.08 last 68.49) - open/close 20000 (time 1786816373.35 total 1070.39 last 72.89) - open/close 20695 (time 1786816383.36 total 1080.40 last 69.43) - open/close 21410 (time 1786816393.36 total 1090.40 last 71.48) - open/close 22099 (time 1786816403.38 total 1100.42 last 68.77) - open/close 22784 (time 1786816413.39 total 1110.43 last 68.42) - open/close 23438 (time 1786816423.40 total 1120.43 last 65.40) - open/close 24033 (time 1786816433.40 total 1130.44 last 59.49) - open/close 24685 (time 1786816443.41 total 1140.45 last 65.13) - open/close 25331 (time 1786816453.41 total 1150.45 last 64.58) - open/close 25980 (time 1786816463.42 total 1160.46 last 64.82) - open/close 26651 (time 1786816473.44 total 1170.48 last 66.98) - open/close 27316 (time 1786816483.44 total 1180.48 last 66.50) - open/close 27935 (time 1786816493.45 total 1190.48 last 61.88) - open/close 28636 (time 1786816503.45 total 1200.48 last 70.10) - open/close 29309 (time 1786816513.45 total 1210.49 last 67.27) - open/close 29991 (time 1786816523.46 total 1220.50 last 68.15) - open/close 30732 (time 1786816533.47 total 1230.51 last 74.03) - open/close 31316 (time 1786816543.47 total 1240.51 last 58.37) - open/close 32079 (time 1786816553.47 total 1250.51 last 76.29) - open/close 32837 (time 1786816563.48 total 1260.51 last 75.80) - open/close 33514 (time 1786816573.48 total 1270.52 last 67.64) - open/close 34134 (time 1786816583.49 total 1280.53 last 61.96) - open/close 34698 (time 1786816593.49 total 1290.53 last 56.37) - open/close 35310 (time 1786816603.51 total 1300.55 last 61.12) - open/close 35919 (time 1786816613.51 total 1310.55 last 60.86) - open/close 36656 (time 1786816623.53 total 1320.57 last 73.60) - open/close 37337 (time 1786816633.53 total 1330.57 last 68.07) - open/close 38060 (time 1786816643.54 total 1340.58 last 72.25) - open/close 38758 (time 1786816653.55 total 1350.59 last 69.74) - open/close 39436 (time 1786816663.55 total 1360.59 last 67.78) - open/close 40000 (time 1786816671.57 total 1368.61 last 70.32) - open/close 40711 (time 1786816681.58 total 1378.62 last 71.04) - open/close 41388 (time 1786816691.61 total 1388.65 last 67.51) - open/close 42042 (time 1786816701.61 total 1398.65 last 65.40) - open/close 42793 (time 1786816711.61 total 1408.65 last 75.07) - open/close 43526 (time 1786816721.61 total 1418.65 last 73.29) - open/close 44247 (time 1786816731.62 total 1428.66 last 72.05) - open/close 44957 (time 1786816741.62 total 1438.66 last 70.99) - open/close 45672 (time 1786816751.64 total 1448.67 last 71.39) - open/close 46380 (time 1786816761.64 total 1458.68 last 70.74) - open/close 47027 (time 1786816771.64 total 1468.68 last 64.69) - open/close 47738 (time 1786816781.66 total 1478.70 last 71.00) - open/close 48332 (time 1786816791.66 total 1488.70 last 59.38) - open/close 48942 (time 1786816801.67 total 1498.71 last 60.95) - open/close 49579 (time 1786816811.68 total 1508.72 last 63.61) total: 50000 open/close in 1516.03 seconds: 32.98 ops/second - unlinked 0 (time 1786816822 ; total 0 ; last 0) - unlinked 10000 (time 1786816980 ; total 158 ; last 158) - unlinked 20000 (time 1786817153 ; total 331 ; last 173) - unlinked 30000 (time 1786817314 ; total 492 ; last 161) - unlinked 40000 (time 1786817467 ; total 645 ; last 153) total: 50000 unlinks in 802 seconds: 62.344139 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg103-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg103-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg103-server stop mds service on oleg103-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg103-server unloading modules via unload_modules_local on: 'oleg103-server' oleg103-server: oleg103-server.virtnet: executing unload_modules_local oleg103-server: modules unloaded. pdsh@oleg103-client: oleg103-client: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-server: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-server: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-client: ssh exited with exit code 2