== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:43:09 (1786758189) start mds service on oleg240-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg240-server' oleg240-server: oleg240-server.virtnet: executing load_modules_local oleg240-server: Loading modules from /home/green/git/lustre-release/lustre oleg240-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg240-server: detected 4 online CPUs by sysfs oleg240-server: MODOPTS_LIBCFS= oleg240-server: Force libcfs to create 2 CPU partitions oleg240-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg240-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg240-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg240-server: oleg240-server.virtnet: executing set_default_debug -1 all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg240-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug -1 all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 244 (time 1786758282.63 total 10.00 last 24.40) - open/close 418 (time 1786758292.87 total 20.23 last 17.01) - open/close 516 (time 1786758303.30 total 30.67 last 9.39) - open/close 576 (time 1786758313.31 total 40.68 last 5.99) - open/close 686 (time 1786758323.32 total 50.69 last 10.99) - open/close 880 (time 1786758334.07 total 61.43 last 18.05) - open/close 1018 (time 1786758344.25 total 71.62 last 13.55) - open/close 1181 (time 1786758354.27 total 81.64 last 16.27) - open/close 1288 (time 1786758364.47 total 91.83 last 10.49) - open/close 1424 (time 1786758374.48 total 101.84 last 13.59) - open/close 1557 (time 1786758385.34 total 112.71 last 12.24) - open/close 1774 (time 1786758395.48 total 122.85 last 21.40) - open/close 1886 (time 1786758405.50 total 132.87 last 11.18) - open/close 2089 (time 1786758415.74 total 143.11 last 19.83) - open/close 2346 (time 1786758425.97 total 153.34 last 25.11) - open/close 2500 (time 1786758436.46 total 163.83 last 14.68) - open/close 2682 (time 1786758446.47 total 173.83 last 18.19) - open/close 2896 (time 1786758456.48 total 183.85 last 21.37) - open/close 2988 (time 1786758466.64 total 194.00 last 9.06) - open/close 3102 (time 1786758476.79 total 204.16 last 11.22) - open/close 3181 (time 1786758487.69 total 215.06 last 7.25) - open/close 3271 (time 1786758498.43 total 225.79 last 8.38) - open/close 3372 (time 1786758511.37 total 238.73 last 7.81) - open/close 3467 (time 1786758521.98 total 249.35 last 8.95) - open/close 3523 (time 1786758531.99 total 259.35 last 5.60) - open/close 3638 (time 1786758541.99 total 269.36 last 11.49) - open/close 3691 (time 1786758556.99 total 284.35 last 3.53) - open/close 3798 (time 1786758567.45 total 294.81 last 10.23) - open/close 3873 (time 1786758578.46 total 305.82 last 6.81) - open/close 4124 (time 1786758588.97 total 316.34 last 23.87) - open/close 4228 (time 1786758598.99 total 326.35 last 10.39) - open/close 4461 (time 1786758608.99 total 336.36 last 23.29) - open/close 4637 (time 1786758619.19 total 346.56 last 17.26) - open/close 4811 (time 1786758632.26 total 359.62 last 13.32) - open/close 4905 (time 1786758642.61 total 369.97 last 9.08) - open/close 5121 (time 1786758653.55 total 380.92 last 19.73) - open/close 5288 (time 1786758663.75 total 391.11 last 16.38) - open/close 5516 (time 1786758673.97 total 401.34 last 22.30) - open/close 5673 (time 1786758686.53 total 413.89 last 12.51) - open/close 5786 (time 1786758696.54 total 423.90 last 11.29) - open/close 5825 (time 1786758706.94 total 434.31 last 3.75) - open/close 5952 (time 1786758717.50 total 444.86 last 12.03) - open/close 6094 (time 1786758728.12 total 455.49 last 13.36) - open/close 6175 (time 1786758738.13 total 465.50 last 8.09) - open/close 6330 (time 1786758750.70 total 478.07 last 12.33) - open/close 6489 (time 1786758760.73 total 488.10 last 15.85) - open/close 6663 (time 1786758770.94 total 498.31 last 17.05) - open/close 6824 (time 1786758780.94 total 508.31 last 16.09) - open/close 7012 (time 1786758791.38 total 518.74 last 18.02) - open/close 7070 (time 1786758804.27 total 531.64 last 4.50) - open/close 7272 (time 1786758814.90 total 542.26 last 19.02) - open/close 7384 (time 1786758825.78 total 553.15 last 10.29) - open/close 7489 (time 1786758836.03 total 563.40 last 10.24) - open/close 7616 (time 1786758846.03 total 573.40 last 12.69) - open/close 7747 (time 1786758856.38 total 583.74 last 12.67) - open/close 7859 (time 1786758867.39 total 594.75 last 10.17) - open/close 7997 (time 1786758877.94 total 605.30 last 13.08) - open/close 8128 (time 1786758888.18 total 615.55 last 12.78) - open/close 8228 (time 1786758898.22 total 625.59 last 9.96) - open/close 8382 (time 1786758909.05 total 636.41 last 14.23) - open/close 8553 (time 1786758919.08 total 646.45 last 17.03) - open/close 8653 (time 1786758929.09 total 656.46 last 9.99) - open/close 8841 (time 1786758944.85 total 672.22 last 11.93) - open/close 9082 (time 1786758954.87 total 682.23 last 24.07) - open/close 9349 (time 1786758964.88 total 692.25 last 26.66) - open/close 9522 (time 1786758975.79 total 703.15 last 15.86) - open/close 9611 (time 1786758988.38 total 715.75 last 7.07) - open/close 9796 (time 1786758998.38 total 725.75 last 18.50) - open/close 10000 (time 1786759006.71 total 734.08 last 24.50) - open/close 10154 (time 1786759017.94 total 745.30 last 13.72) - open/close 10280 (time 1786759028.29 total 755.66 last 12.17) - open/close 10539 (time 1786759038.30 total 765.66 last 25.89) - open/close 10637 (time 1786759048.76 total 776.12 last 9.37) - open/close 10762 (time 1786759058.88 total 786.25 last 12.34) - open/close 10837 (time 1786759069.63 total 796.99 last 6.98) - open/close 10997 (time 1786759079.65 total 807.01 last 15.97) - open/close 11259 (time 1786759090.11 total 817.48 last 25.04) - open/close 11486 (time 1786759100.61 total 827.97 last 21.63) - open/close 11645 (time 1786759110.61 total 837.98 last 15.89) - open/close 11718 (time 1786759123.59 total 850.96 last 5.62) - open/close 11852 (time 1786759134.19 total 861.56 last 12.64) - open/close 12048 (time 1786759144.65 total 872.01 last 18.75) - open/close 12262 (time 1786759155.08 total 882.45 last 20.51) - open/close 12465 (time 1786759165.62 total 892.98 last 19.26) - open/close 12600 (time 1786759176.39 total 903.76 last 12.53) - open/close 12757 (time 1786759187.33 total 914.70 last 14.35) - open/close 13002 (time 1786759198.25 total 925.62 last 22.43) - open/close 13190 (time 1786759208.82 total 936.19 last 17.78) - open/close 13344 (time 1786759218.95 total 946.32 last 15.21) - open/close 13559 (time 1786759229.44 total 956.81 last 20.49) - open/close 13873 (time 1786759239.45 total 966.82 last 31.38) - open/close 14049 (time 1786759251.77 total 979.13 last 14.29) - open/close 14174 (time 1786759262.72 total 990.08 last 11.41) - open/close 14302 (time 1786759273.09 total 1000.46 last 12.34) - open/close 14379 (time 1786759283.51 total 1010.88 last 7.39) - open/close 14448 (time 1786759293.98 total 1021.35 last 6.59) - open/close 14537 (time 1786759304.76 total 1032.13 last 8.26) - open/close 14637 (time 1786759314.96 total 1042.33 last 9.80) - open/close 14739 (time 1786759325.64 total 1053.01 last 9.55) - open/close 14844 (time 1786759336.18 total 1063.54 last 9.97) - open/close 14976 (time 1786759347.08 total 1074.45 last 12.10) - open/close 15153 (time 1786759357.72 total 1085.08 last 16.64) - open/close 15362 (time 1786759367.73 total 1095.10 last 20.87) - open/close 15500 (time 1786759377.79 total 1105.16 last 13.72) - open/close 16004 (time 1786759387.81 total 1115.17 last 50.31) - open/close 16600 (time 1786759397.82 total 1125.19 last 59.52) - open/close 17406 (time 1786759407.82 total 1135.19 last 80.59) - open/close 18277 (time 1786759417.84 total 1145.21 last 86.92) - open/close 19077 (time 1786759427.85 total 1155.21 last 79.95) - open/close 19924 (time 1786759437.86 total 1165.22 last 84.61) - open/close 20695 (time 1786759447.87 total 1175.24 last 76.98) - open/close 21529 (time 1786759457.88 total 1185.25 last 83.35) - open/close 22119 (time 1786759467.89 total 1195.26 last 58.93) - open/close 22833 (time 1786759477.89 total 1205.26 last 71.40) - open/close 23679 (time 1786759487.89 total 1215.26 last 84.59) - open/close 24368 (time 1786759497.90 total 1225.27 last 68.83) - open/close 24883 (time 1786759507.90 total 1235.27 last 51.49) - open/close 25650 (time 1786759517.91 total 1245.28 last 76.65) - open/close 26408 (time 1786759527.92 total 1255.28 last 75.75) - open/close 27167 (time 1786759537.92 total 1265.29 last 75.84) - open/close 27961 (time 1786759547.93 total 1275.29 last 79.37) - open/close 28786 (time 1786759557.93 total 1285.29 last 82.50) - open/close 29444 (time 1786759567.93 total 1295.30 last 65.79) - open/close 30000 (time 1786759574.53 total 1301.90 last 84.21) - open/close 30849 (time 1786759584.54 total 1311.91 last 84.80) - open/close 31764 (time 1786759594.57 total 1321.93 last 91.28) - open/close 32442 (time 1786759604.57 total 1331.94 last 67.79) - open/close 33236 (time 1786759614.57 total 1341.94 last 79.35) - open/close 33805 (time 1786759624.58 total 1351.95 last 56.86) - open/close 34447 (time 1786759634.58 total 1361.95 last 64.20) - open/close 35185 (time 1786759644.58 total 1371.95 last 73.80) - open/close 36078 (time 1786759654.58 total 1381.95 last 89.29) - open/close 36965 (time 1786759664.59 total 1391.96 last 88.64) - open/close 37644 (time 1786759674.60 total 1401.96 last 67.85) - open/close 38304 (time 1786759684.61 total 1411.98 last 65.89) - open/close 38979 (time 1786759694.61 total 1421.98 last 67.50) - open/close 39733 (time 1786759704.62 total 1431.99 last 75.33) - open/close 40000 (time 1786759707.62 total 1434.99 last 88.92) - open/close 40910 (time 1786759717.63 total 1444.99 last 90.98) - open/close 41648 (time 1786759727.70 total 1455.07 last 73.26) - open/close 42541 (time 1786759737.70 total 1465.07 last 89.26) - open/close 43345 (time 1786759747.71 total 1475.08 last 80.37) - open/close 44191 (time 1786759757.71 total 1485.08 last 84.59) - open/close 45103 (time 1786759767.71 total 1495.08 last 91.16) - open/close 45877 (time 1786759777.72 total 1505.09 last 77.33) - open/close 46682 (time 1786759787.73 total 1515.09 last 80.49) - open/close 47497 (time 1786759797.74 total 1525.10 last 81.40) - open/close 48238 (time 1786759807.75 total 1535.11 last 74.03) - open/close 49004 (time 1786759817.75 total 1545.11 last 76.59) - open/close 49819 (time 1786759827.75 total 1555.12 last 81.44) total: 50000 open/close in 1557.55 seconds: 32.10 ops/second - unlinked 0 (time 1786759834 ; total 0 ; last 0) - unlinked 10000 (time 1786759959 ; total 125 ; last 125) - unlinked 20000 (time 1786760088 ; total 254 ; last 129) - unlinked 30000 (time 1786760222 ; total 388 ; last 134) - unlinked 40000 (time 1786760349 ; total 515 ; last 127) total: 50000 unlinks in 659 seconds: 75.872536 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg240-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg240-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg240-server stop mds service on oleg240-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg240-server unloading modules via unload_modules_local on: 'oleg240-server' oleg240-server: oleg240-server.virtnet: executing unload_modules_local oleg240-server: modules unloaded. pdsh@oleg240-client: oleg240-client: ssh exited with exit code 2 pdsh@oleg240-client: oleg240-server: ssh exited with exit code 2 pdsh@oleg240-client: oleg240-server: ssh exited with exit code 2 pdsh@oleg240-client: oleg240-client: ssh exited with exit code 2